It will only grow over time if you're doing it wrong. SVN is not meant to check out entire repository trees with all branches and tags. That could very quickly eat up all available hard disk space on large repositories with a lot of branches and tags.
Instead, just check out the single branch or trunk or tag that you need. You can always use
svn switch to switch between them in a single working copy if you like.
If you must have multiple branches in one working copy for some reason (perhaps you want to do some major moving around between branches or something) then you should create a sparse working copy containing only the branches you need, to prevent taking up needless disk space and bandwidth.
Note, TortoiseSVN has a nice "Choose items..." dialog on the checkout dialog and "update to revision" dialog that allows you to quickly create a sparse working copy with exactly the content you want.