This is a concept that's interested me for a while, although I haven't got around to writing a script for it yet as I've been a little busy!
There is no easy way for software to really know what something will compress to, without running the files through the algorithm first (keep in mind, tar is just an archiver, not a compression utility, but you can use it with gzip). The other (easier) option would be to keep adding files to the archive, then check the size each time. I don't know of any premade solutions out there to achieve this, but it can be implemented in a few lines of code!
The basic logic would be something like this:
This is dependent on which utility you use as well. Some do not have switches which allow you to easily remove a file from the archive, so you may need to add a second input variable for leeway. This second variable could determine your estimated max size of a file in the directory, and if the archive is more than that size away from the preferred size, it would add it to the file. If there wasn't enough room, the archive would close and create a new one. Of course, you could tweak this algorithm into infinity, ensuring the size of the next file is checked and compared with the current archive size. You'd probably also want to use the biggest files first, so as you can see a lot of logic can go into this.
Unfortunately this isn't a full solution, but I hope this is a good starting point!