See title. Assume I have the following list of files and relatives sizes:

cat list_of_files.txt:

1243 001.txt
124  002.txt
4132 003.txt
3000 004.txt

I want to be able to split into multiple chunks having the same size (on disk), i.e, for N=2 it would be:

for N=2 

list_A.txt: 001.txt 002.txt 004.txt
list B.txt: 003.txt


list_A.txt: 001.txt 002.txt
list_B.txt: 003.txt
list_C.txt: 004.tx


What would be the most efficient way of computing those chunks?

FYI: I want to be able to parallelize rsync, if you have any better way of doing it (rsyncing each file individually won't do it), I'd really appreciate suggestions.


This is a classic combinatorial, it's called "partition problem".

You can make a C program to compute that, dont worry about the algorithm, there are many libraries available online. I've just found this, in case you need to run it often you'll have to write the C prog to search in directories and grab the file sizes.

If you want to use it just once or in a more messy way, you can put all filenames and sizes in a file name ALL_FILES, grab the filenames and sizes in a C array and compute partitions using the library.

Dont like C? Plenty of other options, just search "partition problem ur_fav_language"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.