I have a directory containing around 500k files, and want to slice them into t
tar files.
Put formally, let's call the files file_0, ..., file_{N-1}
, where N
around 500k. I want to create t
tar files each containing T=N/t
files, where the i-th tar file contains
file_(i*N), ..., file_((i+1)*N - 1), i in {0, ..., t-1}
What's an efficient way to do this? I was going to write a Python script that just loops over the N
files and divides them into t
folders, and then calls tar
in each, but this feels very unoptimal. I have many cores on the server and feel like this should happen in parallel.
xjobs
has the-l
option to combine several input arguments for each job. You'd have to separately count the files and divide byN
prior to that, though.