I have a directory of ~200.000 .npy files with a total size ~100 gb. All files are stored directly below the main directory (i.e. there's no sub-directories). I need to transfer the directory and would like to do it by first compressing it into a smaller number of gzip files that I then transfer using ssh. I've tried to naïvely gzip the whole directory at once, which made my server freeze, requiring a hard reboot.
How can I easily gzip the directory of files into, say, 1000 .gz files that I can then easily transfer and unzip again?
I'd preferably like to do this in a manner, where the maximum resource consumption on the server at any one point (primarily RAM/IO) is agnostic regarding the characteristics of the directory (total size / # of files). I'm hoping that I can find a method that I'll be able to use with even larger directories without making my server freeze. The solution should preferably be using bash or Python. Thanks!