rI took over responsibility for a piece of Software that preprocesses data into a deep directory structure of millions of small (48k) files with a geohash like structure.
Moving and copying this files is a real pain.
At first i want to backup the data into some archive to reduce the load from the file system. As a next step i want to enhance the tool to work with archives instead of files.
The first problem arises already then packing this files into an archive.
Windows zip tool and 7zip abort the process.
Also the zip tool that ships with gnuWin32
Zip 3.0 (July 5th 2008), by Info-Zip cancels the operation with
zip error: Out of memory (local_to_display_string)
So I am searching for advice for two questions:
what is an appropriate archive file format (and a tool that can handle this use case) that allows random access to files. The data within the files does not need to be randomly accessible. since the files are small enough to load into memory.
Is there a C/C++ Library that implements archive access as close to normal file handling as possible, so that changes at the source code are as minimal as possible? The software itself uses Qt/Boost.