how to handle/archive/access huge amount of small files

Question

rI took over responsibility for a piece of Software that preprocesses data into a deep directory structure of millions of small (48k) files with a geohash like structure.
Moving and copying this files is a real pain.

At first i want to backup the data into some archive to reduce the load from the file system. As a next step i want to enhance the tool to work with archives instead of files.

The first problem arises already then packing this files into an archive.

Windows zip tool and 7zip abort the process. Also the zip tool that ships with gnuWin32 Zip 3.0 (July 5th 2008), by Info-Zip cancels the operation with zip error: Out of memory (local_to_display_string)

So I am searching for advice for two questions:

1. what is an appropriate archive file format (and a tool that can handle this use case) that allows random access to files. The data within the files does not need to be randomly accessible. since the files are small enough to load into memory.

2. Is there a C/C++ Library that implements archive access as close to normal file handling as possible, so that changes at the source code are as minimal as possible? The software itself uses Qt/Boost.

Answer 1

Regarding appropriate archive file format, maybe you don't need to archive it at all? Try using tar instead of zip. If this won't help, try organise them into hierarchical structures, similar how git or squid stores large amount of files in their cache dir by separating top-level vs. second-level subdirs, e.g.: 16/256/64, so you'll have less number of unacknowledged I/O requests when you try to open these files.

Secondly you should consider caching, here are some suggestion:

• use Redis or Memcached to cache the data in memory which were already read,
• try placing them into some filesystem such as ramfs or tmpfs, so that resides in memory,
• use virtual memory toucher such as vmtouch which can map files into virtual memory, e.g. to place the file in the memory: vmtouch -vt file123.geo, or to monitor for the whole folder, try: vmtouch -dl /path/dir

However if the files are small and they consist some specific data (like geohashes), maybe it's consider to import them into database? Such as MariaDB or MongoDB? Or maybe index them using Solr Search or Elastic Search? But at the end, it all depends on your specific technical requirements.