0

rI took over responsibility for a piece of Software that preprocesses data into a deep directory structure of millions of small (48k) files with a geohash like structure.
Moving and copying this files is a real pain.

At first i want to backup the data into some archive to reduce the load from the file system. As a next step i want to enhance the tool to work with archives instead of files.

The first problem arises already then packing this files into an archive.

Windows zip tool and 7zip abort the process. Also the zip tool that ships with gnuWin32 Zip 3.0 (July 5th 2008), by Info-Zip cancels the operation with zip error: Out of memory (local_to_display_string)

So I am searching for advice for two questions:

  1. what is an appropriate archive file format (and a tool that can handle this use case) that allows random access to files. The data within the files does not need to be randomly accessible. since the files are small enough to load into memory.

  2. Is there a C/C++ Library that implements archive access as close to normal file handling as possible, so that changes at the source code are as minimal as possible? The software itself uses Qt/Boost.

  • What kind of error arises during the archiving process with 7zip and Windows? Is it an option the do the process on the command line? – daniel.neumann Jan 11 '16 at 11:25
  • i tried both. the gui and the commandline version of 7zip. I can't remember the GUI error (i think it was unspecified) and the CLI states: ERROR: Can't allocate required memory!. But i just recognize that the used version is really old. checking a newer one – vlad_tepesch Jan 11 '16 at 11:33
  • Did you try to look up that error? It might be related the amount of memory available for your command line session. In order to reduce the amount of memory needed for the compression process itself you could archive the data without packing it. – daniel.neumann Jan 11 '16 at 11:52
  • @daniel.neumann I have all permissions on the computer that has 16G memory. I also tested the newest version of the tool. It starts the compressions and overnight it also aborts with an strange error : "Das System kann die benötigte Speichermenge nicht bereitstellen" translation: The system can not provide the required memory amount out of memory error. Unlike the old verision that brings the error directly after the scanning for files the new quits after many hours of processing :-( – vlad_tepesch Jan 12 '16 at 9:56
0

Regarding appropriate archive file format, maybe you don't need to archive it at all? Try using tar instead of zip. If this won't help, try organise them into hierarchical structures, similar how git or squid stores large amount of files in their cache dir by separating top-level vs. second-level subdirs, e.g.: 16/256/64, so you'll have less number of unacknowledged I/O requests when you try to open these files.

Secondly you should consider caching, here are some suggestion:

  • use Redis or Memcached to cache the data in memory which were already read,
  • try placing them into some filesystem such as ramfs or tmpfs, so that resides in memory,
  • use virtual memory toucher such as vmtouch which can map files into virtual memory, e.g. to place the file in the memory: vmtouch -vt file123.geo, or to monitor for the whole folder, try: vmtouch -dl /path/dir

However if the files are small and they consist some specific data (like geohashes), maybe it's consider to import them into database? Such as MariaDB or MongoDB? Or maybe index them using Solr Search or Elastic Search? But at the end, it all depends on your specific technical requirements.

  • tar does not fulfil the 'random file access' criteria, as far as i know. Also i already have a nested directory structure. Placing in memory is no options since 1st: the data is too large 2: the preprocessing has do be done only once and does take very long. After that alwas the preprocessed data is used in subsequent program runs. Programmatically working with that files is not the problem. But handling/deploying them is. – vlad_tepesch Jan 11 '16 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.