symlink of concatenated files?

Question

In a Linux system, I have a set of very big files, containing a long list of entries, one per line, that are read one by one to create certain indexing files for each of them, that are then used for another process later on. For example:

indexprogram file1.txt
# creates file1.txt.idxA file1.txt.idxB file.txt.idxC file.txt.idxD
analysisprogram file1.txt
# reads file1.txt as well as the index files and produces some output

I would like to combine the files I have by creating something equivalent to a symlink of concatenated files, so that I can reuse the files and save some space, and only produce the index files from the combined input. For example:

create symlink of "cat file1.txt file2.txt file3.txt > file1.2.3.txt"
indexprogram file1.2.3.txt
# creates file1.2.3.txt.idxA file1.2.3.txt.idxB file1.2.3.txt.idxC file1.2.3.txt.idxD
analysisprogram file1.2.3.txt
# reads file1.2.3.txt as if it were a real file, reads index files, produces output

Is there a way of creating this symbolic concatenation of files? Maybe using named pipes or some similar trickery?

Answer 1

No, I don't think this is possible.

A file is a collection of bytes on a disk, identified by a unique ID and some metadata, including filename etc. A symlink points to the unique ID of one other file. I don't know of an implementation where one link points to multiple targets.

To do this you might want to look into using a database, for example sqlite or something similar.

Also see this related topic http://stackoverflow.com/questions/423320/can-i-symlink-multiple-directories-into-one

Answer 2

Try this (untested)

mkfifo file1.2.3.txt
cat file1.txt file2.txt file3.txt > file1.2.3.txt
indexprogram file1.2.3.txt
cat file1.txt file2.txt file3.txt > file1.2.3.txt
analysisprogram file1.2.3.txt

It's probably cleaner to update to indexprogram and analysisprogram to read from stdin if no filenames are given on the command line.

cat file1.txt file2.txt file3.txt | indexprogram 
cat file1.txt file2.txt file3.txt | analysisprogram