Read a file that's actively being written

Question

I am downloading a large file via FTP so that I can process the file on my local system. The program that processes the file reads the file line by line in a single pass, creating two new files from the original.

Would it be possible to start processing the file before it is completely downloaded? What are the potential issues with this?

Note: I'm not really worried about the program reaching the end of the file before it's done downloading--the file is being downloaded more quickly than the program can process the data.

Answer 1

It's possible, if the program reads data sequentially from start. (Some, such as unzip, attempt seek to the end of the file first.)

Answer 2

Assuming that the file is not compressed you can try the following on Linux:

wget -O - www.fileserver.com/file.ext | tee outputfile.out | ./someprogram

-O - will cause wget to output the file to stdout which will then be piped to someprogram while tee will save a copy of the wget output to outputfile.

Answer 3

This might be a good candidate for pipelining. Use an FTP client that writes to standard output have the splitting program read from standard input.

Answer 4

Would it be possible to start processing the file before it is completely downloaded?

yes

What are the potential issues with this?

When reaching EOF - how to determine if it's really EOF or if the reader was just faster than the receiver and it should just wait a second before retrying the read?

Solutions to this would be:

• use timeouts - if the file size does not change anymore for a couple of minutes, the transfer may be considered as complete. I don't like this solution since one never knows, if the transfer was terminated ok or if the connection got lost.

• try to get the file size beforehand - if using an ftp server one can usually perform a 'dir' command to get the file size before downloading. This value may be passed to the reader program

• make the download script to create a 'transfer complete' file which the reader-program may check when it encounters EOF.

Note: I'm not really worried about the program reaching the end of the file before it's done downloading--the file is being downloaded more quickly than the program can process the data.

You should - really! What if the network connection is slower than usual just because of some other transfers going on? Would it be ok to loose data in this case?