Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am downloading a large file via FTP so that I can process the file on my local system. The program that processes the file reads the file line by line in a single pass, creating two new files from the original.

Would it be possible to start processing the file before it is completely downloaded? What are the potential issues with this?

Note: I'm not really worried about the program reaching the end of the file before it's done downloading--the file is being downloaded more quickly than the program can process the data.

share|improve this question
add comment

4 Answers

up vote 4 down vote accepted

It's possible, if the program reads data sequentially from start. (Some, such as unzip, attempt seek to the end of the file first.)

share|improve this answer
    
unzip does so not because of compression, but because of the way Zip files are structured: with all file metadata at the end. Stream-compression programs, such as gzip or xz, do not work this way and can deal with actively-written files. –  grawity Sep 14 '11 at 20:35
add comment

This might be a good candidate for pipelining. Use an FTP client that writes to standard output have the splitting program read from standard input.

share|improve this answer
add comment

Assuming that the file is not compressed you can try the following on Linux:

wget -O - www.fileserver.com/file.ext | tee outputfile.out | ./someprogram

-O - will cause wget to output the file to stdout which will then be piped to someprogram while tee will save a copy of the wget output to outputfile.

share|improve this answer
add comment

Would it be possible to start processing the file before it is completely downloaded?

yes

What are the potential issues with this?

When reaching EOF - how to determine if it's really EOF or if the reader was just faster than the receiver and it should just wait a second before retrying the read?

Solutions to this would be:

  • use timeouts - if the file size does not change anymore for a couple of minutes, the transfer may be considered as complete. I don't like this solution since one never knows, if the transfer was terminated ok or if the connection got lost.

  • try to get the file size beforehand - if using an ftp server one can usually perform a 'dir' command to get the file size before downloading. This value may be passed to the reader program

  • make the download script to create a 'transfer complete' file which the reader-program may check when it encounters EOF.

Note: I'm not really worried about the program reaching the end of the file before it's done downloading--the file is being downloaded more quickly than the program can process the data.

You should - really! What if the network connection is slower than usual just because of some other transfers going on? Would it be ok to loose data in this case?

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.