Would it be possible to start processing the file before it is completely downloaded?
What are the potential issues with this?
When reaching EOF - how to determine if it's really EOF or if the reader was just faster than the receiver and it should just wait a second before retrying the read?
Solutions to this would be:
use timeouts - if the file size does not change anymore for a couple of minutes, the transfer may be considered as complete. I don't like this solution since one never knows, if the transfer was terminated ok or if the connection got lost.
try to get the file size beforehand - if using an ftp server one can usually perform a 'dir' command to get the file size before downloading. This value may be passed to the reader program
make the download script to create a 'transfer complete' file which the reader-program may check when it encounters EOF.
Note: I'm not really worried about the program reaching the end of the file before it's done downloading--the file is being downloaded more quickly than the program can process the data.
You should - really! What if the network connection is slower than usual just because of some other transfers going on? Would it be ok to loose data in this case?