1

An application regularly polls a directory for input csv files which arrive by FTP, so there is an FTP server (currently Filezilla Server) running on that computer.

The problem is, that if a file is in the middle of being uploaded when the application decides to poll the location, then the file is locked and all sorts of unsightly errors are thrown on the application, ruining the user experience.

Now assuming I can't modify the application to wait, is there anything that can be done about this? e.g. The server automatically moving or renaming the file once the upload is complete? Or wait for the upload to be complete before saving the file? Or are there other servers that offer solutions such as this?

  • 2
    "automatically moving or renaming the file once the upload is complete" – Normally you need the client to send proper commands and the server to understand them. FTP as a protocol supports deleting, renaming; see DELE, RNFR, RNTO. But this originates on the client side; you're seeking solution that works on the server. That's why my comment is not an answer. This smells like XY problem. It's good you described a broader picture. Maybe you will consider this client-side approach. – Kamil Maciorowski Dec 14 '17 at 23:32
  • I've seen this before with large files that are many gigs where something on the server touches or moves a partially uploaded file before the upload is complete. Errors such as "Cannot Create Remote File" and "Copying Files to Remote Side Failed". I know storage limits/quotas can cause these too but I have seen this occur when something on the server side messes with the partially uploaded file. – Pimp Juice IT Dec 14 '17 at 23:59
  • Consider uploading to the server across the slower WAN, etc. connection as filename.tmp or something different the process will not touch and then once fully uploaded, run a rename/mv command to rename the fully 100% uploaded file to the file name it needs to be. Furthermore, consider uploading to a /tmp subfolder and then once uploaded, move to the root upload directory or whatever. The upload across the slower WAN usually takes the longest so perhaps the local rename or move command will be much faster and prevent this issue. – Pimp Juice IT Dec 15 '17 at 0:01
  • @KamilMaciorowski - I thought of doing this alright, but will the rename be instant, in the sense that the file will be unlocked as soon as the rename is complete? I ask because sometimes the upload takes several seconds between the file being created and being released, even if it's a very small file, like 30-40 bytes. Also I've edited the question, to make your comment more of an answer. – colmde Dec 15 '17 at 0:53
  • I don't know, I rarely use FTP. If I were you I would just test it. – Kamil Maciorowski Dec 15 '17 at 0:56
0

I would recommend the approach where your application does not poll the location that used by other process(FTP server in this case) to write. Instead, set the process to rnfr/rnto (atomic operation) files after they arrive to the locations that your app polls... Your should try to take readlock on the file and if failed skip it until next attempt. Another aproach is to use a marker file that will be arriving last in the batch of files and signals to you polling app that it can start processing csv files

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.