Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to write an Unix Shell Script tran.sh that moves the csv input files from /exp/files folder to /exp/ready directory.

The csv input files are written to /exp/files folder by an FTP server whose behavior I cannot trivially change. In tran.sh shell script I need to ensure before doing a move of that csv input file from /exp/files directory no longer any other process is writing to the file.

How can I do it.

share|improve this question

migrated from stackoverflow.com Feb 23 '10 at 5:28

This question came from our site for professional and enthusiast programmers.

    
BTW -- knowing that it's an FTP server (as opposed to an arbitrary process) really does change the set of appropriate answers; I've certainly had cause to update mine. Please provide more information in your question next time. –  Charles Duffy Feb 22 '10 at 14:15

5 Answers 5

There is no portable way to do so. You can try fuser, lsof, inotify, FAM, and others though.

share|improve this answer
    
nice answer, cuts straight to the point. i think you could add glib and gamin too –  Matt Joiner Feb 22 '10 at 5:07

Try using fuser [FILE]. It'll return nonzero if the file's not in use.

Here's some example code that will wait for a file to become ready to move:

#!/bin/sh

FROMDIR='/exp/files'
DESTDIR='/exp/ready'

function move_file_if_ready () {
    if [ -f "$1" ]; then
        while fuser "$1" 2>/dev/null 1>&2 ; do
            sleep 1
        done

        mv "$1" "$DESTDIR"
    fi
}

for "$fn" in "$FROMDIR"/*.csv; do
    move_file_if_ready "$fn"
done
share|improve this answer
1  
Not bad, but it still has a race condition. –  dmckee Feb 22 '10 at 4:43
    
Thanks a lot. three remote application processes are ftp' ing the input file to this directory. So i will be able to know with this fuser command if they are writing to the file. –  Anonymous Feb 22 '10 at 5:42
    
@dmckee - True about the race condition, but I assumed that the process writing to it only holds the file handle open long enough to write to it, then closes it, never to open it again. –  amphetamachine Feb 22 '10 at 8:02
    
Thanks a lot. I am planning to check whether the last line inside the csv file is received and validate the file –  Anonymous Feb 22 '10 at 8:35

you can use lsof

r=$(lsof /exp/files )
if [ ! -z "$r" ] ;then
  mv /exp/files/*csv /exp/ready
fi
share|improve this answer
    
Thanks a lot. three remote application processes are ftp' ing the input file to this directory. So i will be able to know with this lsof command if they are writing to the file. –  Anonymous Feb 22 '10 at 5:43
    
But with three (assuming independent) processes writing to the directory's files, then this will not move unless all three are finished. It's more modular to use fuser, and test on a per-file basis. –  amphetamachine Feb 22 '10 at 8:05
    
Thanks a lot. I am planning to check whether the last line inside the csv file is received and validate the file . –  Anonymous Feb 22 '10 at 8:34

The Right Way to do this is to cause the process writing the files to and rename or move them of its own volition when done with the write. Anything else is prone to race conditions and/or permission problems.

Some particular examples of problem cases:

  • If the process moving the files runs as a different user than lsof / fuser / etc, the information isn't guaranteed complete
  • If the process doing the write is a shell script, it may spawn a subprocess which opens the file, let it be closed, spawn another subprocess, etc. In this scenario lsof, fuser and similar tools could legitimately show the file as not being accessed even if further subprocess will be started to write to it later.

Other, more subtle race conditions may exist as well -- and regardless, lsof, fuser and such are not POSIX tools and not available everywhere.

Require as a matter of the protocol that the processes writing the files move them into the final location on completion themselves; it's the only safe and portable approach.

EDIT: It has been clarified that the files are being written not by an arbitrary process (which may close and re-open them) but by an FTP server. In this case, incron can be used to run an arbitrary script whenever a file has been closed in this directory.

share|improve this answer
    
Thanks a lot for the info The Problem is three remote application processes are ftp' ing the file to this directory. So i should not move the file when they are writing it in this /exp/files directory –  Anonymous Feb 22 '10 at 5:44
    
@arav - you can work around this (which is to say, I work around this) with a custom sftp server. (Mine is written in Python using the Paramiko library; there's an ftpdlib for writing standard FTP servers in Python as well, though reasons to avoid FTP are many). –  Charles Duffy Feb 22 '10 at 14:06
    
@arav - ...that said, it sounds to me like another race-condition-free solution that would work for you would be using inotifywatch or incron to run a script of your choice whenever an update is complete. –  Charles Duffy Feb 22 '10 at 14:08

I don't think you can do this without modifying processes that create these files. Whenever we have task like this in our systems we make sure that process that creates file make it executable at the very end. So process that will move or post-process some other way such files can check for executable bit to make sure originating process has finished working with a file.

share|improve this answer
    
The Problem is three remote application processes are ftp' ing the file to this directory. So i can check the three remote app processes are completed. –  Anonymous Feb 22 '10 at 5:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.