Regarding Unix Move Command

Question

I need to write an Unix Shell Script tran.sh that moves the csv input files from /exp/files folder to /exp/ready directory. The csv input files are written to /exp/files folder by an FTP server whose behavior I cannot trivially change. In tran.sh shell script I need to ensure before doing a move of that csv input file from /exp/files directory no longer any other process is writing to the file. how can i do it.

Answer 1

There is no portable way to do so. You can try fuser, lsof, inotify, FAM, and others though.

Answer 2

Try using fuser [FILE]. It'll return nonzero if the file's not in use.

Here's some example code that will wait for a file to become ready to move:

#!/bin/sh

FROMDIR='/exp/files'
DESTDIR='/exp/ready'

function move_file_if_ready () {
    if [ -f "$1" ]; then
        while fuser "$1" 2>/dev/null 1>&2 ; do
            sleep 1
        done

        mv "$1" "$DESTDIR"
    fi
}

for "$fn" in "$FROMDIR"/*.csv; do
    move_file_if_ready "$fn"
done

Answer 3

you can use lsof

r=$(lsof /exp/files )
if [ ! -z "$r" ] ;then
  mv /exp/files/*csv /exp/ready
fi

Answer 4

The Right Way to do this is to cause the process writing the files to and rename or move them of its own volition when done with the write. Anything else is prone to race conditions and/or permission problems.

Some particular examples of problem cases:

• If the process moving the files runs as a different user than lsof / fuser / etc, the information isn't guaranteed complete
• If the process doing the write is a shell script, it may spawn a subprocess which opens the file, let it be closed, spawn another subprocess, etc. In this scenario lsof, fuser and similar tools could legitimately show the file as not being accessed even if further subprocess will be started to write to it later.

Other, more subtle race conditions may exist as well -- and regardless, lsof, fuser and such are not POSIX tools and not available everywhere.

Require as a matter of the protocol that the processes writing the files move them into the final location on completion themselves; it's the only safe and portable approach.

EDIT: It has been clarified that the files are being written not by an arbitrary process (which may close and re-open them) but by an FTP server. In this case, incron can be used to run an arbitrary script whenever a file has been closed in this directory.

Answer 5

I don't think you can do this without modifying processes that create these files. Whenever we have task like this in our systems we make sure that process that creates file make it executable at the very end. So process that will move or post-process some other way such files can check for executable bit to make sure originating process has finished working with a file.