Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a series of files in a directory that I need to carry an action out on using a script. Once the action is done, then I want to keep a log that the file has been processed, so that the next time the script is run, it does not attempt to carry out the action again.

So lets say I can find all the files that should be processed like this:

for i in `find /logfolder -name '20*.log'` ; do
    process_log $i
    echo $i >> processedlogsfile

So I have a file containing the logs I have processed, and my goal would be to modify the for loop such that these processed logs are not processed a second time.

Doing a manual scan each time seems inefficient, particularly as the processedlogfiles gets bigger:

 if grep -iq "$i" processdlogfiles ; then continue; fi

It would be good if these files could be excluded when setting up the for loop.

Note that the OS in question is a linux derivative, a managment appliance, with a limited toolset (no attr command for example) and so no way to install additional utilities (well it is possible but not an option). Most common bash shell commands are available though.

Also, the filenames and locations of the processed files must remain where they are - they can't be altered to reflect their processed status

share|improve this question
up vote 1 down vote accepted

Add | fgrep -vf processedlogfiles to your find command

share|improve this answer

What about splitting things up by folder... New files in one and processed files in the other. Then the "processing" includes moving the files.

share|improve this answer
Thanks Brian, the files cannot be moved or renamed unfortunately, I have updated the question – Paul Jul 11 '12 at 1:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .