Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a series of files in a directory that I need to carry an action out on using a script. Once the action is done, then I want to keep a log that the file has been processed, so that the next time the script is run, it does not attempt to carry out the action again.

So lets say I can find all the files that should be processed like this:

for i in `find /logfolder -name '20*.log'` ; do
    process_log $i
    echo $i >> processedlogsfile
done

So I have a file containing the logs I have processed, and my goal would be to modify the for loop such that these processed logs are not processed a second time.

Doing a manual scan each time seems inefficient, particularly as the processedlogfiles gets bigger:

 if grep -iq "$i" processdlogfiles ; then continue; fi

It would be good if these files could be excluded when setting up the for loop.

Note that the OS in question is a linux derivative, a managment appliance, with a limited toolset (no attr command for example) and so no way to install additional utilities (well it is possible but not an option). Most common bash shell commands are available though.

Also, the filenames and locations of the processed files must remain where they are - they can't be altered to reflect their processed status

share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

Add | fgrep -vf processedlogfiles to your find command

share|improve this answer
add comment

What about splitting things up by folder... New files in one and processed files in the other. Then the "processing" includes moving the files.

share|improve this answer
    
Thanks Brian, the files cannot be moved or renamed unfortunately, I have updated the question –  Paul Jul 11 '12 at 1:43
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.