Before I start writing my own app for this, maybe there is already a better solution for the problem:
I need to check an HTTP Server every day for new files to download and process. Basically these are zip files which need to be extracted.
Old files are deleted some day and new files are uploaded multiple times a day. I do not want to process a file twice.
My current solution is to save all the files locally and use wget
with -nc
options, which is called by a cronjob twice a day:
wget -nc -t 10 -o wget.log -r -l 1 --no-parent --reject "index.html*" http://myserver/
Now I can parse the log file, get all new downloaded files and process them:
grep saved wget.log | awk '{ print $6}' # generate a list of downloaded files
But I will accumulate a bunch of files on my disk that I don't need. So, do I need a database to store already downloaded files and check for each file if it was already processed?