Right now I use python to figure out files that have been modified in the past hour. This is really slow on my network (~50000 files - and checking each one's timestamp). I have a custom script to which I pass this list - it does some transfers/logging in/etc on a remote server.
I want to speed up the file list generation time (it takes ~15-20mins to just figure out the file list). Any suggestions?
One thing that might be helpful is that the network has a netapp filer. That filer creates those .snapshot dirs. Can I somehow hook into the filer (through the API?) and figure out a list that changed recently?
If you're not familiar with the NetApp API, suggestions on how to generate a file list (fast!) with unix commands would be great! (BTW, this is a network filesystem - so there will be multiple machines making changes).