I have a Python script that takes a .txt file and does something with it. I need a Linux shell command that will search for .txt files throughout the system and pass the .txt files to my Python script. I guess with xargs I can run a process for a file to speed up the program. How can I do that?


I believe that

find / -type f -name "*\.txt" -exec script.py '{}' \;

would run script.py on every txt file in or below / , resulting in as many invocations of your script as there are files found.

Now, if you want to pass multiple found files to your script simultaneously, you can do:

find / -type f -name "*\.txt" -exec script.py '{}' +

From the find(1) man page:

This variant of the -exec action runs the specified command on the selected files, but the command line is built by appending each selected file name at the end; the total number of invocations of the command will be much less than the number of matched files. The command line is built in much the same way that xargs builds its command lines [...]

You can obviously achieve the same thing with xargs, but the syntax is less concise, which is why I would prefer using find's -exec action.


With GNU Parallel you can do is:

find . -type f -name "*\.txt" | parallel python myscript.py

It will run one job per cpu core.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.