I am using GNU parallel and want to understand - how can I get the individual string passed to each parallel command?
As an example, GNU Parallel documentation shows how to move files from the current directory to another:
ls | parallel mv {} destdir
So is there a way to get/print each file individually which was passed to parallel?
Case for parallel processing
I need to do parallel processing of checking multiple sites and record
- http return code (2xx, 4xx, 5xx)
- The source URL
- The ultimate destination URL
- the curl exit code
Here is the code which does this:
unset return_code_array
unset destination_url_array
unset exit_code_array
while read -r return_code_var destination_url_var exit_code_var; do
destination_url_array+=("$destination_url_var")
exit_code_array+=("$exit_code_var")
return_code_array+=("$return_code_var")
done < <(printf '%s\n' "${all_valid_URLs_array[@]}" | parallel -j 20 -k 'curl --max-time 20 -sL -o /dev/null -w "%{response_code} %{url_effective} " {}; printf "%s %s\n" "$?" ')
As a result, I have three arrays and they hold the HTTP return code, ultimate destination URL, and the curl exit code status for each corresponding line for the all_valid_URLs_array
entries. I at the same time need to do some processing for each destination_url_var
- like comparing if it matches to the source URL, but have no idea how to get the string which was passed to parallels.
Currently, I am running a second loop after the above one for such processing but want to know if I want to accomplish is possible.
Thanks.
ls
?ls
(without any options) is in practice safe if you know your filenames do not contain newlines. In my 25 years of sysadmin I have only seen that from malicious users, f*cked filesystems, or test files I have created myself.-0
.find
because you need to disable recursion and remove./
from the . The alternative isprintf "%s\0" *
(but it does not work in *csh if the list is too long).