I am using the following bash script to merge various gzip archives together:

wget -O bluelvl1.gz http://list.iblocklist.com/?list=ydxerpxkpcfqjaybcssw&fileformat=p2p&archiveformat=gz
wget -O bluelvl2.gz http://list.iblocklist.com/?list=gyisgnzbhppbvsphucsw&fileformat=p2p&archiveformat=gz
wget -O badpeer.gz http://list.iblocklist.com/?list=cwworuawihqvocglcoss&fileformat=p2p&archiveformat=gz
wget -O microsoft.gz http://list.iblocklist.com/?list=xshktygkujudfnjfioro&fileformat=p2p&archiveformat=gz
wget -O unallocated.gz http://list.iblocklist.com/?list=gihxqmhyunbxhbmgqrla&fileformat=p2p&archiveformat=gz
cat bluelvl1.gz bluelvl2.gz badpeer.gz microsoft.gz unallocated.gz > blocklist.p2p.gz

The problem I'm having is that the "cat" command is creating "blocklist.p2p.gz" with nothing in it. I believe that the problem is caused by Bash not waiting for wget to complete the download.

I believe this because if I copy-paste each line into the command prompt one at a time (and personally wait for the download to finish), Cat creates a merged archive as I expected. If I copy all six lines above from the script then paste the whole lot into the same command prompt, I see the same problem the script is having.

So is there a way to make Bash wait for wget to finish? Is Bash supposed to wait by default and something is causing this to not occur? I would greatly appreciate a solution.

EDIT: As per the comments below, the correct script to use is:

wget -O bluelvl1.gz "http://list.iblocklist.com/?list=ydxerpxkpcfqjaybcssw&fileformat=p2p&archiveformat=gz"
wget -O bluelvl2.gz "http://list.iblocklist.com/?list=gyisgnzbhppbvsphucsw&fileformat=p2p&archiveformat=gz"
wget -O badpeer.gz "http://list.iblocklist.com/?list=cwworuawihqvocglcoss&fileformat=p2p&archiveformat=gz"
wget -O microsoft.gz "http://list.iblocklist.com/?list=xshktygkujudfnjfioro&fileformat=p2p&archiveformat=gz"
wget -O unallocated.gz "http://list.iblocklist.com/?list=gihxqmhyunbxhbmgqrla&fileformat=p2p&archiveformat=gz"
cat bluelvl1.gz bluelvl2.gz badpeer.gz microsoft.gz unallocated.gz > blocklist.p2p.gz

Bash is seeing the & symbols in the URL as a terminator of the command and executing it as a background process. If you enter any command in bash and append a & to it, it will run it in the background and return control to the calling script or the terminal immediately. The fields between them are being executed as commands itself.

To keep bash from interpreting the URL this way, enclose the entire thing in quotes. This will make it pass the whole URL as a string parameter, and it will run as a foreground process.

  • Wouldn't it need to be in single quotes? Or does that only apply to things like variable expansion? – ecube Dec 9 '15 at 23:14
  • The suggestion to add quotes was exactly the solution required. I used double quotes around each URL and now Bash waits for each Wget command to finish. – XJDHDR Dec 10 '15 at 19:18
  • @dma1324 Double quotes allow for variable expansion, correct. Single quotes will not expand variables. But in this particular case, either will be fine since the URLs don't have any $ metacharacters indicating variables. – Ben Richards Dec 10 '15 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.