Tag Info

Hot answers tagged


Download and learn to use GNU Parallel https://www.gnu.org/software/parallel/ With single line commands you will be running things in parallel quite soon, not only web page downloads.


You speed this up by chunking your operation into several smaller operations. For example, you could create 10 curl processes and background them: curl -O "http://website.com/pages/0[000000-999999].html" & curl -O "http://website.com/pages/1[000000-999999].html" & curl -O "http://website.com/pages/2[000000-999999].html" & ... and so on. ...

Only top voted, non community-wiki answers of a minimum length are eligible