Tag Info

Hot answers tagged

1

Download and learn to use GNU Parallel https://www.gnu.org/software/parallel/ With single line commands you will be running things in parallel quite soon, not only web page downloads.


1

You speed this up by chunking your operation into several smaller operations. For example, you could create 10 curl processes and background them: curl -O "http://website.com/pages/0[000000-999999].html" & curl -O "http://website.com/pages/1[000000-999999].html" & curl -O "http://website.com/pages/2[000000-999999].html" & ... and so on. ...



Only top voted, non community-wiki answers of a minimum length are eligible