wget --page-requisites --span-hosts --convert-links --adjust-extension --execute robots=off --user-agent Mozilla --random-wait https://www.invisionapp.com/inside-design/essential-steps-designing-empathy/

The command above provides the following time stats:

Total wall clock time: 35s
Downloaded: 248 files, 39M in 4.2s (9.36 MB/s)

This website takes about 5 seconds to download and display all files on a hard refresh in the browser.

Why is the wall clock time significantly longer than the download time and is there a way to make it faster?

  • Welcome to SuperUser. Please use the search function. You can find similar questions at stackoverflow.com/questions/3430810/… and several other places in the Stack Exchange network. – Christopher Hostage Jun 17 at 22:38
  • Thanks. I did search, but I believe the question is different. If total_download_time represents the total time spent downloading the files, then this matches the speed of the browser, which would be ideal, however, wall_clock - total_download_time = 35s - 4.2s = 30.8s seems like a rather excessive time to write 248 files with cumulative size of 39M to disk. I've also tried removing --convert-links and adding --no-clobber to the wget command and running wget_command & wget_command in the terminal to spawn multiple processes to parallelize the download, but no success. – JustCodin Jun 17 at 23:09
  • A better question might be "What tool should I use instead of wget". Wget is single-threaded. – Christopher Hostage Jun 17 at 23:11
  • @ChristopherHostage Thanks. I may ask a new question. But even with wget as a single-threaded program, shouldn't launching multiple instances wget_command & wget_command (which is from the article you linked that I found before and tried) with --no-clobber effectively achieve parallel downloads? Because --no-clobber checks if any file (or even part of a file) has been downloaded and should skip it and move on to the next file that actually needs to be downloaded. But my results don't achieve this, which is why I asked this question. And 30.8s seems like a long time to write 39M. – JustCodin Jun 17 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.