Is there a way to retrieve the source code of every webpage in a website and save them to a single text file? With the appropriate CRFL characters just as if the pages were retrieved individually under Internet Explorer? I've tried Wget, but can't seem to find the right combination of options.
-
Not generally, no. The source code may not be available for download, depending on the specific language and options used on the website. You can't get the source code to amazon, for example.– ChrisInEdmontonAug 4, 2015 at 19:05
-
Even with wget, you can only download where you have hyperlinks to follow.– barlopAug 4, 2015 at 19:11
1 Answer
If the webserver has been configured correctly, no, this is not possible. This is because if you were able to download the source code of websites, you would be able to retrieve valuable data such as the website's database authentication details.
You may however download the rendered HTML of the webpage, as it appears in your Internet browser, with wget
:
wget -E -H -k -K -p URLHERE
However some webservers may treat wget
as a robot (a program that scrapes and saves data from web-pages), in this event you may use the -e robots=off
switches in your wget
command.