How to download this webpage with wget?

Question

I want to download this webpage:

http://forum.ubuntu-it.org/

but it require username and password. So I have used this

wget --save-cookies cookies.txt --post-data 'user=goyamy&passwrd=mypassword' http://forum.ubuntu-it.org/

but it does not work !! Why?

Answer 1

that's possibly because the server uses session cookies to track authentication. Add this option alongside --save-cookies to force the cookie to be saved. so your commmand looks like this

wget --keep-session-cookies --save-cookies cookies.txt --post-data 'user=goyamy&passwrd=mypassword' http://forum.ubuntu-it.org/

I haven't tested it though.

Answer 2

As Colin suggests, this site is using session cookies for authentication, but his answer won't fully work because it won't get you logged in.

You need to have a cookie for wget to pass to the server on the initial request. Use wget's --load-cookies option (documented here). Note that this uses the old cookies.txt file format rather than the sqlite database format that Firefox and Chrome currently use.

Here's what I would do:

1. Using Firefox or Chrome, go to the site and log in. (Make sure your browser is set to save cookies)
2. Quit your browser
3. Find your cookie file
4. Convert to cookies.txt format (see notes below on this)
5. wget --load-cookies cookies.txt http://forum.ubuntu-it.org/

Options to convert from sqlite format to cookies.txt include a python script or a simpler sqlite script (in the comments on that previous link), but the easiest for you might be to install this Firefox extension.