10

Specifically, I would like to be able to download certain pages from my user profile on the various Stack Exchange sites. I would, however, like to do this automatically (using a cron job), from the command line and in a parsable format. I much prefer using Linux for this, but I could get access to a Mac or Windows machine if necessary.

Ideally, I would like to use a tool like Wget or cURL to fetch the pages. I don't know how to get past the log in though. I have seen suggestions that mention that you can log in via Firefox, export the relevant cookie and import it into Wget through its --load-cookiesoption. For example here and here. While this works if I have just logged in, it doesn't after a while. I guess because the ID token has to be refreshed.

So, just after logging in to SU and exporting my cookies I can do:

wget --load-cookies cookies.txt \
  https://superuser.com/users/151431/terdon?tab=responses

After a few minutes though, I get a 404 error:

wget -O ~/stack/$(date +%s) --load-cookies ~/cookies.txt \
   https://superuser.com/users/151431/terdon?tab=responses

--2013-08-06 04:04:14--  https://superuser.com/users/151431/terdon?tab=responses
Resolving superuser.com (superuser.com)... 198.252.206.16
Connecting to superuser.com (superuser.com)|198.252.206.16|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-08-06 04:04:15 ERROR 404: Not Found.

So, how can I automatically log in to an OpenID enabled website from the command line?


PS. I think this is better suited here than in web applications since my question is really about the command line aspect and not the actual details of the web page in question. I would guess that any solution will be applicable to all OpenID sites.

  • 3
    Have you looked into the SE API (api.stackexchange.com) to see if it provides the information you're looking for? This is the official way to get programmatic access the data and it uses OAuth to authenticate. – heavyd Aug 6 '13 at 4:26
  • @heavyd yeah, I was kinda hoping I would not have to delve into the API for this. If that's the only way I guess I'll have to. From a cursory glance, it does not appear as though I can automate the login process through the API though. Do you know if I can authenticate in a way that requires no active input from me? If I understand the docs correctly to get data that requires authentication I will need to manually log in. – terdon Aug 6 '13 at 13:19
  • I haven't actually used the SE API, but in other OAuth implementations I've used you login once and you're given a token which is good indefinitely. – heavyd Aug 6 '13 at 14:26
1

You cant, because Cookies refresh every so often. This is for security purposes, the only way you could do this is the way you already did. atleast, from my understanding.

  • I just saw this is a really old question.... Oops. – Maarten Mar 8 '16 at 11:01
-3

(Re-) Read the man page for wget and look at the descriptions for the --user and --password flags.

Note that passing a password as a command line argument is not recommended, as anyone who runs ps can see it. The best practice is to not save raw passwords anywhere, but the next best is to put it in a file that only the owner can read.

  • 3
    The flags you mention are for the HTTP based authentication (basic, digest, etc.). SE does not use any of these for authentication. They rely on a third party OpenID provider for authentication. – heavyd Aug 6 '13 at 4:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.