Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Specifically, I would like to be able to download certain pages from my user profile on the various SE sites. I would, however, like to do this automatically (using a cron job), from the command line and in a parseable format. I much prefer using Linux for this but I could get access to a Mac or Windows machine if necessary.

Ideally, I would like to use a tool like wget or curl to fetch the pages. I don't know how to get past the log in though. I have seen suggestions that mention that you can log in via firefox, export the relevant cookie and import it into wget through its --load-cookiesoption. For example here and here. While this works if I have just logged in, it doesn't after a while. I guess because the ID token has to be refreshed.

So, just after logging in to SU and exporting my cookies I can do:

wget --load-cookies cookies.txt \
  http://superuser.com/users/151431/terdon?tab=responses

After a few minutes though, I get a 404 error:

wget -O ~/stack/$(date +%s) --load-cookies ~/cookies.txt \
   http://superuser.com/users/151431/terdon?tab=responses

--2013-08-06 04:04:14--  http://superuser.com/users/151431/terdon?tab=responses
Resolving superuser.com (superuser.com)... 198.252.206.16
Connecting to superuser.com (superuser.com)|198.252.206.16|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-08-06 04:04:15 ERROR 404: Not Found.

So, how can I automatically log in to an OpenID enabled website from the command line?


PS. I think this is better suited here than in webapps since my question is really about the command line aspect and not the actual details of the web page in question. I would guess that any solution will be applicable to all OpenID sites.

share|improve this question
3  
Have you looked into the SE API (api.stackexchange.com) to see if it provides the information you're looking for? This is the official way to get programmatic access the data and it uses OAuth to authenticate. –  heavyd Aug 6 '13 at 4:26
    
@heavyd yeah, I was kinda hoping I would not have to delve into the API for this. If that's the only way I guess I'll have to. From a cursory glance, it does not appear as though I can automate the login process through the API though. Do you know if I can authenticate in a way that requires no active input from me? If I understand the docs correctly to get data that requires authentication I will need to manually log in. –  terdon Aug 6 '13 at 13:19
    
I haven't actually used the SE API, but in other OAuth implementations I've used you login once and you're given a token which is good indefinitely. –  heavyd Aug 6 '13 at 14:26

1 Answer 1

(Re-) Read the man page for wget and look at the descriptions for the --user and --password flags.

Note that passing a password as a command line argument is not recommended, as anyone who runs ps can see it. The best practice is to not save raw passwords anywhere, but the next best is to put it in a file that only the owner can read.

share|improve this answer
2  
The flags you mention are for the HTTP based authentication (basic, digest, etc.). SE does not use any of these for authentication. They rely on a third party OpenID provider for authentication. –  heavyd Aug 6 '13 at 4:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.