Specifically, I would like to be able to download certain pages from my user profile on the various SE sites. I would, however, like to do this automatically (using a
cron job), from the command line and in a parseable format. I much prefer using Linux for this but I could get access to a Mac or Windows machine if necessary.
Ideally, I would like to use a tool like
curl to fetch the pages. I don't know how to get past the log in though. I have seen suggestions that mention that you can log in via firefox, export the relevant cookie and import it into
wget through its
--load-cookiesoption. For example here and here. While this works if I have just logged in, it doesn't after a while. I guess because the ID token has to be refreshed.
So, just after logging in to SU and exporting my cookies I can do:
wget --load-cookies cookies.txt \ http://superuser.com/users/151431/terdon?tab=responses
After a few minutes though, I get a 404 error:
wget -O ~/stack/$(date +%s) --load-cookies ~/cookies.txt \ http://superuser.com/users/151431/terdon?tab=responses --2013-08-06 04:04:14-- http://superuser.com/users/151431/terdon?tab=responses Resolving superuser.com (superuser.com)... 18.104.22.168 Connecting to superuser.com (superuser.com)|22.214.171.124|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2013-08-06 04:04:15 ERROR 404: Not Found.
So, how can I automatically log in to an OpenID enabled website from the command line?
PS. I think this is better suited here than in webapps since my question is really about the command line aspect and not the actual details of the web page in question. I would guess that any solution will be applicable to all OpenID sites.