Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Forgive me for the simple question... I haven't used curl before. I'm trying to automatically download some data from this form: http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500

which posts to available_records_process.cfm. I would use:

curl -F "fromdate=1988-08-26;todate=2009-09-30;rtype=3;site_no=12040500" http://ida.water.usgs.gov/ida/available_records_process.cfm

but the webmaster has limited access to available_records_process.cfm, so I can't post to it directly (I get a 'specified URL could not be found' error). Next, I tried to use curl directly on the preceding page, like:

curl -F "fromdate=1988-08-26;todate=2009-09-30;rtype=3" http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500

But this doesn't seem to post the form, despite trying many variations of flags and post formatting. Any thoughts?

share|improve this question

You might need to use --data-urlencode but this will only work with POST requests. For GET requests you simply append the params separated by & to the URL, e.g. ...available_records.cfm?sn=12040500&fromdate=1988-08-26&todate=2009-09-30&rtype=3

share|improve this answer
    
Thanks for the tip. I'm still having trouble POSTing to the second .cfm (available_records_process). Using cookies seems to at least get a response: curl -d "site_no=12040500" -D mycookies http://ida.water.usgs.gov/ida/available_records.cfm and then curl -d "fromdate=1988-08-26&todate=2009-09-30&rtype=3&submit1:Retrieve+Data" -b mycookies http://ida.water.usgs.gov/ida/available_records_process.cfm Returns "JRun Servlet Error" rather than "URL not found". I'm still stuck on getting a valid POST through to it, though. – Stochtastic Jan 21 '12 at 17:02
    
The key is to emulate exactly what the browser does. Use Firefox+Firebug or Chrome+F12(Network) to see exactly what the browser sends. Also you could try using WireShark to compare the requests sent by curl and by browser. – ccpizza Jan 21 '12 at 19:13
    
Aha, digging through it with firebug did it for me. I'd forgotten to include --referer http://..., which was indeed included in the POST. Thanks! – Stochtastic Jan 23 '12 at 22:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .