Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Forgive me for the simple question... I haven't used curl before. I'm trying to automatically download some data from this form: http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500

which posts to available_records_process.cfm. I would use:

curl -F "fromdate=1988-08-26;todate=2009-09-30;rtype=3;site_no=12040500" http://ida.water.usgs.gov/ida/available_records_process.cfm

but the webmaster has limited access to available_records_process.cfm, so I can't post to it directly (I get a 'specified URL could not be found' error). Next, I tried to use curl directly on the preceding page, like:

curl -F "fromdate=1988-08-26;todate=2009-09-30;rtype=3" http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500

But this doesn't seem to post the form, despite trying many variations of flags and post formatting. Any thoughts?

share|improve this question
add comment

1 Answer 1

You might need to use --data-urlencode but this will only work with POST requests. For GET requests you simply append the params separated by & to the URL, e.g. ...available_records.cfm?sn=12040500&fromdate=1988-08-26&todate=2009-09-30&rtype=3

share|improve this answer
    
Thanks for the tip. I'm still having trouble POSTing to the second .cfm (available_records_process). Using cookies seems to at least get a response: curl -d "site_no=12040500" -D mycookies http://ida.water.usgs.gov/ida/available_records.cfm and then curl -d "fromdate=1988-08-26&todate=2009-09-30&rtype=3&submit1:Retrieve+Data" -b mycookies http://ida.water.usgs.gov/ida/available_records_process.cfm Returns "JRun Servlet Error" rather than "URL not found". I'm still stuck on getting a valid POST through to it, though. –  Stochtastic Jan 21 '12 at 17:02
    
The key is to emulate exactly what the browser does. Use Firefox+Firebug or Chrome+F12(Network) to see exactly what the browser sends. Also you could try using WireShark to compare the requests sent by curl and by browser. –  ccpizza Jan 21 '12 at 19:13
    
Aha, digging through it with firebug did it for me. I'd forgotten to include --referer http://..., which was indeed included in the POST. Thanks! –  Stochtastic Jan 23 '12 at 22:33
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.