curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

0
votes
2answers
1k views

command line curl - file naming example.com/[1-10].txt => 01.txt through 10.txt

Is it possible to set the variable portion of the file name output to be a specific length and fill the extra spaces with zeros. I want things to be ordered for other users, so I would like curl ...
0
votes
1answer
371 views

Allow Incoming Responses Apache. On Ubuntu 11.10 - Curl

I'm trying to get a Curl Response from an outside server, however I noticed I cant neither PING the server in question nor connect to it. I tried disabling the iptables firewall but I had no success. ...
2
votes
1answer
2k views

Cannot connect to website - SSL handshaking fails

So I cannot connect to certain websites. Just a few, most are OK. The one I really care about is paypal.com. I have done the usual things. Let's see: Cleared browsing history Tried different ...
1
vote
1answer
3k views

How to stop running download process by CURL?

I forgot to use > filename after curl operation. linux console is giving contiguously garbage data(hopefully downloading and displaying the content on console). How can we stop this process (ctrl-c ...
2
votes
1answer
6k views

Downloading only images using curl or wget?

UPDATED: I've found using this Bash script fixes the problem of having GIF files with a .jpg extension. I'm attempting to download images from a forum who's url uses the following format: ...
1
vote
1answer
675 views

How to use delicious API from command line

I'm trying to add a delicious bookmark from a shell script, but keep getting the mysterious "something went wrong" error. I've checked the official spec and several examples and everything looks ...
1
vote
1answer
446 views

curl (7) permission denied on cgi script

Can't understand which permissions could be denied. Running cgi script from command line works fine. http_proxy="" curl http://url/ -o file Produces curl(7): permission denied (same issues with ...
2
votes
2answers
2k views

Can I encrypt a username password combo before using it with cURL?

I'm making a shell script that reached out to a web service with basic auth. I don't want to have to put in my username and password every time the script runs, but I also don't want to have the combo ...
1
vote
1answer
1k views

Submitting a form using curl

Forgive me for the simple question... I haven't used curl before. I'm trying to automatically download some data from this form: http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500 which ...
2
votes
0answers
654 views

Restore default CA Certificates on Mac OSX 10.7

I've been screwing around with my Mac, trying to install self-signed certificate as trusted for our WAN site (for which we can't justify paying for a real signed certificate) I've broken something ...
1
vote
1answer
1k views

How do I install hiphop-php in CentOS 6?

I've been trying to install hiphop-php on CentOS 6 with no luck. I found an .rpm for it but it fails dependency checks on boost which fails to install because it depends on curl which fails to ...
3
votes
1answer
326 views

Can cURL use a keyring for client certificates?

I'm using cURL to fetch from an HTTPS location, which requires a client certificate. I have my .curlrc set up to tell cURL where it is, but I don't want to leave my unencrypted private key just lying ...
3
votes
2answers
4k views

How to see HTTP request header by curl?

I can see HTTP header response by curl -I command. Is there any option to see the HTTP request header sent? # curl -I www.google.com HTTP/1.1 200 OK Date: Sat, 31 Dec 2011 00:55:53 GMT Expires: -1 ...
0
votes
1answer
14k views

How to install CURL on Debian Squeeze?

I just can't get CURL to install on Debian Squeeze... keeps giving me this: libtool: compile: gcc -DHAVE_CONFIG_H -I../include/curl -I../include -I../include -I../lib -I../lib -g0 -O2 ...
0
votes
2answers
2k views

How can I curl the output of another command

I want to pass curl the output from awk ./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | curl OUTPUT_FROM_AWK | grep myString How can I achieve this?!
1
vote
1answer
500 views

Scrape a site with curl/wget appending a parameter to each site

I need to scrape a website for all sites and append an Paramater, e.g. www.theSite.com/allpages?myPara=us Now what I got is wget --recursive to get all sites, but that doesn't append the parameter ...
1
vote
1answer
3k views

When using curl to http post a xml file to a url, how to set the parameter?

I want to use curl to http post a file to a url like: curl -d myfile.xml http://www.example.dom/post/file/here Can I set the parameter I want the file to be posted with? Like if it was a ...
23
votes
2answers
68k views

How to send POST with body, headers, and HTTP params using cURL?

I found a lot of examples on how to use simple POST commands in cURL, but I didn't find examples on how to send full HTTP POST commands, which contain: Headers (Basic Authentication) HTTP Params ...
0
votes
1answer
334 views

Only output curl result dependent on mime-type

Is it possible to make curl only save the output of curl -o if the file is of a certain mime-type? For example, I might use the following command to download a bunch of images off a server... curl ...
0
votes
1answer
636 views

No response from CURL at Windows Command Prompt

I've tried following Run cURL commands from Windows console but typing in any curl command returns nothing. See below - I just get back the command prompt. c:\curl>curl curl: try 'curl ...
0
votes
0answers
912 views

cURL does not work on verbose mode

I have installed curl on a new ubuntu installation with apt-get and it behaves really weird. When I do curl http://www.google.co.uk everything work ok, but when I do curl -v ...
41
votes
5answers
33k views

PowerShell equivalent of curl

Is there an equivalent of curl in PowerShell? Does it have some similar built-in capability or is there a 3rd party cmdlet?
1
vote
1answer
1k views

compiling curl with HipHop for PHP Patch [closed]

I'm following this guide: https://github.com/facebook/hiphop-php/wiki/Building-and-installing-on-ubuntu-10.10 It want's me to download curl source, doing the patch and compile this. well, but i get ...
1
vote
2answers
1k views

curl can't connect; wget can

trying to connect to my script on a remote server with curl but: col-pos-spd# curl -v http://***:80/monitor/reply.php * About to connect() to *** port 80 (#0) * Trying 83.xxx.xxx.2... Connection ...
0
votes
1answer
315 views

find xargs and curl give an error with -C- option

This command works fine find ~/mypath -name *.fstq -print|xargs curl -u sdd:dsdsd ftp://ftp.dsdd.nsdh.com/sdsds/ -T but when I add find ~/mypath -name *.fstq -print|xargs curl -C- -u sdd:dsdsd ...
0
votes
1answer
599 views

Using wget on a site with cgi

I need to mirror a website. However, each of the links on the site's webpage is actually a 'submit' to a cgi script that shows up the resulting page. AFAIK wget should fail on this since it needs ...
0
votes
1answer
1k views

Curl won't prompt me for a certificate password

I'm trying to use CURL to test simple HTTPS connections to servers that require a client certificate. I've specified the certificate type in my .curlrc file with cert = /path/to/Cert.p12 and told it ...
2
votes
1answer
1k views

Download a file from a website in command line in Linux

I need to download a file from a website which has a URL formatted like: http://www.server.com/downloads/1234/ This redirects to a .zip file which has to be saved. There is also a need to ...
0
votes
1answer
484 views

How to set downtime for any specific nagios host for certain time from commandline through curl?

I have a curl script which will do the service/host notification disable/enable from the command line through curl. But need to do that for scheduled downtime. Can someone suggest me. Okay it seems ...
1
vote
0answers
579 views

VirtualBox Ubuntu guest on OS X Lion host: Curl won't work

I'm trying to install some gems on an Ubuntu VM on OS X Lion using VirtualBox. I've narrowed my problem down to curl which is not responding at all. It seems to be going through some proxy local host ...
0
votes
2answers
1k views

Search for text in web pages given a list of URLs

I have a list of several thousand URLs, and I'd like to search each of these pages for a given word. How can I do this programmatically on Windows, preferably using VBScript or Powershell?
1
vote
2answers
90 views

Where to store short strings (with my key) on the internet?

Is there simple service to store strings under my key that can be used by bots? Requirements: Simple command line access, automatic posting allowed No need to keep some session with the service ...
3
votes
3answers
2k views

HTTP responses curl and wget different results

To check HTTP response header for a set of urls I send with curl the following request headers foreach ( $urls as $url ) { // Setup headers - I used the same headers from Firefox version 2.0.0.6 ...
27
votes
3answers
26k views

How to wget a file with correct name when redirected?

So after some time of searching on google and superuser (and scanning man pages) I was unable to find an answer to something that (I think) should be simple: If you go here: ...
15
votes
8answers
33k views

wget/cURL alternative native to Windows?

Is there a wget or cURL type command line utility native to Windows Vista? How does it work?
2
votes
2answers
1k views

curl not uploading file to ftp

I am using curl to upload a file to ftp server. The same script is working on another system, but on this one after connecting to ftp it is not doing anything and connection times out. The detailed ...
2
votes
1answer
23k views

How to install curl on ubuntu

System: ibook G4 / linux ubuntu 2.6.28-6-powerpc OS : ubuntu 9.04(jaunty) PHP ver : 5.2.6-3ubuntu4.6 Please let me know how to install curl on my system. Synaptic Package Manager tells me to ...
0
votes
1answer
379 views

Can I set a default hostname for curl?

I would like to reduce the keystrokes when I use curl to test web services on my workstation. Is there a way to set a default hostname of 'localhost:3000' into ~/.curlrc so the following will work? ...
36
votes
2answers
34k views

How do you display POST data with cURL?

As an example, POSTing to a web server with the -v argument: curl -v http://testserver.com/post -d "firstname=john&lastname=doe" And the output > POST /post HTTP/1.1 > User-Agent: ...
0
votes
2answers
360 views

remote backup with curl/wget

I need to backup a clients server (source) from our mediatemple server (destination) and then add it into a git repository while ssh'd in. What is the most efficient/fastest way to do so? wget? or ...
2
votes
1answer
1k views

using cURL to download all files in certain folder, except 404 error page?

I'm using cURL in ubuntu to download some files like pic001.jpg pic002.jpg pic003.jpg ... pic999.jpg While some files may be missing from this sequence but when I just use ...
1
vote
1answer
891 views

Load multiple files simulatenously with cURL

I am looking to download multiple files from a server using cURL from the Terminal on a Mac. I know that using the {} operator, I can load one file after another but I am looking to download all (10) ...
0
votes
2answers
862 views

Formatting cURL Output in the Windows Terminal

When I try to perform the cURL request here, Windows 7 (x64) returns an output with no line breaks: C:\Users\kiwi>curl http://www.documentcloud.org/api/search.json?q=group:nytimes ...
99
votes
6answers
107k views

Getting curl to output HTTP status code?

I'm using curl at the command line on Linux to issue HTTP requests. The response bodies are printed to standard out, which is fine, but I can't see from the man page how to get curl to print the ...
2
votes
2answers
1k views

Curl alternative on windows to get header information [duplicate]

Possible Duplicate: wget/cURL alternative native to Windows? Is there any tool (command line tool preferred) to get the HTTP request header information on Windows machine like Curl does on ...
1
vote
2answers
4k views

curlftpfs doesn't work for a username with a “@”

My hosting company makes all my usernames with a "@" in them. curlftpfs user="user@domain.com:pass" ftp://ftp.domain.com/ ~/domain/ For some reason I get in response Error connecting to ftp: ...
4
votes
4answers
20k views

Downloading file from FTP using cURL

I'm trying to use a cURL command to download a file from an FTP server to a local drive on my computer. I've tried curl "ftp://myftpsite" --user name:password -Q "CWD /users/myfolder/" -O ...
1
vote
1answer
3k views

How do I setup curl / wget with FTP proxy?

I'm having a good deal of trouble getting curl or wget to download files from FTP servers through an FTP proxy. I have setup GUI FTP programs to work... gFTP works with these settings: I get ...
0
votes
2answers
4k views

How to download a file behind authentication via Wget or cURL [duplicate]

Possible Duplicate: Wget a page that requires logging in I wish to download file from http://mbr.nlm.nih.gov/Download/MetaMapped_Medline/2011/. However, in order to download the file, I ...
2
votes
2answers
797 views

bash: source from URL

In addition to my own computer, I sometimes use an Ubuntu cluster at my school. Rather than manually keep my .bashrc's in sync, I would like to make the school cluster's .bashrc source my personal ...