curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

1528
votes
6answers
1.5m views

What is the cURL command-line syntax to do a POST request?

How can I make a POST request with the cURL command-line tool?
220
votes
10answers
240k views

Getting curl to output HTTP status code?

I'm using curl at the command line on Linux to issue HTTP requests. The response bodies are printed to standard out, which is fine, but I can't see from the man page how to get curl to print the ...
63
votes
7answers
159k views

Run cURL commands from Windows console

Is there a way to install cURL in windows in order to run cURL commands from the command prompt?
61
votes
6answers
65k views

PowerShell equivalent of curl

Is there an equivalent of curl in PowerShell? Does it have some similar built-in capability or is there a 3rd party cmdlet?
57
votes
2answers
60k views

How do you display POST data with cURL?

As an example, POSTing to a web server with the -v argument: curl -v http://testserver.com/post -d "firstname=john&lastname=doe" And the output > POST /post HTTP/1.1 > User-Agent: curl/7....
43
votes
2answers
20k views

Can I make cURL fail with an exitCode different than 0 if the HTTP status code is not 200?

I was always assuming that when curl got an HTTP 500 response it was returning an exit code that meant failure (!= 0), but that seems to be not the case. Is there a way I can I make cURL fail with an ...
42
votes
4answers
42k views

How to wget a file with correct name when redirected?

So after some time of searching on Google and Super User (and scanning man pages) I was unable to find an answer to something that (I think) should be simple: If you go here: http://www.vim.org/...
31
votes
1answer
112k views

How to send POST with body, headers, and HTTP params using cURL?

I found a lot of examples on how to use simple POST commands in cURL, but I didn't find examples on how to send full HTTP POST commands, which contain: Headers (Basic Authentication) HTTP Params (s=...
27
votes
3answers
15k views

Curl local host names on Mac OS X Yosemite

I just upgraded from Mavericks to Yosemite, and now curl can't see loopback host names. Set up a simple http server to test: $ python -m SimpleHTTPServer Serving HTTP on 0.0.0.0 port 8000 ... Now ...
26
votes
8answers
58k views

Wget/cURL alternative native to Windows?

Is there a Wget or cURL type command line utility native to Windows Vista? How does it work?
24
votes
4answers
32k views

cURL: how to suppress strange output when redirecting?

I'm trying to print just the verbose sections of a cURL request (which are sent to stderr) from the bash shell. But when I redirect stdout like this: curl -v http://somehost/somepage > /dev/null ...
22
votes
3answers
13k views

Is “curl -u username:password http://example.com” secure?

Is curl -u username:password http://example.com secure? If not, can you give a brief explanation of how someone could obtain your password?
20
votes
2answers
10k views

Can cURL send requests to sockets?

I can find nothing about this on Google or the cURL man pages. I have a HTTP server running at /var/run/my-server.sock, and I want to test it by sending a simple request using cURL. Can this be done ...
16
votes
6answers
69k views

CURL to download a directory

I am trying to download a full website directory using CURL. The following command does not work: curl -LO http://example.com/ It returns an error: curl: Remote file name has no length!. But when ...
15
votes
1answer
16k views

Why can't cURL properly verify a certificate on Windows?

When I try to use Curl on windows, to retrieve an https url, I get the dreaded "connection error (60)." The exact error message is: curl: (60) SSL certificate problem, verify that the CA cert ...
14
votes
3answers
18k views

how to use curl to verify if a site's certificate has been revoked?

To check if the certificate for google.com has been revoked, I tried the following command: curl https://www.google.com --cacert GeoTrust_Global_CA.pem --crlfile gtglobal.pem -v , but I got the ...
11
votes
2answers
6k views

How can I pass my Windows credentials in cURL?

I'm writing a script that uses cURL to pull data from an internal web server. I can explicitly use --user combined with --ntlm and everything works well, but I can't get it to recognize whomever's ...
10
votes
4answers
4k views

How “legal” is site-scraping using cURL? [closed]

Recently I was experimenting with the cURL, and I found lot is possible with it. I built a small script that crawls a musical site, which plays online songs. On the way of my experiment, I found that ...
10
votes
2answers
8k views

How can I use curl with ::1 for ipv6 based loopback?

I tried a few ways but they seem to be confusing curl. root@testt:~# curl localhost:8080 <h1>Hello world!</h1> <p>Boom Bam Splat</p> root@testt:~# curl ::1:8080 curl: (3) ...
9
votes
2answers
2k views

How to download parts of the same file from different sources with curl/wget?

I have a quite large file hosted on five different servers. I would like to be able to download different parts of the file from each server and subsequently concatenate the parts, in order to ...
9
votes
2answers
26k views

How to get and use certificates with curl

1) Entering the following command curl -k email@mail.com:password https://www.example.com Returns the following error <HTML><HEAD><meta http-equiv="content-type" content="text/...
8
votes
2answers
12k views

POSTing a file's contents with CURL

(The following is backstory, you can jump down to "Question:" if you'd like) There's a webpage for one of my courses that accepts input text (expected to be Python code) and then gives some tokenized ...
8
votes
4answers
32k views

Downloading file from FTP using cURL

I'm trying to use a cURL command to download a file from an FTP server to a local drive on my computer. I've tried curl "ftp://myftpsite" --user name:password -Q "CWD /users/myfolder/" -O "myfile....
8
votes
1answer
21k views

cURL Upload file and send POST data

I have a web server running some PHP that checks for an image (curl -F 'imageName=@myimage') and it also checks the POST data for username=&password=. When the PHP checks _REQUEST I cannot just ...
8
votes
2answers
675 views

How can I use a command line tool like Wget to log into an OpenID site?

Specifically, I would like to be able to download certain pages from my user profile on the various Stack Exchange sites. I would, however, like to do this automatically (using a cron job), from ...
7
votes
2answers
6k views

Persistent retrying resuming downloads with curl

I'm on a mac and have a list of files I would like to download from an ftp server. The connection is a bit buggy so I want it to retry and resume if connection is dropped. I know I can do this with ...
6
votes
1answer
4k views

How do I remotely fetch files from redirected URLs from a terminal?

I want to fetch a tarball of this python library from terminal. https://github.com/simplegeo/python-oauth2/downloads However, I cannot simply call wget https://github.com/simplegeo/python-oauth2/...
6
votes
1answer
1k views

How did Google get on my Mac?

I am running a MacBook Pro, and have never installed Chrome, Google Earth, or anything blatantly Google. Just installed Little Snitch (are there no good free firewalls for Mac?) and see that CURL is ...
6
votes
2answers
6k views

How to see HTTP request header by curl?

I can see HTTP header response by curl -I command. Is there any option to see the HTTP request header sent? # curl -I www.google.com HTTP/1.1 200 OK Date: Sat, 31 Dec 2011 00:55:53 GMT Expires: -1 ...
6
votes
1answer
6k views

How can I a download private repository from GitHub via when I have no access to ‘git’ on my local machine?

What I want to do is to download private repository archive from GitHub, extract it, remove archive file and copy some directories that are inside downloaded project. I tried to use wget but I cannot ...
6
votes
2answers
7k views

Specifying minor TLS version when using curl

Is there a way to specify curl to use a specific TLS version? Like 1.1 or 1.2? I can see only sslv3 and tlsv1 options in command help. I took latest src and compiled it with openssl 1.0.1e. Still dont ...
5
votes
2answers
265 views

Firebug and cURL

I have been using Firebug and cURL for quite a while. Firebug is amazing to capture HTTPS POST request as opposed to a network analyser as it is aware of SSL/TLS negotiation. And cURL is amazing to ...
5
votes
1answer
12k views

CURL File with post parameter

How to make a request for uploading a file to facebook using graph api in curl I am making this request but getting Error: curl https://graph.facebook.com/<id>/photos -F "source=@me.jpg" -d "...
5
votes
1answer
3k views

Why does sudo not work with curl?

I ran a command to download something with curl and it threw a permission denied error. So I did sudo curl and I got the same error... Why is that? I was able to run the install command without ...
5
votes
1answer
2k views

cURL hangs trying to upload file from stdin

I'm trying to PUT a file with cURL. This hangs: curl -vvv --digest -u user -T - https://example.com/file.txt < file This does not: curl -vvv --digest -u user -T file https://example.com/file.txt ...
4
votes
3answers
3k views

Downloading XCode using wget or curl [closed]

Is there a way for downloading XCode using curl or wget? EDIT: Why is the following curl snippet not working? First, I login to Apple's site: curl -d "theAccountName=USERNAME&theAccountPW=PASS" ...
4
votes
3answers
3k views

HTTP responses curl and wget different results

To check HTTP response header for a set of urls I send with curl the following request headers foreach ( $urls as $url ) { // Setup headers - I used the same headers from Firefox version 2.0.0.6 ...
4
votes
1answer
1k views

What units does curl use for bandwidth?

curl on the command-line displays progress like this: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent ...
4
votes
1answer
1k views

How to make alias/function in .bash_profile for `curl` that takes an argument?

I found this tip super helpful, for using curl to resume an interrupted file copy. The full syntax is: curl -C - -O file:///Volumes/path/to/file I would like to add an alias or function called ...
4
votes
1answer
309 views

Automatic login to Vodafone Community WiFi (Fon) via Unix script

First I have to say that my question is similar to this one I am using a Vodafone Community Fon AP and it often disconnects. In OS X, which I'm using, there is a tool called Captive Network Assistant ...
4
votes
1answer
509 views

Why is there a very long 6 second delay in start of Curl

I did a full install of Cygwin on my Windows 7 x64 machine. After each machine reboot, Curl can take around 6 seconds to start (even if no command line options were specified). On subsequent uses of ...
4
votes
3answers
16k views

URL to reboot a WebSTAR DPC2100R2 cable modem with curl?

Does anyone know what the magic URL is for rebooting a WebSTAR DPC2100R2 with curl? I used to have a SurfBoard, and the curl command: curl http://192.168.100.1/reset.htm?reset_modem=Restart%20Cable%...
4
votes
2answers
257 views

Get curl to output non-2xx to stderr, but 2xxs to a file

I would like some curl command, possibly with some bash hackery, that: Outputs any 2xx responses to a file Outputs any non-2xx responses to stderr Exits with a nonzero status code when a non-2xx ...
4
votes
1answer
4k views

Where's the ca cert bundle on OSX?

I'm debugging some curl ssl configuration stuff and was wondering where the ca cert bundle is installed on OSX.
3
votes
2answers
2k views

Curl: Difference between 2 ssl commands

I cant tell the difference or when to use one over the other. (Should this be on serverfault?) -E/--cert (SSL) Tells curl to use the specified certificate file when getting a file with ...
3
votes
2answers
3k views

Why sudo curl ignores proxy settings?

$ echo $http_proxy http://my.proxy.com $ curl -v http://files.com/a.txt * About to connect() to proxy my.proxy.com # Correct downloading $ sudo echo $http_proxy http://my.proxy.com $sudo curl -v ...
3
votes
2answers
1k views

bash: source from URL

In addition to my own computer, I sometimes use an Ubuntu cluster at my school. Rather than manually keep my .bashrc's in sync, I would like to make the school cluster's .bashrc source my personal ....
3
votes
2answers
171 views

curl check if file is newer and instead of downloading - execute a bash (or python) script

I am having a bit of an issue. I have a file, hosted on a remote server (http://mywebsite/file.zip). I also have a few embedded linux boxes (running openelec OS). The boxes have fairly limited ...
3
votes
1answer
771 views

Why does “curl” command work normally while “ping” not?

I find a strange phenomenon in my corporate network: I can execute curl google.com command successfully: [root@localhost ~]# curl google.com <HTML><HEAD><meta http-equiv="content-...
3
votes
1answer
2k views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl >>...