curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

1
vote
1answer
1k views

How do I install hiphop-php in CentOS 6?

I've been trying to install hiphop-php on CentOS 6 with no luck. I found an .rpm for it but it fails dependency checks on boost which fails to install because it depends on curl which fails to ...
3
votes
1answer
304 views

Can cURL use a keyring for client certificates?

I'm using cURL to fetch from an HTTPS location, which requires a client certificate. I have my .curlrc set up to tell cURL where it is, but I don't want to leave my unencrypted private key just lying ...
3
votes
2answers
4k views

How to see HTTP request header by curl?

I can see HTTP header response by curl -I command. Is there any option to see the HTTP request header sent? # curl -I www.google.com HTTP/1.1 200 OK Date: Sat, 31 Dec 2011 00:55:53 GMT Expires: -1 ...
0
votes
1answer
14k views

How to install CURL on Debian Squeeze?

I just can't get CURL to install on Debian Squeeze... keeps giving me this: libtool: compile: gcc -DHAVE_CONFIG_H -I../include/curl -I../include -I../include -I../lib -I../lib -g0 -O2 ...
0
votes
2answers
2k views

How can I curl the output of another command

I want to pass curl the output from awk ./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | curl OUTPUT_FROM_AWK | grep myString How can I achieve this?!
1
vote
1answer
485 views

Scrape a site with curl/wget appending a parameter to each site

I need to scrape a website for all sites and append an Paramater, e.g. www.theSite.com/allpages?myPara=us Now what I got is wget --recursive to get all sites, but that doesn't append the parameter ...
1
vote
1answer
2k views

When using curl to http post a xml file to a url, how to set the parameter?

I want to use curl to http post a file to a url like: curl -d myfile.xml http://www.example.dom/post/file/here Can I set the parameter I want the file to be posted with? Like if it was a ...
21
votes
2answers
63k views

How to send POST with body data, headers, and HTTP params over the cURL command line?

I found a lot of examples on how to use simple POST commands in cURL, but I didn't find examples on how to send full HTTP POST commands, which contain: Headers (Basic Authentication) HTTP Params ...
0
votes
1answer
319 views

Only output curl result dependent on mime-type

Is it possible to make curl only save the output of curl -o if the file is of a certain mime-type? For example, I might use the following command to download a bunch of images off a server... curl ...
0
votes
1answer
600 views

No response from CURL at Windows Command Prompt

I've tried following Run cURL commands from Windows console but typing in any curl command returns nothing. See below - I just get back the command prompt. c:\curl>curl curl: try 'curl ...
0
votes
0answers
867 views

cURL does not work on verbose mode

I have installed curl on a new ubuntu installation with apt-get and it behaves really weird. When I do curl http://www.google.co.uk everything work ok, but when I do curl -v ...
37
votes
5answers
29k views

PowerShell equivalent of curl

Is there an equivalent of curl in PowerShell? Does it have some similar built-in capability or is there a 3rd party cmdlet?
1
vote
1answer
1k views

compiling curl with HipHop for PHP Patch [closed]

I'm following this guide: https://github.com/facebook/hiphop-php/wiki/Building-and-installing-on-ubuntu-10.10 It want's me to download curl source, doing the patch and compile this. well, but i get ...
1
vote
2answers
1k views

curl can't connect; wget can

trying to connect to my script on a remote server with curl but: col-pos-spd# curl -v http://***:80/monitor/reply.php * About to connect() to *** port 80 (#0) * Trying 83.xxx.xxx.2... Connection ...
0
votes
1answer
303 views

find xargs and curl give an error with -C- option

This command works fine find ~/mypath -name *.fstq -print|xargs curl -u sdd:dsdsd ftp://ftp.dsdd.nsdh.com/sdsds/ -T but when I add find ~/mypath -name *.fstq -print|xargs curl -C- -u sdd:dsdsd ...
0
votes
1answer
584 views

Using wget on a site with cgi

I need to mirror a website. However, each of the links on the site's webpage is actually a 'submit' to a cgi script that shows up the resulting page. AFAIK wget should fail on this since it needs ...
0
votes
1answer
991 views

Curl won't prompt me for a certificate password

I'm trying to use CURL to test simple HTTPS connections to servers that require a client certificate. I've specified the certificate type in my .curlrc file with cert = /path/to/Cert.p12 and told it ...
2
votes
1answer
1k views

Download a file from a website in command line in Linux

I need to download a file from a website which has a URL formatted like: http://www.server.com/downloads/1234/ This redirects to a .zip file which has to be saved. There is also a need to ...
0
votes
1answer
469 views

How to set downtime for any specific nagios host for certain time from commandline through curl?

I have a curl script which will do the service/host notification disable/enable from the command line through curl. But need to do that for scheduled downtime. Can someone suggest me. Okay it seems ...
1
vote
0answers
562 views

VirtualBox Ubuntu guest on OS X Lion host: Curl won't work

I'm trying to install some gems on an Ubuntu VM on OS X Lion using VirtualBox. I've narrowed my problem down to curl which is not responding at all. It seems to be going through some proxy local host ...
0
votes
2answers
1k views

Search for text in web pages given a list of URLs

I have a list of several thousand URLs, and I'd like to search each of these pages for a given word. How can I do this programmatically on Windows, preferably using VBScript or Powershell?
1
vote
2answers
89 views

Where to store short strings (with my key) on the internet?

Is there simple service to store strings under my key that can be used by bots? Requirements: Simple command line access, automatic posting allowed No need to keep some session with the service ...
3
votes
3answers
2k views

HTTP responses curl and wget different results

To check HTTP response header for a set of urls I send with curl the following request headers foreach ( $urls as $url ) { // Setup headers - I used the same headers from Firefox version 2.0.0.6 ...
25
votes
3answers
24k views

How to wget a file with correct name when redirected?

So after some time of searching on google and superuser (and scanning man pages) I was unable to find an answer to something that (I think) should be simple: If you go here: ...
14
votes
8answers
31k views

wget/cURL alternative native to Windows?

Is there a wget or cURL type command line utility native to Windows Vista? How does it work?
2
votes
2answers
1k views

curl not uploading file to ftp

I am using curl to upload a file to ftp server. The same script is working on another system, but on this one after connecting to ftp it is not doing anything and connection times out. The detailed ...
2
votes
1answer
22k views

How to install curl on ubuntu

System: ibook G4 / linux ubuntu 2.6.28-6-powerpc OS : ubuntu 9.04(jaunty) PHP ver : 5.2.6-3ubuntu4.6 Please let me know how to install curl on my system. Synaptic Package Manager tells me to ...
0
votes
1answer
368 views

Can I set a default hostname for curl?

I would like to reduce the keystrokes when I use curl to test web services on my workstation. Is there a way to set a default hostname of 'localhost:3000' into ~/.curlrc so the following will work? ...
35
votes
3answers
31k views

How do you display POST data with cURL?

As an example, POSTing to a web server with the -v argument: curl -v http://testserver.com/post -d "firstname=john&lastname=doe" And the output > POST /post HTTP/1.1 > User-Agent: ...
0
votes
2answers
357 views

remote backup with curl/wget

I need to backup a clients server (source) from our mediatemple server (destination) and then add it into a git repository while ssh'd in. What is the most efficient/fastest way to do so? wget? or ...
2
votes
1answer
1k views

using cURL to download all files in certain folder, except 404 error page?

I'm using cURL in ubuntu to download some files like pic001.jpg pic002.jpg pic003.jpg ... pic999.jpg While some files may be missing from this sequence but when I just use ...
1
vote
1answer
865 views

Load multiple files simulatenously with cURL

I am looking to download multiple files from a server using cURL from the Terminal on a Mac. I know that using the {} operator, I can load one file after another but I am looking to download all (10) ...
0
votes
2answers
825 views

Formatting cURL Output in the Windows Terminal

When I try to perform the cURL request here, Windows 7 (x64) returns an output with no line breaks: C:\Users\kiwi>curl http://www.documentcloud.org/api/search.json?q=group:nytimes ...
91
votes
6answers
91k views

Getting curl to output HTTP status code?

I'm using curl at the command line on Linux to issue HTTP requests. The response bodies are printed to standard out, which is fine, but I can't see from the man page how to get curl to print the ...
2
votes
2answers
1k views

Curl alternative on windows to get header information [duplicate]

Possible Duplicate: wget/cURL alternative native to Windows? Is there any tool (command line tool preferred) to get the HTTP request header information on Windows machine like Curl does on ...
1
vote
2answers
3k views

curlftpfs doesn't work for a username with a “@”

My hosting company makes all my usernames with a "@" in them. curlftpfs user="user@domain.com:pass" ftp://ftp.domain.com/ ~/domain/ For some reason I get in response Error connecting to ftp: ...
4
votes
4answers
19k views

Downloading file from FTP using cURL

I'm trying to use a cURL command to download a file from an FTP server to a local drive on my computer. I've tried curl "ftp://myftpsite" --user name:password -Q "CWD /users/myfolder/" -O ...
1
vote
1answer
2k views

How do I setup curl / wget with FTP proxy?

I'm having a good deal of trouble getting curl or wget to download files from FTP servers through an FTP proxy. I have setup GUI FTP programs to work... gFTP works with these settings: I get ...
0
votes
2answers
4k views

How to download a file behind authentication via wget or curl [duplicate]

Possible Duplicate: WGet a Page that Requires Logging in I wish to download file from http://mbr.nlm.nih.gov/Download/MetaMapped_Medline/2011/ however in order to download the file, i need ...
2
votes
2answers
755 views

bash: source from URL

In addition to my own computer, I sometimes use an Ubuntu cluster at my school. Rather than manually keep my .bashrc's in sync, I would like to make the school cluster's .bashrc source my personal ...
6
votes
2answers
2k views

How can I pass my Windows credentials in cURL?

I'm writing a script that uses cURL to pull data from an internal web server. I can explicitly use --user combined with --ntlm and everything works well, but I can't get it to recognize whomever's ...
4
votes
1answer
3k views

Where's the ca cert bundle on OSX?

I'm debugging some curl ssl configuration stuff and was wondering where the ca cert bundle is installed on OSX.
8
votes
1answer
19k views

cURL Upload file and send POST data

I have a web server running some PHP that checks for an image (curl -F 'imageName=@myimage') and it also checks the POST data for username=&password=. When the PHP checks _REQUEST I cannot just ...
0
votes
2answers
865 views

How to escape “@” in the username when logging in through FTPES with curl?

$ curl -T "index.html" -k --ftp-ssl -u "MYUSER@MYDOMAIN.COM" MYDOMAIN.COM Enter host password for user 'MYUSER@MYDOMAIN.COM': % Total % Received % Xferd Average Speed Time Time Time ...
0
votes
2answers
7k views

curl windows command line [duplicate]

Possible Duplicate: Run cURL commands from Windows console I saw this questions I downloaded the CURL and placed it under C. Still I can't run it from CMD (with or without full path). as ...
0
votes
3answers
2k views

Wget hangs in the middle of get request

Sometimes wget or curl randomly hangs in the middle of get request in Ubuntu 10.10: curl "http://api.indeed.com/ads/apisearch?publisher=7570038743238473" wget ...
1
vote
0answers
201 views

how to use to proxy in the intranet?

i have an intranet site which is inside only our company and feeds information. i download an weather API from http://www.foreca.com/Philippines/ and hook to my site. the API has a search engine and ...
1
vote
2answers
258 views

Where does curl store data whilest it's still downloading

Hi I'm using curl on the command line to automate some downloading of many files from a server . However I'm wondering where curl stores it's data whilst it's downloading? The reason I need this ...
12
votes
6answers
45k views

CURL to download a directory

I am trying to download a full website directory using CURL. The following command does not work: curl -LO http://example.com/ It returns an error: curl: Remote file name has no length!. But when ...
3
votes
3answers
725 views

What's the equivalent of “wget -i FILE” in Curl?

What's the equivalent of "wget -i FILE" in Curl?