curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

1
vote
1answer
501 views

How to use delicious API from command line

I'm trying to add a delicious bookmark from a shell script, but keep getting the mysterious "something went wrong" error. I've checked the official spec and several examples and everything looks ...
3
votes
1answer
2k views

VPN routing issue in linux- how to make applications that can only bind to IP addresses work?

Right now I have a PPTP VPN set up on my remote ubuntu VPS. I would like to have certain applications use the VPN to access the internet (rtorrent, irssi, among others). Currently, it is set up like ...
2
votes
1answer
2k views

Cannot connect to website - SSL handshaking fails

So I cannot connect to certain websites. Just a few, most are OK. The one I really care about is paypal.com. I have done the usual things. Let's see: Cleared browsing history Tried different ...
3
votes
1answer
876 views

Force curl to download an empty file

In a bash script, I am using curl command on linux to download files from a FTP server. The command I am using is the following (the information about the FTP server connexion is stored in a .netrc ...
0
votes
1answer
282 views

Allow Incoming Responses Apache. On Ubuntu 11.10 - Curl

I'm trying to get a Curl Response from an outside server, however I noticed I cant neither PING the server in question nor connect to it. I tried disabling the iptables firewall but I had no success. ...
1
vote
1answer
2k views

How to stop running download process by CURL?

I forgot to use > filename after curl operation. linux console is giving contiguously garbage data(hopefully downloading and displaying the content on console). How can we stop this process (ctrl-c ...
0
votes
1answer
338 views

curl (7) permission denied on cgi script

Can't understand which permissions could be denied. Running cgi script from command line works fine. http_proxy="" curl http://url/ -o file Produces curl(7): permission denied (same issues with ...
1
vote
2answers
1k views

Can I encrypt a username password combo before using it with cURL?

I'm making a shell script that reached out to a web service with basic auth. I don't want to have to put in my username and password every time the script runs, but I also don't want to have the combo ...
7
votes
2answers
1k views

How to download parts of the same file from different sources with curl/wget?

I have a quite large file hosted on five different servers. I would like to be able to download different parts of the file from each server and subsequently concatenate the parts, in order to ...
21
votes
2answers
17k views

How do you display POST data with cURL?

As an example, POSTing to a web server with the -v argument: curl -v http://testserver.com/post -d "firstname=john&lastname=doe" And the output > POST /post HTTP/1.1 > User-Agent: ...
1
vote
1answer
958 views

Submitting a form using curl

Forgive me for the simple question... I haven't used curl before. I'm trying to automatically download some data from this form: http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500 which ...
2
votes
0answers
456 views

Restore default CA Certificates on Mac OSX 10.7

I've been screwing around with my Mac, trying to install self-signed certificate as trusted for our WAN site (for which we can't justify paying for a real signed certificate) I've broken something ...
0
votes
1answer
273 views

Only output curl result dependent on mime-type

Is it possible to make curl only save the output of curl -o if the file is of a certain mime-type? For example, I might use the following command to download a bunch of images off a server... curl ...
3
votes
1answer
235 views

Can cURL use a keyring for client certificates?

I'm using cURL to fetch from an HTTPS location, which requires a client certificate. I have my .curlrc set up to tell cURL where it is, but I don't want to leave my unencrypted private key just lying ...
0
votes
1answer
716 views

Curl won't prompt me for a certificate password

I'm trying to use CURL to test simple HTTPS connections to servers that require a client certificate. I've specified the certificate type in my .curlrc file with cert = /path/to/Cert.p12 and told it ...
3
votes
2answers
2k views

How to see HTTP request header by curl?

I can see HTTP header response by curl -I command. Is there any option to see the HTTP request header sent? # curl -I www.google.com HTTP/1.1 200 OK Date: Sat, 31 Dec 2011 00:55:53 GMT Expires: -1 ...
0
votes
1answer
11k views

How to install CURL on Debian Squeeze?

I just can't get CURL to install on Debian Squeeze... keeps giving me this: libtool: compile: gcc -DHAVE_CONFIG_H -I../include/curl -I../include -I../include -I../lib -I../lib -g0 -O2 ...
0
votes
2answers
1k views

How can I curl the output of another command

I want to pass curl the output from awk ./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | curl OUTPUT_FROM_AWK | grep myString How can I achieve this?!
1
vote
1answer
399 views

Scrape a site with curl/wget appending a parameter to each site

I need to scrape a website for all sites and append an Paramater, e.g. www.theSite.com/allpages?myPara=us Now what I got is wget --recursive to get all sites, but that doesn't append the parameter ...
0
votes
1answer
419 views

No response from CURL at Windows Command Prompt

I've tried following Run cURL commands from Windows console but typing in any curl command returns nothing. See below - I just get back the command prompt. c:\curl>curl curl: try 'curl ...
0
votes
0answers
541 views

cURL does not work on verbose mode

I have installed curl on a new ubuntu installation with apt-get and it behaves really weird. When I do curl http://www.google.co.uk everything work ok, but when I do curl -v ...
1
vote
1answer
2k views

How do I setup curl / wget with FTP proxy?

I'm having a good deal of trouble getting curl or wget to download files from FTP servers through an FTP proxy. I have setup GUI FTP programs to work... gFTP works with these settings: I get ...
1
vote
1answer
1k views

compiling curl with HipHop for PHP Patch [closed]

I'm following this guide: https://github.com/facebook/hiphop-php/wiki/Building-and-installing-on-ubuntu-10.10 It want's me to download curl source, doing the patch and compile this. well, but i get ...
1
vote
1answer
759 views

curl can't connect; wget can

trying to connect to my script on a remote server with curl but: col-pos-spd# curl -v http://***:80/monitor/reply.php * About to connect() to *** port 80 (#0) * Trying 83.xxx.xxx.2... Connection ...
0
votes
1answer
229 views

find xargs and curl give an error with -C- option

This command works fine find ~/mypath -name *.fstq -print|xargs curl -u sdd:dsdsd ftp://ftp.dsdd.nsdh.com/sdsds/ -T but when I add find ~/mypath -name *.fstq -print|xargs curl -C- -u sdd:dsdsd ...
0
votes
1answer
462 views

Using wget on a site with cgi

I need to mirror a website. However, each of the links on the site's webpage is actually a 'submit' to a cgi script that shows up the resulting page. AFAIK wget should fail on this since it needs ...
1
vote
2answers
3k views

curlftpfs doesn't work for a username with a “@”

My hosting company makes all my usernames with a "@" in them. curlftpfs user="user@domain.com:pass" ftp://ftp.domain.com/ ~/domain/ For some reason I get in response Error connecting to ftp: ...
2
votes
1answer
821 views

Download a file from a website in command line in Linux

I need to download a file from a website which has a URL formatted like: http://www.server.com/downloads/1234/ This redirects to a .zip file which has to be saved. There is also a need to ...
0
votes
1answer
344 views

How to set downtime for any specific nagios host for certain time from commandline through curl?

I have a curl script which will do the service/host notification disable/enable from the command line through curl. But need to do that for scheduled downtime. Can someone suggest me. Okay it seems ...
1
vote
0answers
511 views

VirtualBox Ubuntu guest on OS X Lion host: Curl won't work

I'm trying to install some gems on an Ubuntu VM on OS X Lion using VirtualBox. I've narrowed my problem down to curl which is not responding at all. It seems to be going through some proxy local host ...
0
votes
2answers
925 views

Search for text in web pages given a list of URLs

I have a list of several thousand URLs, and I'd like to search each of these pages for a given word. How can I do this programmatically on Windows, preferably using VBScript or Powershell?
1
vote
2answers
222 views

Where does curl store data whilest it's still downloading

Hi I'm using curl on the command line to automate some downloading of many files from a server . However I'm wondering where curl stores it's data whilst it's downloading? The reason I need this ...
2
votes
3answers
1k views

HTTP responses curl and wget different results

To check HTTP response header for a set of urls I send with curl the following request headers foreach ( $urls as $url ) { // Setup headers - I used the same headers from Firefox version 2.0.0.6 ...
0
votes
1answer
322 views

Can I set a default hostname for curl?

I would like to reduce the keystrokes when I use curl to test web services on my workstation. Is there a way to set a default hostname of 'localhost:3000' into ~/.curlrc so the following will work? ...
0
votes
2answers
340 views

remote backup with curl/wget

I need to backup a clients server (source) from our mediatemple server (destination) and then add it into a git repository while ssh'd in. What is the most efficient/fastest way to do so? wget? or ...
1
vote
1answer
98 views

screen scraper templates for various websites

I'm looking specifically for a convenient way to locally archive posts from this and other similar sites. I'd like to separate the question itself from the answers, or maybe crop the question and ...
1
vote
1answer
715 views

Load multiple files simulatenously with cURL

I am looking to download multiple files from a server using cURL from the Terminal on a Mac. I know that using the {} operator, I can load one file after another but I am looking to download all (10) ...
0
votes
2answers
3k views

How to download a file behind authentication via wget or curl [duplicate]

Possible Duplicate: WGet a Page that Requires Logging in I wish to download file from http://mbr.nlm.nih.gov/Download/MetaMapped_Medline/2011/ however in order to download the file, i need ...
2
votes
1answer
2k views

Where's the ca cert bundle on OSX?

I'm debugging some curl ssl configuration stuff and was wondering where the ca cert bundle is installed on OSX.
3
votes
3answers
2k views

Downloading XCode using wget or curl [closed]

Is there a way for downloading XCode using curl or wget? EDIT: Why is the following curl snippet not working? First, I login to Apple's site: curl -d "theAccountName=USERNAME&theAccountPW=PASS" ...
1
vote
2answers
556 views

bash: source from URL

In addition to my own computer, I sometimes use an Ubuntu cluster at my school. Rather than manually keep my .bashrc's in sync, I would like to make the school cluster's .bashrc source my personal ...
1
vote
1answer
339 views

help with Outlook Exchange server and curl

I work on a mac in a building full of PCs, and the IT department here doesn't have IMAP access turned on on the exchange servers. So I miss a lot of meetings because I don't get reminders because I ...
4
votes
2answers
1k views

How can I pass my Windows credentials in cURL?

I'm writing a script that uses cURL to pull data from an internal web server. I can explicitly use --user combined with --ntlm and everything works well, but I can't get it to recognize whomever's ...
0
votes
2answers
711 views

How to escape “@” in the username when logging in through FTPES with curl?

$ curl -T "index.html" -k --ftp-ssl -u "MYUSER@MYDOMAIN.COM" MYDOMAIN.COM Enter host password for user 'MYUSER@MYDOMAIN.COM': % Total % Received % Xferd Average Speed Time Time Time ...
0
votes
3answers
1k views

Wget hangs in the middle of get request

Sometimes wget or curl randomly hangs in the middle of get request in Ubuntu 10.10: curl "http://api.indeed.com/ads/apisearch?publisher=7570038743238473" wget ...
1
vote
0answers
187 views

how to use to proxy in the intranet?

i have an intranet site which is inside only our company and feeds information. i download an weather API from http://www.foreca.com/Philippines/ and hook to my site. the API has a search engine and ...
3
votes
3answers
674 views

What's the equivalent of “wget -i FILE” in Curl?

What's the equivalent of "wget -i FILE" in Curl?
7
votes
4answers
3k views

How “legal” is site-scraping using cURL? [closed]

Recently I was experimenting with the cURL, and I found lot is possible with it. I built a small script that crawls a musical site, which plays online songs. On the way of my experiment, I found that ...
0
votes
2answers
317 views

How to automatically get images from an SSL Site that uses Java Servlets

This is tricky. I want to use curl, wget or any other tool to login into a SSL Website which provides a Login Form. Then I want to visit several Links within that Domain and fetch certain images. I ...
0
votes
1answer
62 views

how can i check the size on a server?

how can I check the size of file on server in bytes? i need to do it for following