curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

11
votes
1answer
7k views

Why can't cURL properly verify a certificate on Windows?

When I try to use Curl on windows, to retrieve an https url, I get the dreaded "connection error (60)." The exact error message is: curl: (60) SSL certificate problem, verify that the CA cert ...
3
votes
1answer
788 views

How to make alias/function in .bash_profile for `curl` that takes an argument?

I found this tip super helpful, for using curl to resume an interrupted file copy. The full syntax is: curl -C - -O file:///Volumes/path/to/file I would like to add an alias or function called ...
0
votes
1answer
1k views

piping curl's response headers

I'm getting weird results when trying to capture curl's response headers in a variable: pattern="< Content-Length: " val=$(curl --verbose \ --request POST --data "desc=hello world" \ ...
2
votes
1answer
245 views

curl works at the command line, but how do i put it in a php file?

The following curl format works for me from the command line. The line below is just sample data, but the format works. curl -u email@email.com:1234567 https://api.email.com/status How do I ...
3
votes
2answers
3k views

Persistent retrying resuming downloads with curl

I'm on a mac and have a list of files I would like to download from an ftp server. The connection is a bit buggy so I want it to retry and resume if connection is dropped. I know I can do this with ...
4
votes
3answers
14k views

URL to reboot a WebSTAR DPC2100R2 cable modem with curl?

Does anyone know what the magic URL is for rebooting a WebSTAR DPC2100R2 with curl? I used to have a SurfBoard, and the curl command: curl ...
2
votes
1answer
22k views

How to install curl on ubuntu

System: ibook G4 / linux ubuntu 2.6.28-6-powerpc OS : ubuntu 9.04(jaunty) PHP ver : 5.2.6-3ubuntu4.6 Please let me know how to install curl on my system. Synaptic Package Manager tells me to ...
1
vote
1answer
2k views

How do I remotely fetch files from redirected URLs from a terminal?

I want to fetch a tarball of this python library from terminal. https://github.com/simplegeo/python-oauth2/downloads However, I cannot simply call wget ...
1
vote
1answer
730 views

Grab source from Google Code using wget or curl?

Is it possible to grab the source-code from a Google Code repository (hg) using wget or curl? Else using Python? Thanks for all suggestions FYI: I'm on a shared WebFaction host, so can't install hg ...
25
votes
3answers
24k views

How to wget a file with correct name when redirected?

So after some time of searching on google and superuser (and scanning man pages) I was unable to find an answer to something that (I think) should be simple: If you go here: ...
0
votes
1answer
4k views

Curl Couldn't resolve host for HTTPS

I am trying to use curl on a Red Hat Enterprise Linux Server release 6.1. When I try to use URL with HTTPS it gives an error message "Couldn't resolve host". But for the same URL when tried with HTTP ...
0
votes
2answers
1k views

command line curl - file naming example.com/[1-10].txt => 01.txt through 10.txt

Is it possible to set the variable portion of the file name output to be a specific length and fill the extra spaces with zeros. I want things to be ordered for other users, so I would like curl ...
0
votes
2answers
628 views

test of ICMP block

In my bash scripts I have been using something like: until fping -u google.com; do echo "$0[$$] Network/DNS down?? $(date)" 1>&2 && sleep $(($RANDOM%(1 + ++trynum * 1) +1)).222; done ...
0
votes
1answer
398 views

apt-get install new packeges, avoid others

Look at this entry: root@vivid-75:~# apt-get --no-install-recommends install curl Reading package lists... Done Building dependency tree Reading state information... Done The following extra ...
0
votes
1answer
3k views

How can I use curl with a SSL URL to get a 200 OK?

I'm trying to use curl to get the HTTP status of a GET request: curl --insecure --silent --show-error --connect-timeout 1 -I https://host:8443/health Note: I'm using the --insecure flag in this ...
0
votes
1answer
482 views

How to make curl reconnect after 60"?

Sometimes, curl freezes: $ curl -O 'http://www.site.com/index.html' www.site.com/index.html --> index.html % Total % Received % Xferd Average Speed Time Time Time Current ...
0
votes
1answer
659 views

Apache and PHP with cURL

I am running Sun server with Solaris 10/SPARC. I had/have Apache 1.37 with PHP 5.2.1 running as a dynamic extension, and want to enable cURL support. I have tried to re-compile PHP with the ...
1
vote
1answer
654 views

How to use delicious API from command line

I'm trying to add a delicious bookmark from a shell script, but keep getting the mysterious "something went wrong" error. I've checked the official spec and several examples and everything looks ...
3
votes
1answer
2k views

VPN routing issue in linux- how to make applications that can only bind to IP addresses work?

Right now I have a PPTP VPN set up on my remote ubuntu VPS. I would like to have certain applications use the VPN to access the internet (rtorrent, irssi, among others). Currently, it is set up like ...
2
votes
1answer
2k views

Cannot connect to website - SSL handshaking fails

So I cannot connect to certain websites. Just a few, most are OK. The one I really care about is paypal.com. I have done the usual things. Let's see: Cleared browsing history Tried different ...
3
votes
1answer
1k views

Force curl to download an empty file

In a bash script, I am using curl command on linux to download files from a FTP server. The command I am using is the following (the information about the FTP server connexion is stored in a .netrc ...
0
votes
1answer
360 views

Allow Incoming Responses Apache. On Ubuntu 11.10 - Curl

I'm trying to get a Curl Response from an outside server, however I noticed I cant neither PING the server in question nor connect to it. I tried disabling the iptables firewall but I had no success. ...
1
vote
1answer
3k views

How to stop running download process by CURL?

I forgot to use > filename after curl operation. linux console is giving contiguously garbage data(hopefully downloading and displaying the content on console). How can we stop this process (ctrl-c ...
1
vote
1answer
424 views

curl (7) permission denied on cgi script

Can't understand which permissions could be denied. Running cgi script from command line works fine. http_proxy="" curl http://url/ -o file Produces curl(7): permission denied (same issues with ...
1
vote
2answers
2k views

Can I encrypt a username password combo before using it with cURL?

I'm making a shell script that reached out to a web service with basic auth. I don't want to have to put in my username and password every time the script runs, but I also don't want to have the combo ...
8
votes
2answers
2k views

How to download parts of the same file from different sources with curl/wget?

I have a quite large file hosted on five different servers. I would like to be able to download different parts of the file from each server and subsequently concatenate the parts, in order to ...
1
vote
1answer
1k views

Submitting a form using curl

Forgive me for the simple question... I haven't used curl before. I'm trying to automatically download some data from this form: http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500 which ...
2
votes
0answers
600 views

Restore default CA Certificates on Mac OSX 10.7

I've been screwing around with my Mac, trying to install self-signed certificate as trusted for our WAN site (for which we can't justify paying for a real signed certificate) I've broken something ...
0
votes
1answer
319 views

Only output curl result dependent on mime-type

Is it possible to make curl only save the output of curl -o if the file is of a certain mime-type? For example, I might use the following command to download a bunch of images off a server... curl ...
3
votes
1answer
304 views

Can cURL use a keyring for client certificates?

I'm using cURL to fetch from an HTTPS location, which requires a client certificate. I have my .curlrc set up to tell cURL where it is, but I don't want to leave my unencrypted private key just lying ...
0
votes
1answer
991 views

Curl won't prompt me for a certificate password

I'm trying to use CURL to test simple HTTPS connections to servers that require a client certificate. I've specified the certificate type in my .curlrc file with cert = /path/to/Cert.p12 and told it ...
3
votes
2answers
4k views

How to see HTTP request header by curl?

I can see HTTP header response by curl -I command. Is there any option to see the HTTP request header sent? # curl -I www.google.com HTTP/1.1 200 OK Date: Sat, 31 Dec 2011 00:55:53 GMT Expires: -1 ...
0
votes
1answer
14k views

How to install CURL on Debian Squeeze?

I just can't get CURL to install on Debian Squeeze... keeps giving me this: libtool: compile: gcc -DHAVE_CONFIG_H -I../include/curl -I../include -I../include -I../lib -I../lib -g0 -O2 ...
0
votes
2answers
2k views

How can I curl the output of another command

I want to pass curl the output from awk ./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | curl OUTPUT_FROM_AWK | grep myString How can I achieve this?!
1
vote
1answer
485 views

Scrape a site with curl/wget appending a parameter to each site

I need to scrape a website for all sites and append an Paramater, e.g. www.theSite.com/allpages?myPara=us Now what I got is wget --recursive to get all sites, but that doesn't append the parameter ...
0
votes
1answer
599 views

No response from CURL at Windows Command Prompt

I've tried following Run cURL commands from Windows console but typing in any curl command returns nothing. See below - I just get back the command prompt. c:\curl>curl curl: try 'curl ...
0
votes
0answers
866 views

cURL does not work on verbose mode

I have installed curl on a new ubuntu installation with apt-get and it behaves really weird. When I do curl http://www.google.co.uk everything work ok, but when I do curl -v ...
1
vote
1answer
2k views

How do I setup curl / wget with FTP proxy?

I'm having a good deal of trouble getting curl or wget to download files from FTP servers through an FTP proxy. I have setup GUI FTP programs to work... gFTP works with these settings: I get ...
1
vote
1answer
1k views

compiling curl with HipHop for PHP Patch [closed]

I'm following this guide: https://github.com/facebook/hiphop-php/wiki/Building-and-installing-on-ubuntu-10.10 It want's me to download curl source, doing the patch and compile this. well, but i get ...
0
votes
1answer
303 views

find xargs and curl give an error with -C- option

This command works fine find ~/mypath -name *.fstq -print|xargs curl -u sdd:dsdsd ftp://ftp.dsdd.nsdh.com/sdsds/ -T but when I add find ~/mypath -name *.fstq -print|xargs curl -C- -u sdd:dsdsd ...
0
votes
1answer
584 views

Using wget on a site with cgi

I need to mirror a website. However, each of the links on the site's webpage is actually a 'submit' to a cgi script that shows up the resulting page. AFAIK wget should fail on this since it needs ...
1
vote
2answers
3k views

curlftpfs doesn't work for a username with a “@”

My hosting company makes all my usernames with a "@" in them. curlftpfs user="user@domain.com:pass" ftp://ftp.domain.com/ ~/domain/ For some reason I get in response Error connecting to ftp: ...
2
votes
1answer
1k views

Download a file from a website in command line in Linux

I need to download a file from a website which has a URL formatted like: http://www.server.com/downloads/1234/ This redirects to a .zip file which has to be saved. There is also a need to ...
0
votes
1answer
469 views

How to set downtime for any specific nagios host for certain time from commandline through curl?

I have a curl script which will do the service/host notification disable/enable from the command line through curl. But need to do that for scheduled downtime. Can someone suggest me. Okay it seems ...
1
vote
0answers
562 views

VirtualBox Ubuntu guest on OS X Lion host: Curl won't work

I'm trying to install some gems on an Ubuntu VM on OS X Lion using VirtualBox. I've narrowed my problem down to curl which is not responding at all. It seems to be going through some proxy local host ...
0
votes
2answers
1k views

Search for text in web pages given a list of URLs

I have a list of several thousand URLs, and I'd like to search each of these pages for a given word. How can I do this programmatically on Windows, preferably using VBScript or Powershell?
1
vote
2answers
258 views

Where does curl store data whilest it's still downloading

Hi I'm using curl on the command line to automate some downloading of many files from a server . However I'm wondering where curl stores it's data whilst it's downloading? The reason I need this ...
3
votes
3answers
2k views

HTTP responses curl and wget different results

To check HTTP response header for a set of urls I send with curl the following request headers foreach ( $urls as $url ) { // Setup headers - I used the same headers from Firefox version 2.0.0.6 ...
0
votes
1answer
368 views

Can I set a default hostname for curl?

I would like to reduce the keystrokes when I use curl to test web services on my workstation. Is there a way to set a default hostname of 'localhost:3000' into ~/.curlrc so the following will work? ...
0
votes
2answers
357 views

remote backup with curl/wget

I need to backup a clients server (source) from our mediatemple server (destination) and then add it into a git repository while ssh'd in. What is the most efficient/fastest way to do so? wget? or ...