curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

0
votes
1answer
450 views

Downloading and extracting a .tar to a specific directory

I want to download a tar of a git repo and extract its contents to another folder. The following line (broken up for clarity) works: curl -L -0 nt-dotfiles.tar.gz ...
0
votes
4answers
118 views

Simple script parsing text, what is wrong here?

I'm a big user of https://www.grc.com/passwords.htm to get strong passwords. However, having to go to the site and manually copy the password every time gets old fast, so I decided to do a little ...
2
votes
1answer
1k views

using cURL to download all files in certain folder, except 404 error page?

I'm using cURL in ubuntu to download some files like pic001.jpg pic002.jpg pic003.jpg ... pic999.jpg While some files may be missing from this sequence but when I just use ...
1
vote
0answers
116 views

No output returned when curl ing any URL in Red Hat 6 but the same url works in lynx browser

I am trying to install some packages in Python using easy install. But i get a 'Download Error' for any package i try to install. So i tried to test if internet is accessible at all from the machine ...
0
votes
1answer
141 views

CURL -I issue stop responding when contain “=”

i did this command : curl -I ...
0
votes
1answer
945 views

Why can't I fetch this sites contents with curl?

I'm trying to download the contents of https://aarhustech.itslearning.com/elogin/default.aspx with curl, but I get this error: $ curl https://aarhustech.itslearning.com/elogin/default.aspx curl: (35) ...
0
votes
0answers
2k views

curl http_code of 000

I have a shell script that I use to monitor loading times and response codes on my live server cluster. It runs a total of 250 iterations every 5 minutes, distributed across 10 servers and 6 sites. It ...
1
vote
2answers
225 views

Curl Compressed Command not Recognized in Windows

I saw this post and successfully added curl to my Windows command prompt as instructed. However, when I attempted to run curl --compressed, it says: "the installed libcurl version doesn't ...
4
votes
4answers
15k views

Downloading file from FTP using cURL

I'm trying to use a cURL command to download a file from an FTP server to a local drive on my computer. I've tried curl "ftp://myftpsite" --user name:password -Q "CWD /users/myfolder/" -O ...
0
votes
2answers
1k views

How to use wget to grab copy of Google Code site documents?

I have a Google Code project which has a lot of wiki'ed documentation. I would like to create a copy of this documentation for offline browsing. I would like to use wget or a similar utility. I have ...
2
votes
2answers
1k views

Curl alternative on windows to get header information [duplicate]

Possible Duplicate: wget/cURL alternative native to Windows? Is there any tool (command line tool preferred) to get the HTTP request header information on Windows machine like Curl does on ...
1
vote
2answers
1k views

Why sudo curl ignores proxy settings?

$ echo $http_proxy http://my.proxy.com $ curl -v http://files.com/a.txt * About to connect() to proxy my.proxy.com # Correct downloading $ sudo echo $http_proxy http://my.proxy.com $sudo curl -v ...
2
votes
1answer
2k views

curl: (7) couldn't connect to host

I know some people have asked this question before but I don't understand some of the answers. I'm a Mac user trying to instal Homebrew but I keep getting this error message: curl: (7) couldn't ...
1
vote
0answers
113 views

Download web server structure with empty files

I want to make a mirror of a Web server, but downloading the actual files will take too long. So I thought of having just the directory and file structure, and when I need the actual contents of the ...
5
votes
1answer
443 views

How did Google get on my Mac?

Am running a MacBook Pro, and have never installed Chrome, Google Earth, or anything blatantly Google. Just installed Little Snitch (are there no good free firewalls for Mac?) and see that CURL is ...
1
vote
3answers
634 views

How to retrieve all *.html files from website using Unix command line tools and regular expression

I would like to retrieve all .html files from a website that has a certain text on its name: eg. this_is_good_site.html So, I would like to download .html files with a word "good" on its name. I ...
1
vote
1answer
6k views

Downloading only images using curl or wget?

UPDATED: I've found using this Bash script fixes the problem of having GIF files with a .jpg extension. I'm attempting to download images from a forum who's url uses the following format: ...
0
votes
1answer
809 views

How do I enable the curl progress bar in cygwin?

When you curl a file, you get this progress bar whether you're uploading or downloading a file. % Total % Received % Xferd Average Speed Time Time Time Current ...
11
votes
1answer
6k views

Why can't cURL properly verify a certificate on Windows?

When I try to use Curl on windows, to retrieve an https url, I get the dreaded "connection error (60)." The exact error message is: curl: (60) SSL certificate problem, verify that the CA cert ...
3
votes
1answer
633 views

How to make alias/function in .bash_profile for `curl` that takes an argument?

I found this tip super helpful, for using curl to resume an interrupted file copy. The full syntax is: curl -C - -O file:///Volumes/path/to/file I would like to add an alias or function called ...
0
votes
1answer
1k views

piping curl's response headers

I'm getting weird results when trying to capture curl's response headers in a variable: pattern="< Content-Length: " val=$(curl --verbose \ --request POST --data "desc=hello world" \ ...
2
votes
1answer
221 views

curl works at the command line, but how do i put it in a php file?

The following curl format works for me from the command line. The line below is just sample data, but the format works. curl -u email@email.com:1234567 https://api.email.com/status How do I ...
3
votes
2answers
2k views

Persistent retrying resuming downloads with curl

I'm on a mac and have a list of files I would like to download from an ftp server. The connection is a bit buggy so I want it to retry and resume if connection is dropped. I know I can do this with ...
4
votes
3answers
13k views

URL to reboot a WebSTAR DPC2100R2 cable modem with curl?

Does anyone know what the magic URL is for rebooting a WebSTAR DPC2100R2 with curl? I used to have a SurfBoard, and the curl command: curl ...
2
votes
1answer
21k views

How to install curl on ubuntu

System: ibook G4 / linux ubuntu 2.6.28-6-powerpc OS : ubuntu 9.04(jaunty) PHP ver : 5.2.6-3ubuntu4.6 Please let me know how to install curl on my system. Synaptic Package Manager tells me to ...
1
vote
1answer
1k views

How do I remotely fetch files from redirected URLs from a terminal?

I want to fetch a tarball of this python library from terminal. https://github.com/simplegeo/python-oauth2/downloads However, I cannot simply call wget ...
1
vote
1answer
656 views

Grab source from Google Code using wget or curl?

Is it possible to grab the source-code from a Google Code repository (hg) using wget or curl? Else using Python? Thanks for all suggestions FYI: I'm on a shared WebFaction host, so can't install hg ...
5
votes
1answer
9k views

CURL File with post parameter

How to make a request for uploading a file to facebook using graph api in curl I am making this request but getting Error: curl https://graph.facebook.com/<id>/photos -F "source=@me.jpg" -d ...
21
votes
3answers
20k views

How to wget a file with correct name when redirected?

So after some time of searching on google and superuser (and scanning man pages) I was unable to find an answer to something that (I think) should be simple: If you go here: ...
0
votes
1answer
3k views

Curl Couldn't resolve host for HTTPS

I am trying to use curl on a Red Hat Enterprise Linux Server release 6.1. When I try to use URL with HTTPS it gives an error message "Couldn't resolve host". But for the same URL when tried with HTTP ...
0
votes
2answers
807 views

command line curl - file naming example.com/[1-10].txt => 01.txt through 10.txt

Is it possible to set the variable portion of the file name output to be a specific length and fill the extra spaces with zeros. I want things to be ordered for other users, so I would like curl ...
0
votes
2answers
549 views

test of ICMP block

In my bash scripts I have been using something like: until fping -u google.com; do echo "$0[$$] Network/DNS down?? $(date)" 1>&2 && sleep $(($RANDOM%(1 + ++trynum * 1) +1)).222; done ...
0
votes
1answer
384 views

apt-get install new packeges, avoid others

Look at this entry: root@vivid-75:~# apt-get --no-install-recommends install curl Reading package lists... Done Building dependency tree Reading state information... Done The following extra ...
0
votes
1answer
2k views

How can I use curl with a SSL URL to get a 200 OK?

I'm trying to use curl to get the HTTP status of a GET request: curl --insecure --silent --show-error --connect-timeout 1 -I https://host:8443/health Note: I'm using the --insecure flag in this ...
0
votes
1answer
418 views

How to make curl reconnect after 60"?

Sometimes, curl freezes: $ curl -O 'http://www.site.com/index.html' www.site.com/index.html --> index.html % Total % Received % Xferd Average Speed Time Time Time Current ...
0
votes
1answer
560 views

Apache and PHP with cURL

I am running Sun server with Solaris 10/SPARC. I had/have Apache 1.37 with PHP 5.2.1 running as a dynamic extension, and want to enable cURL support. I have tried to re-compile PHP with the ...
1
vote
1answer
624 views

How to use delicious API from command line

I'm trying to add a delicious bookmark from a shell script, but keep getting the mysterious "something went wrong" error. I've checked the official spec and several examples and everything looks ...
3
votes
1answer
2k views

VPN routing issue in linux- how to make applications that can only bind to IP addresses work?

Right now I have a PPTP VPN set up on my remote ubuntu VPS. I would like to have certain applications use the VPN to access the internet (rtorrent, irssi, among others). Currently, it is set up like ...
2
votes
1answer
2k views

Cannot connect to website - SSL handshaking fails

So I cannot connect to certain websites. Just a few, most are OK. The one I really care about is paypal.com. I have done the usual things. Let's see: Cleared browsing history Tried different ...
3
votes
1answer
960 views

Force curl to download an empty file

In a bash script, I am using curl command on linux to download files from a FTP server. The command I am using is the following (the information about the FTP server connexion is stored in a .netrc ...
0
votes
1answer
333 views

Allow Incoming Responses Apache. On Ubuntu 11.10 - Curl

I'm trying to get a Curl Response from an outside server, however I noticed I cant neither PING the server in question nor connect to it. I tried disabling the iptables firewall but I had no success. ...
1
vote
1answer
2k views

How to stop running download process by CURL?

I forgot to use > filename after curl operation. linux console is giving contiguously garbage data(hopefully downloading and displaying the content on console). How can we stop this process (ctrl-c ...
0
votes
1answer
383 views

curl (7) permission denied on cgi script

Can't understand which permissions could be denied. Running cgi script from command line works fine. http_proxy="" curl http://url/ -o file Produces curl(7): permission denied (same issues with ...
1
vote
2answers
2k views

Can I encrypt a username password combo before using it with cURL?

I'm making a shell script that reached out to a web service with basic auth. I don't want to have to put in my username and password every time the script runs, but I also don't want to have the combo ...
8
votes
2answers
2k views

How to download parts of the same file from different sources with curl/wget?

I have a quite large file hosted on five different servers. I would like to be able to download different parts of the file from each server and subsequently concatenate the parts, in order to ...
1
vote
1answer
1k views

Submitting a form using curl

Forgive me for the simple question... I haven't used curl before. I'm trying to automatically download some data from this form: http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500 which ...
2
votes
0answers
533 views

Restore default CA Certificates on Mac OSX 10.7

I've been screwing around with my Mac, trying to install self-signed certificate as trusted for our WAN site (for which we can't justify paying for a real signed certificate) I've broken something ...
0
votes
1answer
301 views

Only output curl result dependent on mime-type

Is it possible to make curl only save the output of curl -o if the file is of a certain mime-type? For example, I might use the following command to download a bunch of images off a server... curl ...
3
votes
1answer
276 views

Can cURL use a keyring for client certificates?

I'm using cURL to fetch from an HTTPS location, which requires a client certificate. I have my .curlrc set up to tell cURL where it is, but I don't want to leave my unencrypted private key just lying ...
0
votes
1answer
855 views

Curl won't prompt me for a certificate password

I'm trying to use CURL to test simple HTTPS connections to servers that require a client certificate. I've specified the certificate type in my .curlrc file with cert = /path/to/Cert.p12 and told it ...