Questions tagged [curl]

curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

Filter by
Sorted by
Tagged with
0 votes
2 answers
2k views

cURL or xargs: show progress

I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into ...
William Entriken's user avatar
4 votes
1 answer
2k views

Unable to resolve host using curl/wget while page loads in browser

The problem I am facing is that while I open a link in my browser it loads quickly but whenever I try using the link in terminal with for e.g. curl -0 www.google.com or wget www.google....
Debopam Mitra's user avatar
1 vote
0 answers
404 views

the right options to traverse/download the pages/directories of a subdomain

Let's suppose exist a site with the following directories (subdomain) index.php |-sub1 |-index.php |-sub1sub1 |-index.php |-other.php |-sub1sub1sub1 |-sub2 |-index.php |- …....
Lorraine Bernard's user avatar
16 votes
4 answers
49k views

curl http_code of 000

I have a shell script that I use to monitor loading times and response codes on my live server cluster. It runs a total of 250 iterations every 5 minutes, distributed across 10 servers and 6 sites. It ...
Mikkel's user avatar
  • 825
6 votes
4 answers
14k views

Curl Compressed Command not Recognized in Windows

I saw this post and successfully added curl to my Windows command prompt as instructed. However, when I attempted to run curl --compressed, it says: "the installed libcurl version doesn't support ...
derigible's user avatar
  • 163
6 votes
2 answers
8k views

Why sudo curl ignores proxy settings?

$ echo $http_proxy http://my.proxy.com $ curl -v http://files.com/a.txt * About to connect() to proxy my.proxy.com # Correct downloading $ sudo echo $http_proxy http://my.proxy.com $sudo curl -v ...
Nelson Tatius's user avatar
3 votes
1 answer
4k views

curl: (7) couldn't connect to host

I know some people have asked this question before but I don't understand some of the answers. I'm a Mac user trying to instal Homebrew but I keep getting this error message: curl: (7) couldn't ...
Siya's user avatar
  • 131
1 vote
0 answers
256 views

Download web server structure with empty files

I want to make a mirror of a Web server, but downloading the actual files will take too long. So I thought of having just the directory and file structure, and when I need the actual contents of the ...
golimar's user avatar
  • 1,652
6 votes
1 answer
3k views

How did Google get on my Mac?

I am running a MacBook Pro, and have never installed Chrome, Google Earth, or anything blatantly Google. Just installed Little Snitch (are there no good free firewalls for Mac?) and see that CURL is ...
SamGoody's user avatar
  • 395
2 votes
2 answers
7k views

How do I unzip a compressed file as it's downloading?

I'm downloading a large file that's not an archive, and I want to combine the tasks of downloading and decompressing. How can I do them simultaneously?
Max's user avatar
  • 709
5 votes
1 answer
3k views

How to make alias/function in .bash_profile for `curl` that takes an argument?

I found this tip super helpful, for using curl to resume an interrupted file copy. The full syntax is: curl -C - -O file:///Volumes/path/to/file I would like to add an alias or function called ...
d0g's user avatar
  • 2,280
0 votes
1 answer
3k views

How do I enable the curl progress bar in cygwin?

When you curl a file, you get this progress bar whether you're uploading or downloading a file. % Total % Received % Xferd Average Speed Time Time Time Current ...
obesechicken13's user avatar
2 votes
1 answer
979 views

curl works at the command line, but how do i put it in a php file?

The following curl format works for me from the command line. The line below is just sample data, but the format works. curl -u [email protected]:1234567 https://api.email.com/status How do I ...
OahuRE's user avatar
  • 21
0 votes
2 answers
2k views

What should I do in order to build curl without an error?

This failed when I run ./buildconf The error information is as follows: [mirror@home curl]$ ls acinclude.m4 CMakeLists.txt GIT-INFO MacOSX-Framework packages TODO-RELEASE ...
hugemeow's user avatar
  • 2,299
1 vote
2 answers
2k views

What tool does yum use to download rpm packages from remote server?

My system is centos, and now I am installing packages using yum command. Updating for dependencies: apr-util i386 1.3.9-1.el5 lxcenter-updates 203 k gcc-gfortran ...
hugemeow's user avatar
  • 2,299
4 votes
4 answers
20k views

Silence curl with >/dev/null 2>&1 when it's piped

So I have a curl that is piped to a grep and a sed. Where would I apply the >/dev/null 2>&1 ? curl www.site.com | grep stuff | sed "other stuff" At the very end or after the curl?
tzippy's user avatar
  • 454
15 votes
1 answer
14k views

How do I remotely fetch files from redirected URLs from a terminal?

I want to fetch a tarball of this python library from terminal. https://github.com/simplegeo/python-oauth2/downloads However, I cannot simply call wget https://github.com/simplegeo/python-oauth2/...
David Faux's user avatar
  • 4,829
1 vote
1 answer
1k views

Grab source from Google Code using wget or curl?

Is it possible to grab the source-code from a Google Code repository (hg) using wget or curl? Else using Python? Thanks for all suggestions FYI: I'm on a shared WebFaction host, so can't install hg
A T's user avatar
  • 791
9 votes
1 answer
20k views

CURL File with post parameter

How to make a request for uploading a file to facebook using graph api in curl I am making this request but getting Error: curl https://graph.facebook.com/<id>/photos -F "[email protected]" -d "...
behinddwalls's user avatar
32 votes
3 answers
88k views

Why can't cURL properly verify a certificate on Windows?

When I try to use Curl on windows, to retrieve an https url, I get the dreaded "connection error (60)." The exact error message is: curl: (60) SSL certificate problem, verify that the CA cert is ...
Cheeso's user avatar
  • 2,139
0 votes
1 answer
26k views

Curl Couldn't resolve host for HTTPS

I am trying to use curl on a Red Hat Enterprise Linux Server release 6.1. When I try to use URL with HTTPS it gives an error message "Couldn't resolve host". But for the same URL when tried with HTTP (...
Indika K's user avatar
  • 177
1 vote
2 answers
2k views

test of ICMP block

In my bash scripts I have been using something like: until fping -u google.com; do echo "$0[$$] Network/DNS down?? $(date)" 1>&2 && sleep $(($RANDOM%(1 + ++trynum * 1) +1)).222; done ...
Marcos's user avatar
  • 1,161
0 votes
1 answer
2k views

Why can't I fetch this sites contents with curl?

I'm trying to download the contents of https://aarhustech.itslearning.com/elogin/default.aspx with curl, but I get this error: $ curl https://aarhustech.itslearning.com/elogin/default.aspx curl: (35) ...
Tyilo's user avatar
  • 2,737
0 votes
1 answer
470 views

apt-get install new packeges, avoid others

Look at this entry: root@vivid-75:~# apt-get --no-install-recommends install curl Reading package lists... Done Building dependency tree Reading state information... Done The following extra ...
DontVoteMeDown's user avatar
1 vote
2 answers
31k views

How can I use curl with a SSL URL to get a 200 OK?

I'm trying to use curl to get the HTTP status of a GET request: curl --insecure --silent --show-error --connect-timeout 1 -I https://host:8443/health Note: I'm using the --insecure flag in this ...
redspike's user avatar
  • 125
0 votes
1 answer
2k views

How to make curl reconnect after 60"?

Sometimes, curl freezes: $ curl -O 'http://www.site.com/index.html' www.site.com/index.html --> index.html % Total % Received % Xferd Average Speed Time Time Time Current ...
kev's user avatar
  • 12.7k
0 votes
1 answer
2k views

Apache and PHP with cURL

I am running Sun server with Solaris 10/SPARC. I had/have Apache 1.37 with PHP 5.2.1 running as a dynamic extension, and want to enable cURL support. I have tried to re-compile PHP with the --with-...
Rooneyl's user avatar
  • 185
2 votes
1 answer
15k views

curl failed setting cipher list

I'm trying to make curl use GOST2001-GOST89-GOST89 cipher which is available and usable by OpenSSL but keep getting failed setting cipher list error despite the fact that curl sees gost engine and can ...
synapse's user avatar
  • 243
3 votes
1 answer
2k views

VPN routing issue in linux- how to make applications that can only bind to IP addresses work?

Right now I have a PPTP VPN set up on my remote ubuntu VPS. I would like to have certain applications use the VPN to access the internet (rtorrent, irssi, among others). Currently, it is set up like ...
thatfunkymunki's user avatar
0 votes
2 answers
2k views

How to use wget to grab copy of Google Code site documents?

I have a Google Code project which has a lot of wiki'ed documentation. I would like to create a copy of this documentation for offline browsing. I would like to use wget or a similar utility. I have ...
Alex Reynolds's user avatar
2 votes
2 answers
2k views

command line curl - file naming example.com/[1-10].txt => 01.txt through 10.txt

Is it possible to set the variable portion of the file name output to be a specific length and fill the extra spaces with zeros. I want things to be ordered for other users, so I would like curl ...
TST's user avatar
  • 23
0 votes
1 answer
525 views

Allow Incoming Responses Apache. On Ubuntu 11.10 - Curl

I'm trying to get a Curl Response from an outside server, however I noticed I cant neither PING the server in question nor connect to it. I tried disabling the iptables firewall but I had no success. ...
Daniel Adarve's user avatar
2 votes
1 answer
5k views

Cannot connect to website - SSL handshaking fails

So I cannot connect to certain websites. Just a few, most are OK. The one I really care about is paypal.com. I have done the usual things. Let's see: Cleared browsing history Tried different ...
ravenspoint's user avatar
1 vote
1 answer
20k views

How to stop running download process by CURL?

I forgot to use > filename after curl operation. linux console is giving contiguously garbage data(hopefully downloading and displaying the content on console). How can we stop this process (ctrl-c ...
P K's user avatar
  • 2,203
2 votes
1 answer
11k views

Downloading only images using curl or wget?

UPDATED: I've found using this Bash script fixes the problem of having GIF files with a .jpg extension. I'm attempting to download images from a forum who's url uses the following format: http://...
Ashley's user avatar
  • 227
1 vote
1 answer
1k views

How to use delicious API from command line

I'm trying to add a delicious bookmark from a shell script, but keep getting the mysterious "something went wrong" error. I've checked the official spec and several examples and everything looks right....
Stan James's user avatar
1 vote
3 answers
1k views

curl (7) permission denied on cgi script

Can't understand which permissions could be denied. Running cgi script from command line works fine. http_proxy="" curl http://url/ -o file Produces curl(7): permission denied (same issues with ...
grammar31's user avatar
  • 286
3 votes
2 answers
7k views

Can I encrypt a username password combo before using it with cURL?

I'm making a shell script that reached out to a web service with basic auth. I don't want to have to put in my username and password every time the script runs, but I also don't want to have the combo ...
Greg Guida's user avatar
1 vote
1 answer
3k views

Submitting a form using curl

Forgive me for the simple question... I haven't used curl before. I'm trying to automatically download some data from this form: http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500 which ...
Stochtastic's user avatar
2 votes
0 answers
2k views

Restore default CA Certificates on Mac OSX 10.7

I've been screwing around with my Mac, trying to install self-signed certificate as trusted for our WAN site (for which we can't justify paying for a real signed certificate) I've broken something ...
Lee Hambley's user avatar
1 vote
1 answer
1k views

How do I install hiphop-php in CentOS 6?

I've been trying to install hiphop-php on CentOS 6 with no luck. I found an .rpm for it but it fails dependency checks on boost which fails to install because it depends on curl which fails to ...
Dai's user avatar
  • 11
4 votes
1 answer
1k views

Can cURL use a keyring for client certificates?

I'm using cURL to fetch from an HTTPS location, which requires a client certificate. I have my .curlrc set up to tell cURL where it is, but I don't want to leave my unencrypted private key just lying ...
Coderer's user avatar
  • 1,620
8 votes
1 answer
11k views

How to see HTTP request header by curl?

I can see HTTP header response by curl -I command. Is there any option to see the HTTP request header sent? curl -I www.google.com HTTP/1.1 200 OK Date: Sat, 31 Dec 2011 00:55:53 GMT Expires: -1 ...
P K's user avatar
  • 2,203
0 votes
1 answer
19k views

How to install CURL on Debian Squeeze?

I just can't get CURL to install on Debian Squeeze... keeps giving me this: libtool: compile: gcc -DHAVE_CONFIG_H -I../include/curl -I../include -I../include -I../lib -I../lib -g0 -O2 -Wno-system-...
Alasdair's user avatar
  • 857
2 votes
2 answers
8k views

How can I curl the output of another command

I want to pass curl the output from awk ./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | curl OUTPUT_FROM_AWK | grep myString How can I achieve this?!
user avatar
1 vote
1 answer
686 views

Scrape a site with curl/wget appending a parameter to each site

I need to scrape a website for all sites and append an Paramater, e.g. www.theSite.com/allpages?myPara=us Now what I got is wget --recursive to get all sites, but that doesn't append the parameter ...
user avatar
3 votes
1 answer
14k views

When using curl to http post a xml file to a url, how to set the parameter?

I want to use curl to http post a file to a url like: curl -d myfile.xml http://www.example.dom/post/file/here Can I set the parameter I want the file to be posted with? Like if it was a ...
user27449's user avatar
  • 6,920
47 votes
3 answers
219k views

How to send POST with body, headers, and HTTP params using cURL?

I found a lot of examples on how to use simple POST commands in cURL, but I didn't find examples on how to send full HTTP POST commands, which contain: Headers (Basic Authentication) HTTP Params (s=1&...
user71020's user avatar
  • 607
0 votes
1 answer
741 views

Only output curl result dependent on mime-type

Is it possible to make curl only save the output of curl -o if the file is of a certain mime-type? For example, I might use the following command to download a bunch of images off a server... curl -...
Alex Coplan's user avatar
3 votes
2 answers
10k views

No response from CURL at Windows Command Prompt

I've tried following Run cURL commands from Windows console but typing in any curl command returns nothing. See below - I just get back the command prompt. c:\curl>curl curl: try 'curl --...
bryankym's user avatar