Questions tagged [curl]
curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).
716
questions
0
votes
2
answers
2k
views
cURL or xargs: show progress
I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into ...
4
votes
1
answer
2k
views
Unable to resolve host using curl/wget while page loads in browser
The problem I am facing is that while I open a link in my browser it loads quickly but whenever I try using the link in terminal with
for e.g. curl -0 www.google.com or
wget www.google....
1
vote
0
answers
404
views
the right options to traverse/download the pages/directories of a subdomain
Let's suppose exist a site with the following directories (subdomain)
index.php
|-sub1
|-index.php
|-sub1sub1
|-index.php
|-other.php
|-sub1sub1sub1
|-sub2
|-index.php
|- …....
16
votes
4
answers
49k
views
curl http_code of 000
I have a shell script that I use to monitor loading times and response codes on my live server cluster. It runs a total of 250 iterations every 5 minutes, distributed across 10 servers and 6 sites. It ...
6
votes
4
answers
14k
views
Curl Compressed Command not Recognized in Windows
I saw this post and successfully added curl to my Windows command prompt as instructed.
However, when I attempted to run curl --compressed, it says:
"the installed libcurl version doesn't support ...
6
votes
2
answers
8k
views
Why sudo curl ignores proxy settings?
$ echo $http_proxy
http://my.proxy.com
$ curl -v http://files.com/a.txt
* About to connect() to proxy my.proxy.com
# Correct downloading
$ sudo echo $http_proxy
http://my.proxy.com
$sudo curl -v ...
3
votes
1
answer
4k
views
curl: (7) couldn't connect to host
I know some people have asked this question before but I don't understand some of the answers. I'm a Mac user trying to instal Homebrew but I keep getting this error message:
curl: (7) couldn't ...
1
vote
0
answers
256
views
Download web server structure with empty files
I want to make a mirror of a Web server, but downloading the actual files will take too long. So I thought of having just the directory and file structure, and when I need the actual contents of the ...
6
votes
1
answer
3k
views
How did Google get on my Mac?
I am running a MacBook Pro, and have never installed Chrome, Google Earth, or anything blatantly Google.
Just installed Little Snitch (are there no good free firewalls for Mac?) and see that CURL is ...
2
votes
2
answers
7k
views
How do I unzip a compressed file as it's downloading?
I'm downloading a large file that's not an archive, and I want to combine the tasks of downloading and decompressing.
How can I do them simultaneously?
5
votes
1
answer
3k
views
How to make alias/function in .bash_profile for `curl` that takes an argument?
I found this tip super helpful, for using curl to resume an interrupted file copy.
The full syntax is:
curl -C - -O file:///Volumes/path/to/file
I would like to add an alias or function called ...
0
votes
1
answer
3k
views
How do I enable the curl progress bar in cygwin?
When you curl a file, you get this progress bar whether you're uploading or downloading a file.
% Total % Received % Xferd Average Speed Time Time Time Current
...
2
votes
1
answer
979
views
curl works at the command line, but how do i put it in a php file?
The following curl format works for me from the command line. The line below is just sample data, but the format works.
curl -u [email protected]:1234567 https://api.email.com/status
How do I ...
0
votes
2
answers
2k
views
What should I do in order to build curl without an error?
This failed when I run ./buildconf
The error information is as follows:
[mirror@home curl]$ ls
acinclude.m4 CMakeLists.txt GIT-INFO MacOSX-Framework packages TODO-RELEASE
...
1
vote
2
answers
2k
views
What tool does yum use to download rpm packages from remote server?
My system is centos, and now I am installing packages using yum command.
Updating for dependencies:
apr-util i386 1.3.9-1.el5 lxcenter-updates 203 k
gcc-gfortran ...
4
votes
4
answers
20k
views
Silence curl with >/dev/null 2>&1 when it's piped
So I have a curl that is piped to a grep and a sed.
Where would I apply the >/dev/null 2>&1 ?
curl www.site.com | grep stuff | sed "other stuff"
At the very end or after the curl?
15
votes
1
answer
14k
views
How do I remotely fetch files from redirected URLs from a terminal?
I want to fetch a tarball of this python library from terminal.
https://github.com/simplegeo/python-oauth2/downloads
However, I cannot simply call
wget https://github.com/simplegeo/python-oauth2/...
1
vote
1
answer
1k
views
Grab source from Google Code using wget or curl?
Is it possible to grab the source-code from a Google Code repository (hg) using wget or curl?
Else using Python?
Thanks for all suggestions
FYI: I'm on a shared WebFaction host, so can't install hg
9
votes
1
answer
20k
views
CURL File with post parameter
How to make a request for uploading a file to facebook using graph api in curl
I am making this request but getting Error:
curl https://graph.facebook.com/<id>/photos -F "[email protected]" -d "...
32
votes
3
answers
88k
views
Why can't cURL properly verify a certificate on Windows?
When I try to use Curl on windows, to retrieve an https url, I get the dreaded "connection error (60)."
The exact error message is:
curl: (60) SSL certificate problem, verify that the CA cert is ...
0
votes
1
answer
26k
views
Curl Couldn't resolve host for HTTPS
I am trying to use curl on a Red Hat Enterprise Linux Server release 6.1. When I try to use URL with HTTPS it gives an error message "Couldn't resolve host". But for the same URL when tried with HTTP (...
1
vote
2
answers
2k
views
test of ICMP block
In my bash scripts I have been using something like:
until fping -u google.com; do echo "$0[$$] Network/DNS down?? $(date)" 1>&2 && sleep $(($RANDOM%(1 + ++trynum * 1) +1)).222; done
...
0
votes
1
answer
2k
views
Why can't I fetch this sites contents with curl?
I'm trying to download the contents of https://aarhustech.itslearning.com/elogin/default.aspx with curl, but I get this error:
$ curl https://aarhustech.itslearning.com/elogin/default.aspx
curl: (35) ...
0
votes
1
answer
470
views
apt-get install new packeges, avoid others
Look at this entry:
root@vivid-75:~# apt-get --no-install-recommends install curl
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra ...
1
vote
2
answers
31k
views
How can I use curl with a SSL URL to get a 200 OK?
I'm trying to use curl to get the HTTP status of a GET request:
curl --insecure --silent --show-error --connect-timeout 1 -I https://host:8443/health
Note: I'm using the --insecure flag in this ...
0
votes
1
answer
2k
views
How to make curl reconnect after 60"?
Sometimes, curl freezes:
$ curl -O 'http://www.site.com/index.html'
www.site.com/index.html --> index.html
% Total % Received % Xferd Average Speed Time Time Time Current
...
0
votes
1
answer
2k
views
Apache and PHP with cURL
I am running Sun server with Solaris 10/SPARC.
I had/have Apache 1.37 with PHP 5.2.1 running as a dynamic extension, and want to enable cURL support. I have tried to re-compile PHP with the --with-...
2
votes
1
answer
15k
views
curl failed setting cipher list
I'm trying to make curl use GOST2001-GOST89-GOST89 cipher which is available and usable by OpenSSL but keep getting failed setting cipher list error despite the fact that curl sees gost engine and can ...
3
votes
1
answer
2k
views
VPN routing issue in linux- how to make applications that can only bind to IP addresses work?
Right now I have a PPTP VPN set up on my remote ubuntu VPS. I would like to have certain applications use the VPN to access the internet (rtorrent, irssi, among others). Currently, it is set up like ...
0
votes
2
answers
2k
views
How to use wget to grab copy of Google Code site documents?
I have a Google Code project which has a lot of wiki'ed documentation. I would like to create a copy of this documentation for offline browsing. I would like to use wget or a similar utility.
I have ...
2
votes
2
answers
2k
views
command line curl - file naming example.com/[1-10].txt => 01.txt through 10.txt
Is it possible to set the variable portion of the file name output to be a specific length and fill the extra spaces with zeros.
I want things to be ordered for other users, so I would like
curl ...
0
votes
1
answer
525
views
Allow Incoming Responses Apache. On Ubuntu 11.10 - Curl
I'm trying to get a Curl Response from an outside server, however I noticed I cant neither PING the server in question nor connect to it.
I tried disabling the iptables firewall but I had no success.
...
2
votes
1
answer
5k
views
Cannot connect to website - SSL handshaking fails
So I cannot connect to certain websites. Just a few, most are OK. The one I really care about is paypal.com.
I have done the usual things. Let's see:
Cleared browsing history
Tried different ...
1
vote
1
answer
20k
views
How to stop running download process by CURL?
I forgot to use > filename after curl operation.
linux console is giving contiguously garbage data(hopefully downloading and displaying the content on console).
How can we stop this process (ctrl-c ...
2
votes
1
answer
11k
views
Downloading only images using curl or wget?
UPDATED:
I've found using this Bash script fixes the problem of having GIF files with a .jpg extension.
I'm attempting to download images from a forum who's url uses the following format:
http://...
1
vote
1
answer
1k
views
How to use delicious API from command line
I'm trying to add a delicious bookmark from a shell script, but keep getting the mysterious "something went wrong" error. I've checked the official spec and several examples and everything looks right....
1
vote
3
answers
1k
views
curl (7) permission denied on cgi script
Can't understand which permissions could be denied. Running cgi script from command line works fine.
http_proxy="" curl http://url/ -o file
Produces curl(7): permission denied
(same issues with ...
3
votes
2
answers
7k
views
Can I encrypt a username password combo before using it with cURL?
I'm making a shell script that reached out to a web service with basic auth. I don't want to have to put in my username and password every time the script runs, but I also don't want to have the combo ...
1
vote
1
answer
3k
views
Submitting a form using curl
Forgive me for the simple question... I haven't used curl before. I'm trying to automatically download some data from this form: http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500
which ...
2
votes
0
answers
2k
views
Restore default CA Certificates on Mac OSX 10.7
I've been screwing around with my Mac, trying to install self-signed certificate as trusted for our WAN site (for which we can't justify paying for a real signed certificate)
I've broken something ...
1
vote
1
answer
1k
views
How do I install hiphop-php in CentOS 6?
I've been trying to install hiphop-php on CentOS 6 with no luck. I found an .rpm for it but it fails dependency checks on boost which fails to install because it depends on curl which fails to ...
4
votes
1
answer
1k
views
Can cURL use a keyring for client certificates?
I'm using cURL to fetch from an HTTPS location, which requires a client certificate. I have my .curlrc set up to tell cURL where it is, but I don't want to leave my unencrypted private key just lying ...
8
votes
1
answer
11k
views
How to see HTTP request header by curl?
I can see HTTP header response by curl -I command.
Is there any option to see the HTTP request header sent?
curl -I www.google.com
HTTP/1.1 200 OK
Date: Sat, 31 Dec 2011 00:55:53 GMT
Expires: -1
...
0
votes
1
answer
19k
views
How to install CURL on Debian Squeeze?
I just can't get CURL to install on Debian Squeeze... keeps giving me this:
libtool: compile: gcc -DHAVE_CONFIG_H -I../include/curl -I../include -I../include -I../lib -I../lib -g0 -O2 -Wno-system-...
2
votes
2
answers
8k
views
How can I curl the output of another command
I want to pass curl the output from awk
./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | curl OUTPUT_FROM_AWK | grep myString
How can I achieve this?!
1
vote
1
answer
686
views
Scrape a site with curl/wget appending a parameter to each site
I need to scrape a website for all sites and append an Paramater, e.g.
www.theSite.com/allpages?myPara=us
Now what I got is wget --recursive to get all sites, but that doesn't append the parameter ...
3
votes
1
answer
14k
views
When using curl to http post a xml file to a url, how to set the parameter?
I want to use curl to http post a file to a url like:
curl -d myfile.xml http://www.example.dom/post/file/here
Can I set the parameter I want the file to be posted with? Like if it was a ...
47
votes
3
answers
219k
views
How to send POST with body, headers, and HTTP params using cURL?
I found a lot of examples on how to use simple POST commands in cURL, but I didn't find examples on how to send full HTTP POST commands, which contain:
Headers (Basic Authentication)
HTTP Params (s=1&...
0
votes
1
answer
741
views
Only output curl result dependent on mime-type
Is it possible to make curl only save the output of curl -o if the file is of a certain mime-type?
For example, I might use the following command to download a bunch of images off a server...
curl -...
3
votes
2
answers
10k
views
No response from CURL at Windows Command Prompt
I've tried following Run cURL commands from Windows console but typing in any curl command returns nothing.
See below - I just get back the command prompt.
c:\curl>curl
curl: try 'curl --...