curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

1
vote
2answers
343 views

How to stop cURL from writing over downloaded files

I'm using $ xargs -n 1 curl -O < gwurls.txt to grab a long list of files. Unfortunately, the site I'm grabbing from relies on the path to provide uniqueness, so -O doesn't know the difference ...
1
vote
1answer
2k views

How to split the HTTP error code from the contents in cURL?

Yes, this is related to Getting curl to output HTTP status code? but unfortunately not the same. In a script I would like to run: curl -qSfsw %{http_code} URL where the -f option ensures that the ...
1
vote
0answers
406 views

Libcurl-devel on Mac OS X

I am trying to build an application that depends on libcurl version 7.21 or newer on Mac OS X 10.5.8. I tried downloading and installing the latest version from libcurl's website, but the application ...
2
votes
0answers
94 views

curl works but ping/urllib2 don't, in a virtual machine

I have a Crunchbang 10 VM running on VirtualBox on a Windows 7 Home host. I have a python script that uses urllib2 that was failing, and discovered that: ordinary browsing and curl work normally on ...
1
vote
1answer
95 views

Is it possible to send the output of -w and -o Curl streams to different files? How?

I'd like to run a curl command, grep the output (response payload) for some keywords, then also print some metrics via the -w option. I think this would be easy to do if I could direct the -o ...
2
votes
0answers
257 views

PHP : curl is and Not working in ubuntu10.04 and Nginx

I am running Nginx server on ubuntu 10.04 I am using the php version 5.3.2 .I installed php5-curl using apt-get install php5-curl. Unfortunately the curl package is not displaying in phpinfo() . ...
0
votes
1answer
128 views

Get Directory/file structure without payload

I need to recursively retreive the file/directory structure of an FTP server, getting only file names (without payload). Can you help me how to do this (curl, wget, ...)? The result should be the ...
1
vote
2answers
2k views

CURL Error version OPENSSL_1.0.1 'not found …'

I have the last few days, a concern with CURL. I get this error: curl: / lib/i386-linux-gnu/libssl.so.1.0.0: version `OPENSSL_1.0.1 'not found (required by / usr/lib/i386-linux-gnu / libcurl.so.4) ...
0
votes
0answers
333 views

Why can't I proxy SSL connections in Charles under Linux (ubuntu)?

I want to proxy some API requests over SSL from a PHP application using Charles Proxy which should allow me to view unencrytped requests. I've done this on Windows before but on Linux, I'm seeing the ...
4
votes
1answer
782 views

Why does sudo not work with curl?

I ran a command to download something with curl and it threw a permission denied error. So I did sudo curl and I got the same error... Why is that? I was able to run the install command without ...
0
votes
1answer
304 views

How to check for a value after curl or wget?

I need to curl/wget a page and make sure it returned a particular value (else return 1), something like this: curl http://example.com/something/run | grep -e '^success!$' || return 1 Which flags ...
5
votes
2answers
4k views

POSTing a file's contents with CURL

(The following is backstory, you can jump down to "Question:" if you'd like) There's a webpage for one of my courses that accepts input text (expected to be Python code) and then gives some tokenized ...
2
votes
2answers
96 views

Use of curl command via HTTP

Using curl command, how to transfer a file from one location to other via HTTP request?
0
votes
1answer
462 views

piping output from curl to rpm2cpio

I download alot of srpms to rummage through, I used to go through a long winded approach, root@server$ curl rpm -o file.rpm root@server$ rpm2cpio file.rpm | cpio -id but that leaves behind an ugly ...
0
votes
1answer
409 views

Downloading and extracting a .tar to a specific directory

I want to download a tar of a git repo and extract its contents to another folder. The following line (broken up for clarity) works: curl -L -0 nt-dotfiles.tar.gz ...
0
votes
4answers
115 views

Simple script parsing text, what is wrong here?

I'm a big user of https://www.grc.com/passwords.htm to get strong passwords. However, having to go to the site and manually copy the password every time gets old fast, so I decided to do a little ...
0
votes
1answer
126 views

CURL -I issue stop responding when contain “=”

i did this command : curl -I ...
1
vote
0answers
107 views

No output returned when curl ing any URL in Red Hat 6 but the same url works in lynx browser

I am trying to install some packages in Python using easy install. But i get a 'Download Error' for any package i try to install. So i tried to test if internet is accessible at all from the machine ...
0
votes
2answers
399 views

cURL or xargs: show progress

I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into ...
2
votes
1answer
251 views

Unable to resolve host using curl/wget while page loads in browser

The problem I am facing is that while I open a link in my browser it loads quickly but whenever I try using the link in terminal with for e.g. curl -0 www.google.com or wget ...
1
vote
0answers
117 views

the right options to traverse/download the pages/directories of a subdomain

Let's suppose exist a site with the following directories (subdomain) index.php |-sub1 |-index.php |-sub1sub1 |-index.php |-other.php |-sub1sub1sub1 |-sub2 |-index.php |- ...
0
votes
0answers
2k views

curl http_code of 000

I have a shell script that I use to monitor loading times and response codes on my live server cluster. It runs a total of 250 iterations every 5 minutes, distributed across 10 servers and 6 sites. It ...
1
vote
2answers
204 views

Curl Compressed Command not Recognized in Windows

I saw this post and successfully added curl to my Windows command prompt as instructed. However, when I attempted to run curl --compressed, it says: "the installed libcurl version doesn't ...
0
votes
2answers
941 views

Why sudo curl ignores proxy settings?

$ echo $http_proxy http://my.proxy.com $ curl -v http://files.com/a.txt * About to connect() to proxy my.proxy.com # Correct downloading $ sudo echo $http_proxy http://my.proxy.com $sudo curl -v ...
2
votes
1answer
2k views

curl: (7) couldn't connect to host

I know some people have asked this question before but I don't understand some of the answers. I'm a Mac user trying to instal Homebrew but I keep getting this error message: curl: (7) couldn't ...
1
vote
0answers
102 views

Download web server structure with empty files

I want to make a mirror of a Web server, but downloading the actual files will take too long. So I thought of having just the directory and file structure, and when I need the actual contents of the ...
5
votes
1answer
412 views

How did Google get on my Mac?

Am running a MacBook Pro, and have never installed Chrome, Google Earth, or anything blatantly Google. Just installed Little Snitch (are there no good free firewalls for Mac?) and see that CURL is ...
2
votes
2answers
2k views

How do I unzip a compressed file as it's downloading?

I'm downloading a large file that's not an archive, and I want to combine the tasks of downloading and decompressing. How can I do them simultaneously?
3
votes
1answer
568 views

How to make alias/function in .bash_profile for `curl` that takes an argument?

I found this tip super helpful, for using curl to resume an interrupted file copy. The full syntax is: curl -C - -O file:///Volumes/path/to/file I would like to add an alias or function called ...
0
votes
1answer
754 views

How do I enable the curl progress bar in cygwin?

When you curl a file, you get this progress bar whether you're uploading or downloading a file. % Total % Received % Xferd Average Speed Time Time Time Current ...
2
votes
1answer
209 views

curl works at the command line, but how do i put it in a php file?

The following curl format works for me from the command line. The line below is just sample data, but the format works. curl -u email@email.com:1234567 https://api.email.com/status How do I ...
0
votes
2answers
640 views

What should I do in order to build curl without an error?

This failed when I run ./buildconf The error information is as follows: [mirror@home curl]$ ls acinclude.m4 CMakeLists.txt GIT-INFO MacOSX-Framework packages TODO-RELEASE ...
0
votes
1answer
237 views

What tool does yum use to download rpm packages from remote server?

My system is centos, and now I am installing packages using yum command. Updating for dependencies: apr-util i386 1.3.9-1.el5 lxcenter-updates 203 k gcc-gfortran ...
1
vote
2answers
3k views

Silence curl with >/dev/null 2>&1 when it's piped

So I have a curl that is piped to a grep and a sed. Where would I apply the >/dev/null 2>&1 ? curl www.site.com | grep stuff | sed "other stuff" At the very end or after the curl?
1
vote
1answer
1k views

How do I remotely fetch files from redirected URLs from a terminal?

I want to fetch a tarball of this python library from terminal. https://github.com/simplegeo/python-oauth2/downloads However, I cannot simply call wget ...
1
vote
1answer
622 views

Grab source from Google Code using wget or curl?

Is it possible to grab the source-code from a Google Code repository (hg) using wget or curl? Else using Python? Thanks for all suggestions FYI: I'm on a shared WebFaction host, so can't install hg ...
5
votes
1answer
8k views

CURL File with post parameter

How to make a request for uploading a file to facebook using graph api in curl I am making this request but getting Error: curl https://graph.facebook.com/<id>/photos -F "source=@me.jpg" -d ...
10
votes
1answer
5k views

Why can't cURL properly verify a certificate on Windows?

When I try to use Curl on windows, to retrieve an https url, I get the dreaded "connection error (60)." The exact error message is: curl: (60) SSL certificate problem, verify that the CA cert ...
0
votes
1answer
2k views

Curl Couldn't resolve host for HTTPS

I am trying to use curl on a Red Hat Enterprise Linux Server release 6.1. When I try to use URL with HTTPS it gives an error message "Couldn't resolve host". But for the same URL when tried with HTTP ...
0
votes
2answers
496 views

test of ICMP block

In my bash scripts I have been using something like: until fping -u google.com; do echo "$0[$$] Network/DNS down?? $(date)" 1>&2 && sleep $(($RANDOM%(1 + ++trynum * 1) +1)).222; done ...
0
votes
1answer
844 views

Why can't I fetch this sites contents with curl?

I'm trying to download the contents of https://aarhustech.itslearning.com/elogin/default.aspx with curl, but I get this error: $ curl https://aarhustech.itslearning.com/elogin/default.aspx curl: (35) ...
0
votes
1answer
371 views

apt-get install new packeges, avoid others

Look at this entry: root@vivid-75:~# apt-get --no-install-recommends install curl Reading package lists... Done Building dependency tree Reading state information... Done The following extra ...
0
votes
1answer
2k views

How can I use curl with a SSL URL to get a 200 OK?

I'm trying to use curl to get the HTTP status of a GET request: curl --insecure --silent --show-error --connect-timeout 1 -I https://host:8443/health Note: I'm using the --insecure flag in this ...
0
votes
1answer
391 views

How to make curl reconnect after 60"?

Sometimes, curl freezes: $ curl -O 'http://www.site.com/index.html' www.site.com/index.html --> index.html % Total % Received % Xferd Average Speed Time Time Time Current ...
0
votes
1answer
520 views

Apache and PHP with cURL

I am running Sun server with Solaris 10/SPARC. I had/have Apache 1.37 with PHP 5.2.1 running as a dynamic extension, and want to enable cURL support. I have tried to re-compile PHP with the ...
0
votes
1answer
2k views

curl failed setting cipher list

I'm trying to make curl use GOST2001-GOST89-GOST89 cipher which is available and usable by OpenSSL but keep getting failed setting cipher list error despite the fact that curl sees gost engine and can ...
3
votes
1answer
2k views

VPN routing issue in linux- how to make applications that can only bind to IP addresses work?

Right now I have a PPTP VPN set up on my remote ubuntu VPS. I would like to have certain applications use the VPN to access the internet (rtorrent, irssi, among others). Currently, it is set up like ...
0
votes
2answers
1k views

How to use wget to grab copy of Google Code site documents?

I have a Google Code project which has a lot of wiki'ed documentation. I would like to create a copy of this documentation for offline browsing. I would like to use wget or a similar utility. I have ...
0
votes
2answers
709 views

command line curl - file naming example.com/[1-10].txt => 01.txt through 10.txt

Is it possible to set the variable portion of the file name output to be a specific length and fill the extra spaces with zeros. I want things to be ordered for other users, so I would like curl ...
0
votes
1answer
307 views

Allow Incoming Responses Apache. On Ubuntu 11.10 - Curl

I'm trying to get a Curl Response from an outside server, however I noticed I cant neither PING the server in question nor connect to it. I tried disabling the iptables firewall but I had no success. ...