curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

1
vote
1answer
547 views

Have a txt file keep it's line breaks after being send over FTP via Curl

I'm using Curl to send a file over FTP. It is just a generated text file from a screen hardcopy command. When I cat the file, it displays correctly, it looks like this: System up for 14844 1, ...
0
votes
2answers
963 views

tar piped to curl: This does not look like a tar archive

curl -L http://download.transmissionbt.com/files/transmission-2.80.tar.xz -s -o - | tar -xf - The error: tar: This does not look like a tar archive tar: Skipping to next header tar: Exiting with ...
1
vote
1answer
3k views

How can I send a http get request from command line under WinPE

I need to send a http get request under WinPE. My first thought was curl.exe. It works well under Windows but not WinPE. I committed curl.exe to boot.wim but when I tested it under WinPE, it returns ...
1
vote
0answers
128 views

Curl not working on OS X

I recently upgraded XCode and while I'm not sure if that caused the problem, I can't curl out of my computer anymore. This seems to break rvm and homebrew but git still works. I upgraded curl and ...
2
votes
1answer
345 views

What is more transparent, curl or wget in this web crawler project?

I have files on a Windows 7, xampp server (using altered port, 181). I am triggering the script from another computer (linux) using curl http://xxx.xxx.xxx:181/phpproject.php This PHP project is a ...
0
votes
2answers
260 views

Using curl on centos 5.9 for the same server hosted sites giving 404 [closed]

I am using the following command on my ssh terminal- curl -v -L http://mysite/path to request some data. However, its returning a 404 error(the site is hosted on the same server). But using curl ...
0
votes
1answer
211 views

Can't copy website with curl

I want to get all the images from a website behind our company firewall, similar to what was asked here. I can access the site from Firefox on OS-X, but when I try to use curl I get: You ...
2
votes
1answer
395 views

Do curl and wget open gzipped files automatically?

I made two http calls to the same gzipped resource from two different machines - RHEL and OSX. curl http://example.com/file.gz The first call returned an expected binary output: /�IQ���} The ...
0
votes
0answers
32 views

Curl ignores the -r switch when -C is present

I wanted to download a part of a file. (Let's say http://mysite.com/myfile.zip) So I used this command: curl -r 200000-400000 -O http://mysite.com/myfile.zip Unfortunately my Internet ...
4
votes
1answer
2k views

Specifying minor TLS version when using curl

Is there a way to specify curl to use a specific TLS version? Like 1.1 or 1.2? I can see only sslv3 and tlsv1 options in command help. I took latest src and compiled it with openssl 1.0.1e. Still dont ...
1
vote
0answers
119 views

Permissions tar extraction to nfs share from a Perl script

As part of an installation procedure of a piece of software, I use: curl -s <url-to-targz> | tar -p -x -z -C / inside a Perl script (I use qx($command) or system($command), either will ...
0
votes
1answer
4k views

Curl Returns “Connection Refused” when trying to access localhost

I am running Xampp (Apache, Mysql and PHP) on my Windows Vista computer. I have curl installed on my machine. When I try to curl my localhost in the command prompt, I get the following: curl -v -i ...
0
votes
1answer
166 views

Login webpage using wget

I have problems with logging in to this webpage using wget: http://dealers.volare-kinderfietsen.nl/index.php?dispatch=auth.login_form Normally this is no problem for me. I just want to store the ...
1
vote
1answer
1k views

cURL SFTP upload speed only 50% compared to command line SFTP

I need to use cURL to upload some files to an SFTP server, it has to be using cURL because I'll be later using cURL_multi etc to transfer files simultaneously (unless anyone knows a command line ...
2
votes
2answers
665 views

I am not able access certain website [closed]

I am trying to access www.psychologytoday.com and it fails every time. I tried proxysites and still but every time connection got refused. I've checked Google cached page of the same. Only a few ...
2
votes
0answers
144 views

Curl --insecure error in Cygwin (works in Linux)

Curl is returning an error in Cygwin. How do I eliminate this error? curl --insecure -X POST -d '<run><log ...
0
votes
1answer
437 views

How to add self signed certificate to certificate bundle?

How to add self signed certificate to certificate bundle so that the Curl http client can verify the self signed certificate as valid one?
1
vote
2answers
4k views

Upload folder with cURL and FTP using Batch file on Windows?

I am trying to upload a whole folder, including it's sub-folders and files to a folder on my server from a Windows batch (.bat) file. Below is the command I have for the Upload part... curl -T ...
1
vote
1answer
1k views

Upload a file with a comma in its name with curl

How can I upload a file named yes, this filename has a comma.txt with the cli curl? You would normally do this to upload a file with curl: curl --progress-bar -F "fileUpload=@filename.txt" However ...
16
votes
2answers
5k views

Can I make cURL fail with an exitCode different than 0 if the HTTP status code is not 200?

I was always assuming that when curl got an HTTP 500 response it was returning an exit code that meant failure (!= 0), but that seems to be not the case. Is there a way I can I make cURL fail with an ...
-1
votes
1answer
302 views

cURL for Mac OS X Lion

I am looking for cURL for Mac OS X 10. I have found versions that go up to OS X 8 (Link), but no later. Is there a version for OS X 10 (Mac OS X Lion 10.7.5 (11G63)) or another tool?
0
votes
3answers
711 views

Batch - Get the url of to a file from html document

I want to use curl to download the latest version of this file. The site has a robots.txt, however, which is what I think is stopping me from just using curl -L -z WorldGuard.zip ...
0
votes
1answer
101 views

Why doesn't my script work?

The script I am using is FOR /F "tokens=91 delims=<>" %%v IN ('curl http://ci.onarandombox.com/job/Multiverse-Core/lastStableBuild/api/xml') DO ECHO %%v What I want it to do is search the output ...
0
votes
0answers
139 views

cURL wont download for me?

OS X 10.8.3, from a terminal I am executing: $ curl -OCv http://<URL>/<file>.dmg I get (for 2 seconds) % Total % Received % Xferd Average Speed Time Time Time Current ...
3
votes
1answer
3k views

Download a file from the latest stable Jenkins build

Using a batch file and wget/curl, how can I download the Multiverse-Core-.jar from here? http://ci.onarandombox.com/job/Multiverse-Core/lastStableBuild/artifact/target/ I want to use the timestamping ...
1
vote
1answer
275 views

curl ifconfig.me and ifconfig showing different results

If I run curl ifconfig.me I get one IP, and if I run ifconfig, I get other IP's. Why is it so ?
0
votes
1answer
621 views

curl (48) An unknown option was passed in to libcurl, cannot install composer

I'm trying to install composer via cURL but I cannot find how to download the phar archive. I've installed cURL from source activating in it's build ssl and ssh compatibility; I need composer to have ...
1
vote
2answers
284 views

cURL unable to interpret hostnames, proxy issue?

When using cURL from my ubuntu machine to merely GET from a remote machine, I seem unable to resolve the hostname and just get a Proxy Issue - The proxy has encountered an error (Error 21: Server ...
1
vote
2answers
423 views

How to stop cURL from writing over downloaded files

I'm using $ xargs -n 1 curl -O < gwurls.txt to grab a long list of files. Unfortunately, the site I'm grabbing from relies on the path to provide uniqueness, so -O doesn't know the difference ...
2
votes
1answer
2k views

How to split the HTTP error code from the contents in cURL?

Yes, this is related to Getting curl to output HTTP status code? but unfortunately not the same. In a script I would like to run: curl -qSfsw %{http_code} URL where the -f option ensures that the ...
1
vote
0answers
475 views

Libcurl-devel on Mac OS X

I am trying to build an application that depends on libcurl version 7.21 or newer on Mac OS X 10.5.8. I tried downloading and installing the latest version from libcurl's website, but the application ...
2
votes
0answers
110 views

curl works but ping/urllib2 don't, in a virtual machine

I have a Crunchbang 10 VM running on VirtualBox on a Windows 7 Home host. I have a python script that uses urllib2 that was failing, and discovered that: ordinary browsing and curl work normally on ...
1
vote
1answer
102 views

Is it possible to send the output of -w and -o Curl streams to different files? How?

I'd like to run a curl command, grep the output (response payload) for some keywords, then also print some metrics via the -w option. I think this would be easy to do if I could direct the -o ...
2
votes
0answers
305 views

PHP : curl is and Not working in ubuntu10.04 and Nginx

I am running Nginx server on ubuntu 10.04 I am using the php version 5.3.2 .I installed php5-curl using apt-get install php5-curl. Unfortunately the curl package is not displaying in phpinfo() . ...
0
votes
1answer
158 views

Get Directory/file structure without payload

I need to recursively retreive the file/directory structure of an FTP server, getting only file names (without payload). Can you help me how to do this (curl, wget, ...)? The result should be the ...
1
vote
2answers
2k views

CURL Error version OPENSSL_1.0.1 'not found …'

I have the last few days, a concern with CURL. I get this error: curl: / lib/i386-linux-gnu/libssl.so.1.0.0: version `OPENSSL_1.0.1 'not found (required by / usr/lib/i386-linux-gnu / libcurl.so.4) ...
0
votes
0answers
417 views

Why can't I proxy SSL connections in Charles under Linux (ubuntu)?

I want to proxy some API requests over SSL from a PHP application using Charles Proxy which should allow me to view unencrytped requests. I've done this on Windows before but on Linux, I'm seeing the ...
4
votes
1answer
1k views

Why does sudo not work with curl?

I ran a command to download something with curl and it threw a permission denied error. So I did sudo curl and I got the same error... Why is that? I was able to run the install command without ...
0
votes
1answer
433 views

How to check for a value after curl or wget?

I need to curl/wget a page and make sure it returned a particular value (else return 1), something like this: curl http://example.com/something/run | grep -e '^success!$' || return 1 Which flags ...
5
votes
2answers
6k views

POSTing a file's contents with CURL

(The following is backstory, you can jump down to "Question:" if you'd like) There's a webpage for one of my courses that accepts input text (expected to be Python code) and then gives some tokenized ...
2
votes
2answers
113 views

Use of curl command via HTTP

Using curl command, how to transfer a file from one location to other via HTTP request?
0
votes
1answer
641 views

piping output from curl to rpm2cpio

I download alot of srpms to rummage through, I used to go through a long winded approach, root@server$ curl rpm -o file.rpm root@server$ rpm2cpio file.rpm | cpio -id but that leaves behind an ugly ...
0
votes
1answer
505 views

Downloading and extracting a .tar to a specific directory

I want to download a tar of a git repo and extract its contents to another folder. The following line (broken up for clarity) works: curl -L -0 nt-dotfiles.tar.gz ...
0
votes
4answers
121 views

Simple script parsing text, what is wrong here?

I'm a big user of https://www.grc.com/passwords.htm to get strong passwords. However, having to go to the site and manually copy the password every time gets old fast, so I decided to do a little ...
0
votes
1answer
153 views

CURL -I issue stop responding when contain “=”

i did this command : curl -I ...
1
vote
0answers
122 views

No output returned when curl ing any URL in Red Hat 6 but the same url works in lynx browser

I am trying to install some packages in Python using easy install. But i get a 'Download Error' for any package i try to install. So i tried to test if internet is accessible at all from the machine ...
0
votes
2answers
478 views

cURL or xargs: show progress

I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into ...
2
votes
1answer
318 views

Unable to resolve host using curl/wget while page loads in browser

The problem I am facing is that while I open a link in my browser it loads quickly but whenever I try using the link in terminal with for e.g. curl -0 www.google.com or wget ...
1
vote
0answers
156 views

the right options to traverse/download the pages/directories of a subdomain

Let's suppose exist a site with the following directories (subdomain) index.php |-sub1 |-index.php |-sub1sub1 |-index.php |-other.php |-sub1sub1sub1 |-sub2 |-index.php |- ...
0
votes
0answers
2k views

curl http_code of 000

I have a shell script that I use to monitor loading times and response codes on my live server cluster. It runs a total of 250 iterations every 5 minutes, distributed across 10 servers and 6 sites. It ...