curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

1
vote
0answers
63 views

How do I log in to “The Cloud” wifi hotspots using curl?

In England at least, there's this accursèd thing called 'The Cloud', which increasingly provides all public wifi hotspots. Has anyone already knocked up a curl command/bash script/etc to log in to it ...
0
votes
1answer
148 views

File operations on FTPS server with cURL

I need to perform some file transfer operations on some remote FTPS server that uses certificate authentication from a Windows 2008 scripting environment, so that's why I'm stuck with cURL pretty ...
0
votes
1answer
87 views

A Script That is Aware of Remote File Change

curl -z pci.ids --remote-time --silent -f -L -O http://pciids.sourceforge.net/v2.2/pci.ids This Code Checks For The Remote File Change and Download If Remote File Has Been Modified but Is There a ...
0
votes
1answer
537 views

Command line utility to convert html to text

What would be the best command line utility to convert html to text? I want to curl a query to google, convert the output html to text and read converted text on my terminal. I am using rhel6.
0
votes
1answer
829 views

cURL: multiple POST requests while reusing the TCP connection

If multiple URLs are passed to cURL, it reuses the TCP connection when possible. For example: curl -o 1.jpg http://example.com/1.jpg -o 2.jpg http://example.com/2.jpg I need to do the same, but ...
2
votes
1answer
54 views

Shell commands executed from Vim behave differently

I have mapped a key in Vim to run curl-based tests on my API with :map ,t :w\|:!quizzical tests.sh <CR>. The tests run as expected when I press the key; however, the tests which take longer to ...
4
votes
2answers
12k views

how to get and use certificates with curl

1) Entering the following command curl -k email@mail.com:password https://www.site.com Returns the following error <HTML><HEAD><meta http-equiv="content-type" ...
1
vote
1answer
589 views

Omnisharp for Emacs doesn't work

I tried recently to work with C# with Emacs24. I found https://github.com/sp3ctum/omnisharp-emacs I installed it with MELPA. Then I run it. Nothing happened. I wonder what went wrong. I think I don't ...
-1
votes
1answer
242 views

Curl with ipv6 not working by default

I am trying to curl ipv6 adresses, but without success. If i try lets say: curl https://google.com I wait for a timeout and get network unreachable If i force ipv4, everything is fine. ...
2
votes
0answers
398 views

loop curl could not resolve host

I've had working script, which suddenly doesn't works. It contains something like this: while read url; do curl $url done < links links file: http://duckduckgo.com http://google.com And I ...
0
votes
2answers
464 views

SFTP - Recursively fetch new files

I have a remove file system structure like this: + /measure + / 2013-09 + / 2013-09-04 + / fooinstrument + result03343445845.csv + ...
1
vote
4answers
790 views

How does ping figure out the IP address of a website?

I'm trying to understand better the different network-query tools such as nc, curl, whois, dig, nslookup and at the same time a bit more about the architecture of internet requests, servers, and the ...
0
votes
1answer
329 views

Error with curl

I'm having this error with curl: curl https://www.google.com -v * About to connect() to www.google.com port 443 (#0) * Trying 2607:f8b0:400e:c01::67... * Failed to connect to ...
3
votes
1answer
1k views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
3
votes
1answer
333 views

Why is there a very long 6 second delay in start of Curl

I did a full install of Cygwin on my Windows 7 x64 machine. After each machine reboot, Curl can take around 6 seconds to start (even if no command line options were specified). On subsequent uses of ...
6
votes
1answer
478 views

How can I use a command line tool like wget to log into an OpenID site?

Specifically, I would like to be able to download certain pages from my user profile on the various SE sites. I would, however, like to do this automatically (using a cron job), from the command line ...
0
votes
0answers
254 views

Make curl work on ssh connection

On my windows xp machine, I can connect to a remote host via putty and psftp. However I fail to establish a connection with curl. When I enter curl -v --key priv_key.ppk --pubkey pub_key.pub ...
1
vote
1answer
590 views

Have a txt file keep it's line breaks after being send over FTP via Curl

I'm using Curl to send a file over FTP. It is just a generated text file from a screen hardcopy command. When I cat the file, it displays correctly, it looks like this: System up for 14844 1, ...
0
votes
2answers
1k views

tar piped to curl: This does not look like a tar archive

curl -L http://download.transmissionbt.com/files/transmission-2.80.tar.xz -s -o - | tar -xf - The error: tar: This does not look like a tar archive tar: Skipping to next header tar: Exiting with ...
1
vote
1answer
4k views

How can I send a http get request from command line under WinPE

I need to send a http get request under WinPE. My first thought was curl.exe. It works well under Windows but not WinPE. I committed curl.exe to boot.wim but when I tested it under WinPE, it returns ...
1
vote
0answers
149 views

Curl not working on OS X

I recently upgraded XCode and while I'm not sure if that caused the problem, I can't curl out of my computer anymore. This seems to break rvm and homebrew but git still works. I upgraded curl and ...
2
votes
1answer
379 views

What is more transparent, curl or wget in this web crawler project?

I have files on a Windows 7, xampp server (using altered port, 181). I am triggering the script from another computer (linux) using curl http://xxx.xxx.xxx:181/phpproject.php This PHP project is a ...
0
votes
2answers
286 views

Using curl on centos 5.9 for the same server hosted sites giving 404 [closed]

I am using the following command on my ssh terminal- curl -v -L http://mysite/path to request some data. However, its returning a 404 error(the site is hosted on the same server). But using curl ...
0
votes
1answer
226 views

Can't copy website with curl

I want to get all the images from a website behind our company firewall, similar to what was asked here. I can access the site from Firefox on OS-X, but when I try to use curl I get: You ...
2
votes
1answer
447 views

Do curl and wget open gzipped files automatically?

I made two http calls to the same gzipped resource from two different machines - RHEL and OSX. curl http://example.com/file.gz The first call returned an expected binary output: /�IQ���} The ...
0
votes
0answers
33 views

Curl ignores the -r switch when -C is present

I wanted to download a part of a file. (Let's say http://mysite.com/myfile.zip) So I used this command: curl -r 200000-400000 -O http://mysite.com/myfile.zip Unfortunately my Internet ...
4
votes
1answer
2k views

Specifying minor TLS version when using curl

Is there a way to specify curl to use a specific TLS version? Like 1.1 or 1.2? I can see only sslv3 and tlsv1 options in command help. I took latest src and compiled it with openssl 1.0.1e. Still dont ...
1
vote
0answers
124 views

Permissions tar extraction to nfs share from a Perl script

As part of an installation procedure of a piece of software, I use: curl -s <url-to-targz> | tar -p -x -z -C / inside a Perl script (I use qx($command) or system($command), either will ...
0
votes
1answer
4k views

Curl Returns “Connection Refused” when trying to access localhost

I am running Xampp (Apache, Mysql and PHP) on my Windows Vista computer. I have curl installed on my machine. When I try to curl my localhost in the command prompt, I get the following: curl -v -i ...
0
votes
1answer
175 views

Login webpage using wget

I have problems with logging in to this webpage using wget: http://dealers.volare-kinderfietsen.nl/index.php?dispatch=auth.login_form Normally this is no problem for me. I just want to store the ...
1
vote
1answer
1k views

cURL SFTP upload speed only 50% compared to command line SFTP

I need to use cURL to upload some files to an SFTP server, it has to be using cURL because I'll be later using cURL_multi etc to transfer files simultaneously (unless anyone knows a command line ...
2
votes
2answers
669 views

I am not able access certain website [closed]

I am trying to access www.psychologytoday.com and it fails every time. I tried proxysites and still but every time connection got refused. I've checked Google cached page of the same. Only a few ...
2
votes
0answers
147 views

Curl --insecure error in Cygwin (works in Linux)

Curl is returning an error in Cygwin. How do I eliminate this error? curl --insecure -X POST -d '<run><log ...
0
votes
1answer
490 views

How to add self signed certificate to certificate bundle?

How to add self signed certificate to certificate bundle so that the Curl http client can verify the self signed certificate as valid one?
1
vote
2answers
5k views

Upload folder with cURL and FTP using Batch file on Windows?

I am trying to upload a whole folder, including it's sub-folders and files to a folder on my server from a Windows batch (.bat) file. Below is the command I have for the Upload part... curl -T ...
1
vote
1answer
1k views

Upload a file with a comma in its name with curl

How can I upload a file named yes, this filename has a comma.txt with the cli curl? You would normally do this to upload a file with curl: curl --progress-bar -F "fileUpload=@filename.txt" However ...
18
votes
2answers
6k views

Can I make cURL fail with an exitCode different than 0 if the HTTP status code is not 200?

I was always assuming that when curl got an HTTP 500 response it was returning an exit code that meant failure (!= 0), but that seems to be not the case. Is there a way I can I make cURL fail with an ...
-1
votes
1answer
320 views

cURL for Mac OS X Lion

I am looking for cURL for Mac OS X 10. I have found versions that go up to OS X 8 (Link), but no later. Is there a version for OS X 10 (Mac OS X Lion 10.7.5 (11G63)) or another tool?
0
votes
3answers
743 views

Batch - Get the url of to a file from html document

I want to use curl to download the latest version of this file. The site has a robots.txt, however, which is what I think is stopping me from just using curl -L -z WorldGuard.zip ...
0
votes
1answer
105 views

Why doesn't my script work?

The script I am using is FOR /F "tokens=91 delims=<>" %%v IN ('curl http://ci.onarandombox.com/job/Multiverse-Core/lastStableBuild/api/xml') DO ECHO %%v What I want it to do is search the output ...
0
votes
0answers
143 views

cURL wont download for me?

OS X 10.8.3, from a terminal I am executing: $ curl -OCv http://<URL>/<file>.dmg I get (for 2 seconds) % Total % Received % Xferd Average Speed Time Time Time Current ...
3
votes
1answer
4k views

Download a file from the latest stable Jenkins build

Using a batch file and wget/curl, how can I download the Multiverse-Core-.jar from here? http://ci.onarandombox.com/job/Multiverse-Core/lastStableBuild/artifact/target/ I want to use the timestamping ...
1
vote
1answer
294 views

curl ifconfig.me and ifconfig showing different results

If I run curl ifconfig.me I get one IP, and if I run ifconfig, I get other IP's. Why is it so ?
0
votes
1answer
662 views

curl (48) An unknown option was passed in to libcurl, cannot install composer

I'm trying to install composer via cURL but I cannot find how to download the phar archive. I've installed cURL from source activating in it's build ssl and ssh compatibility; I need composer to have ...
1
vote
2answers
330 views

cURL unable to interpret hostnames, proxy issue?

When using cURL from my ubuntu machine to merely GET from a remote machine, I seem unable to resolve the hostname and just get a Proxy Issue - The proxy has encountered an error (Error 21: Server ...
1
vote
2answers
458 views

How to stop cURL from writing over downloaded files

I'm using $ xargs -n 1 curl -O < gwurls.txt to grab a long list of files. Unfortunately, the site I'm grabbing from relies on the path to provide uniqueness, so -O doesn't know the difference ...
2
votes
1answer
3k views

How to split the HTTP error code from the contents in cURL?

Yes, this is related to Getting curl to output HTTP status code? but unfortunately not the same. In a script I would like to run: curl -qSfsw %{http_code} URL where the -f option ensures that the ...
1
vote
0answers
515 views

Libcurl-devel on Mac OS X

I am trying to build an application that depends on libcurl version 7.21 or newer on Mac OS X 10.5.8. I tried downloading and installing the latest version from libcurl's website, but the application ...
2
votes
0answers
119 views

curl works but ping/urllib2 don't, in a virtual machine

I have a Crunchbang 10 VM running on VirtualBox on a Windows 7 Home host. I have a python script that uses urllib2 that was failing, and discovered that: ordinary browsing and curl work normally on ...
1
vote
1answer
112 views

Is it possible to send the output of -w and -o Curl streams to different files? How?

I'd like to run a curl command, grep the output (response payload) for some keywords, then also print some metrics via the -w option. I think this would be easy to do if I could direct the -o ...