curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

0
votes
0answers
19 views

Can One Extract a Hyphenated Filename File From a Server Using Curl?

The images I'm dealing with have file names like part-1.jpg, object-a.jpg, Aspect-0.jpg. What I'm wondering is what I put in place of ???: curl -O http://server/[???].jpg
0
votes
0answers
540 views

Libcurl not usable on mac os x

I'm running the configure command to install a particular software package, but it fails when it reaches it's libcurl requirement. This is the output it gives: checking for the version of ...
1
vote
1answer
108 views

Accessing a remote machine with or without ssh tunneling

I just want to educate myself on an interesting issue here. When I curl a URL I get this error curl "http://myserver:8888/payload" curl: (7) couldn't connect to host However when I setup a tunnel ...
0
votes
1answer
676 views

cURL: multiple POST requests while reusing the TCP connection

If multiple URLs are passed to cURL, it reuses the TCP connection when possible. For example: curl -o 1.jpg http://example.com/1.jpg -o 2.jpg http://example.com/2.jpg I need to do the same, but ...
0
votes
0answers
332 views

Apahce proxy throws error connecting to a third party

Connection to a certain thirdparty service via php curl throws this error: CURL error: Received HTTP code 403 from proxy after CONNECT The apahe error log on the proxy throws this: Connect to ...
0
votes
0answers
223 views

curl with ftp URL works for SSL, but ftps URL doesn't: unknown protocol

I compiled and installed ProFTPD on my Mac for testing purposes and generated self-signed SSL certificates because I want to test ftps. When I do: curl --ftp-ssl-reqd --insecure ...
1
vote
0answers
227 views

pipe unlimited curl stdout into chunked files

I've got an HTTP-endpoint that serves an unlimited stream of events. Now I'd like to record that stream using curl, but splitting the recorded files at the same time - preferably based on the line ...
0
votes
1answer
127 views

File operations on FTPS server with cURL

I need to perform some file transfer operations on some remote FTPS server that uses certificate authentication from a Windows 2008 scripting environment, so that's why I'm stuck with cURL pretty ...
1
vote
0answers
57 views

How do I log in to “The Cloud” wifi hotspots using curl?

In England at least, there's this accursèd thing called 'The Cloud', which increasingly provides all public wifi hotspots. Has anyone already knocked up a curl command/bash script/etc to log in to it ...
48
votes
5answers
100k views

Run cURL commands from Windows console

Is there a way to install cURL in windows in order to run cURL commands from the command prompt?
0
votes
1answer
74 views

A Script That is Aware of Remote File Change

curl -z pci.ids --remote-time --silent -f -L -O http://pciids.sourceforge.net/v2.2/pci.ids This Code Checks For The Remote File Change and Download If Remote File Has Been Modified but Is There a ...
0
votes
1answer
432 views

Command line utility to convert html to text

What would be the best command line utility to convert html to text? I want to curl a query to google, convert the output html to text and read converted text on my terminal. I am using rhel6.
0
votes
2answers
7k views

curl windows command line [duplicate]

Possible Duplicate: Run cURL commands from Windows console I saw this questions I downloaded the CURL and placed it under C. Still I can't run it from CMD (with or without full path). as ...
2
votes
1answer
53 views

Shell commands executed from Vim behave differently

I have mapped a key in Vim to run curl-based tests on my API with :map ,t :w\|:!quizzical tests.sh <CR>. The tests run as expected when I press the key; however, the tests which take longer to ...
16
votes
2answers
5k views

Can I make cURL fail with an exitCode different than 0 if the HTTP status code is not 200?

I was always assuming that when curl got an HTTP 500 response it was returning an exit code that meant failure (!= 0), but that seems to be not the case. Is there a way I can I make cURL fail with an ...
-1
votes
1answer
225 views

Curl with ipv6 not working by default

I am trying to curl ipv6 adresses, but without success. If i try lets say: curl https://google.com I wait for a timeout and get network unreachable If i force ipv4, everything is fine. ...
2
votes
2answers
2k views

How do I unzip a compressed file as it's downloading?

I'm downloading a large file that's not an archive, and I want to combine the tasks of downloading and decompressing. How can I do them simultaneously?
1
vote
1answer
480 views

Omnisharp for Emacs doesn't work

I tried recently to work with C# with Emacs24. I found https://github.com/sp3ctum/omnisharp-emacs I installed it with MELPA. Then I run it. Nothing happened. I wonder what went wrong. I think I don't ...
0
votes
2answers
426 views

SFTP - Recursively fetch new files

I have a remove file system structure like this: + /measure + / 2013-09 + / 2013-09-04 + / fooinstrument + result03343445845.csv + ...
1
vote
2answers
89 views

Where to store short strings (with my key) on the internet?

Is there simple service to store strings under my key that can be used by bots? Requirements: Simple command line access, automatic posting allowed No need to keep some session with the service ...
1
vote
0answers
155 views

the right options to traverse/download the pages/directories of a subdomain

Let's suppose exist a site with the following directories (subdomain) index.php |-sub1 |-index.php |-sub1sub1 |-index.php |-other.php |-sub1sub1sub1 |-sub2 |-index.php |- ...
1
vote
4answers
752 views

How does ping figure out the IP address of a website?

I'm trying to understand better the different network-query tools such as nc, curl, whois, dig, nslookup and at the same time a bit more about the architecture of internet requests, servers, and the ...
2
votes
0answers
144 views

Curl --insecure error in Cygwin (works in Linux)

Curl is returning an error in Cygwin. How do I eliminate this error? curl --insecure -X POST -d '<run><log ...
0
votes
1answer
287 views

Error with curl

I'm having this error with curl: curl https://www.google.com -v * About to connect() to www.google.com port 443 (#0) * Trying 2607:f8b0:400e:c01::67... * Failed to connect to ...
3
votes
1answer
1k views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
3
votes
1answer
293 views

Why is there a very long 6 second delay in start of Curl

I did a full install of Cygwin on my Windows 7 x64 machine. After each machine reboot, Curl can take around 6 seconds to start (even if no command line options were specified). On subsequent uses of ...
1
vote
1answer
1k views

How do I install hiphop-php in CentOS 6?

I've been trying to install hiphop-php on CentOS 6 with no luck. I found an .rpm for it but it fails dependency checks on boost which fails to install because it depends on curl which fails to ...
0
votes
0answers
229 views

Make curl work on ssh connection

On my windows xp machine, I can connect to a remote host via putty and psftp. However I fail to establish a connection with curl. When I enter curl -v --key priv_key.ppk --pubkey pub_key.pub ...
1
vote
1answer
545 views

Have a txt file keep it's line breaks after being send over FTP via Curl

I'm using Curl to send a file over FTP. It is just a generated text file from a screen hardcopy command. When I cat the file, it displays correctly, it looks like this: System up for 14844 1, ...
0
votes
2answers
959 views

tar piped to curl: This does not look like a tar archive

curl -L http://download.transmissionbt.com/files/transmission-2.80.tar.xz -s -o - | tar -xf - The error: tar: This does not look like a tar archive tar: Skipping to next header tar: Exiting with ...
2
votes
0answers
110 views

curl works but ping/urllib2 don't, in a virtual machine

I have a Crunchbang 10 VM running on VirtualBox on a Windows 7 Home host. I have a python script that uses urllib2 that was failing, and discovered that: ordinary browsing and curl work normally on ...
1
vote
1answer
3k views

How can I send a http get request from command line under WinPE

I need to send a http get request under WinPE. My first thought was curl.exe. It works well under Windows but not WinPE. I committed curl.exe to boot.wim but when I tested it under WinPE, it returns ...
1
vote
0answers
128 views

Curl not working on OS X

I recently upgraded XCode and while I'm not sure if that caused the problem, I can't curl out of my computer anymore. This seems to break rvm and homebrew but git still works. I upgraded curl and ...
2
votes
1answer
342 views

What is more transparent, curl or wget in this web crawler project?

I have files on a Windows 7, xampp server (using altered port, 181). I am triggering the script from another computer (linux) using curl http://xxx.xxx.xxx:181/phpproject.php This PHP project is a ...
0
votes
2answers
259 views

Using curl on centos 5.9 for the same server hosted sites giving 404 [closed]

I am using the following command on my ssh terminal- curl -v -L http://mysite/path to request some data. However, its returning a 404 error(the site is hosted on the same server). But using curl ...
0
votes
1answer
209 views

Can't copy website with curl

I want to get all the images from a website behind our company firewall, similar to what was asked here. I can access the site from Firefox on OS-X, but when I try to use curl I get: You ...
0
votes
0answers
31 views

Curl ignores the -r switch when -C is present

I wanted to download a part of a file. (Let's say http://mysite.com/myfile.zip) So I used this command: curl -r 200000-400000 -O http://mysite.com/myfile.zip Unfortunately my Internet ...
2
votes
1answer
390 views

Do curl and wget open gzipped files automatically?

I made two http calls to the same gzipped resource from two different machines - RHEL and OSX. curl http://example.com/file.gz The first call returned an expected binary output: /�IQ���} The ...
1
vote
0answers
119 views

Permissions tar extraction to nfs share from a Perl script

As part of an installation procedure of a piece of software, I use: curl -s <url-to-targz> | tar -p -x -z -C / inside a Perl script (I use qx($command) or system($command), either will ...
0
votes
2answers
475 views

cURL or xargs: show progress

I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into ...
0
votes
1answer
166 views

Login webpage using wget

I have problems with logging in to this webpage using wget: http://dealers.volare-kinderfietsen.nl/index.php?dispatch=auth.login_form Normally this is no problem for me. I just want to store the ...
1
vote
1answer
1k views

cURL SFTP upload speed only 50% compared to command line SFTP

I need to use cURL to upload some files to an SFTP server, it has to be using cURL because I'll be later using cURL_multi etc to transfer files simultaneously (unless anyone knows a command line ...
2
votes
2answers
664 views

I am not able access certain website [closed]

I am trying to access www.psychologytoday.com and it fails every time. I tried proxysites and still but every time connection got refused. I've checked Google cached page of the same. Only a few ...
0
votes
1answer
433 views

How to add self signed certificate to certificate bundle?

How to add self signed certificate to certificate bundle so that the Curl http client can verify the self signed certificate as valid one?
1
vote
1answer
1k views

Upload a file with a comma in its name with curl

How can I upload a file named yes, this filename has a comma.txt with the cli curl? You would normally do this to upload a file with curl: curl --progress-bar -F "fileUpload=@filename.txt" However ...
-1
votes
1answer
300 views

cURL for Mac OS X Lion

I am looking for cURL for Mac OS X 10. I have found versions that go up to OS X 8 (Link), but no later. Is there a version for OS X 10 (Mac OS X Lion 10.7.5 (11G63)) or another tool?
0
votes
3answers
708 views

Batch - Get the url of to a file from html document

I want to use curl to download the latest version of this file. The site has a robots.txt, however, which is what I think is stopping me from just using curl -L -z WorldGuard.zip ...
0
votes
1answer
100 views

Why doesn't my script work?

The script I am using is FOR /F "tokens=91 delims=<>" %%v IN ('curl http://ci.onarandombox.com/job/Multiverse-Core/lastStableBuild/api/xml') DO ECHO %%v What I want it to do is search the output ...
3
votes
1answer
3k views

Download a file from the latest stable Jenkins build

Using a batch file and wget/curl, how can I download the Multiverse-Core-.jar from here? http://ci.onarandombox.com/job/Multiverse-Core/lastStableBuild/artifact/target/ I want to use the timestamping ...
0
votes
0answers
139 views

cURL wont download for me?

OS X 10.8.3, from a terminal I am executing: $ curl -OCv http://<URL>/<file>.dmg I get (for 2 seconds) % Total % Received % Xferd Average Speed Time Time Time Current ...