curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

0
votes
0answers
188 views

Curl to download large CSV

I am using curl through Cygwin on Windows 7 to download content from Apache Solr in CSV format. This is my command which is running in a for loop curl -o ret-key-data-$i.csv $URL As the content is ...
0
votes
1answer
191 views

cURL very long delay before starting download from ftp

I'm trying to write a script to download a file, convert it, then execute a program on the converted file. For some reason curl takes over 1 minute to start the file transfer. Command: curl ...
1
vote
0answers
30 views

Bug or syntax change on Debian's “curl” 7.36? Mess when trying to save sequential files with brackets/hash

(Perhaps specifically 7.36.0-1, which is shown as the most recent available on my package manager). I think there was such a bug some time ago, but perhaps it was back in curl 7.33, I can't ...
1
vote
2answers
1k views

How to stop shell script if curl failed

I have script that used curl when i pass wrong parameters to script curl failed but script continue executing. I have tried use curl -f/--fail parameter but problem does not solved. What is the best ...
11
votes
2answers
8k views

how to use curl to verify if a site's certificate has been revoked?

To check if the certificate for google.com has been revoked, I tried the following command: curl https://www.google.com --cacert GeoTrust_Global_CA.pem --crlfile gtglobal.pem -v , but I got the ...
0
votes
0answers
356 views

Why is SSL broken for one domain on one computer?

I've been having a strange issue, where I cannot connect to https://www.humblebundle.com. Plain HTTP works fine, as does https://humblebundle.com. These all redirect to the above address. I haven't ...
0
votes
0answers
39 views

Gzipped file arrives unzipped

I've recently set up an ArchLinux system at work and added the ArchHaskell repo. This is far from the first time I do this. I have a number of systems where it's worked just fine, but this last time ...
1
vote
1answer
266 views

posting data to a web server using curl

I am trying to post data to a web sever using curl -d and curl -X POST. The command is getting succeeded but nothing is there in the file. Tried the following: [root@oooatis santoshi]# curl ...
1
vote
0answers
44 views

SOAP-request realization in DOS 6.22 16-bit

I have a task to develop some application providing SOAP-requests realization for DOS. I have already developed a similar application under Linux, using C++/STL and curl and openssl libraries. Is ...
3
votes
1answer
3k views

How can I download private repo from github?

What I want to do is to download private repository archive from GitHub, extract it, remove archive file and copy some directories that are inside downloaded project. I tried to use wget but I cannot ...
1
vote
1answer
562 views

How do I comment out a line in my .curlrc?

I frequently have to switch between two different auth token request headers, that I keep in my .curlrc as such: -H "X-Auth-Token: 12345678-1234" It would make my life easier, though, if I could ...
0
votes
2answers
467 views

curl shell command always returns error status 0 - even when fails on input errors

Example: trying to upload a non existing file - curl -d @no-real-file http://google.com | echo $? - returns 0. How can I make curl fail on input errors?
3
votes
1answer
633 views

How can I get a list of all messages in my gmail account in linux command line?

How can I get a list of all messages in my gmail account in linux command line? Ideally I would want to see something similar to: from:sender1@domain.com to:me@gmail.com Date:11/22/33 subj:Foo1 ...
0
votes
0answers
400 views

Choose the filename when using POST with curl?

I'd like to POST a file upload with curl, and choose the filename at request time. Here's the curl line I have right now, which works to upload the file: curl --form "upload_file=@content.txt" ...
0
votes
0answers
164 views

Pinging local machine with external IP

On a Gentoo server, I am trying to wget or curl a page from its url: wget http://example.com/test.html (I can get result for this url from my browser) All I receive is Timeout. With cUrl I get ...
0
votes
0answers
19 views

Can One Extract a Hyphenated Filename File From a Server Using Curl?

The images I'm dealing with have file names like part-1.jpg, object-a.jpg, Aspect-0.jpg. What I'm wondering is what I put in place of ???: curl -O http://server/[???].jpg
0
votes
0answers
831 views

Libcurl not usable on mac os x

I'm running the configure command to install a particular software package, but it fails when it reaches it's libcurl requirement. This is the output it gives: checking for the version of ...
1
vote
1answer
205 views

Accessing a remote machine with or without ssh tunneling

I just want to educate myself on an interesting issue here. When I curl a URL I get this error curl "http://myserver:8888/payload" curl: (7) couldn't connect to host However when I setup a tunnel ...
0
votes
0answers
586 views

Apahce proxy throws error connecting to a third party

Connection to a certain thirdparty service via php curl throws this error: CURL error: Received HTTP code 403 from proxy after CONNECT The apahe error log on the proxy throws this: Connect to ...
2
votes
0answers
288 views

pipe unlimited curl stdout into chunked files

I've got an HTTP-endpoint that serves an unlimited stream of events. Now I'd like to record that stream using curl, but splitting the recorded files at the same time - preferably based on the line ...
1
vote
1answer
416 views

calling json-rpc via curl

I'am running Bitcoin-qt 0.8.6.1-beta on Qt 4.8.3 on windows 7 64. below is my .conf content in %appdata%/bitcoin folder rpcuser=bitcoinrpc rpcpassword=ahything rpcallowip=127.0.0.1 ...
0
votes
1answer
852 views

Mac OSX 10.9 resume folder copy to NAS

I have a really big folder (174GB) that I wanted to move to a NAS using Ethernet. I started the transfer and when it was neary finished the computer stopped (I accidentally disabled caffeine) and the ...
1
vote
0answers
71 views

How do I log in to “The Cloud” wifi hotspots using curl?

In England at least, there's this accursèd thing called 'The Cloud', which increasingly provides all public wifi hotspots. Has anyone already knocked up a curl command/bash script/etc to log in to it ...
0
votes
1answer
178 views

File operations on FTPS server with cURL

I need to perform some file transfer operations on some remote FTPS server that uses certificate authentication from a Windows 2008 scripting environment, so that's why I'm stuck with cURL pretty ...
0
votes
1answer
124 views

A Script That is Aware of Remote File Change

curl -z pci.ids --remote-time --silent -f -L -O http://pciids.sourceforge.net/v2.2/pci.ids This Code Checks For The Remote File Change and Download If Remote File Has Been Modified but Is There a ...
0
votes
1answer
807 views

Command line utility to convert html to text

What would be the best command line utility to convert html to text? I want to curl a query to google, convert the output html to text and read converted text on my terminal. I am using rhel6.
0
votes
1answer
1k views

cURL: multiple POST requests while reusing the TCP connection

If multiple URLs are passed to cURL, it reuses the TCP connection when possible. For example: curl -o 1.jpg http://example.com/1.jpg -o 2.jpg http://example.com/2.jpg I need to do the same, but ...
2
votes
1answer
62 views

Shell commands executed from Vim behave differently

I have mapped a key in Vim to run curl-based tests on my API with :map ,t :w\|:!quizzical tests.sh <CR>. The tests run as expected when I press the key; however, the tests which take longer to ...
5
votes
2answers
16k views

how to get and use certificates with curl

1) Entering the following command curl -k email@mail.com:password https://www.site.com Returns the following error <HTML><HEAD><meta http-equiv="content-type" ...
2
votes
1answer
755 views

Omnisharp for Emacs doesn't work

I tried recently to work with C# with Emacs24. I found https://github.com/sp3ctum/omnisharp-emacs I installed it with MELPA. Then I run it. Nothing happened. I wonder what went wrong. I think I don't ...
-1
votes
2answers
274 views

Curl with ipv6 not working by default

I am trying to curl ipv6 adresses, but without success. If i try lets say: curl https://google.com I wait for a timeout and get network unreachable If i force ipv4, everything is fine. ...
2
votes
0answers
452 views

loop curl could not resolve host

I've had working script, which suddenly doesn't works. It contains something like this: while read url; do curl $url done < links links file: http://duckduckgo.com http://google.com And I ...
0
votes
2answers
545 views

SFTP - Recursively fetch new files

I have a remove file system structure like this: + /measure + / 2013-09 + / 2013-09-04 + / fooinstrument + result03343445845.csv + ...
1
vote
4answers
894 views

How does ping figure out the IP address of a website?

I'm trying to understand better the different network-query tools such as nc, curl, whois, dig, nslookup and at the same time a bit more about the architecture of internet requests, servers, and the ...
0
votes
1answer
437 views

Error with curl

I'm having this error with curl: curl https://www.google.com -v * About to connect() to www.google.com port 443 (#0) * Trying 2607:f8b0:400e:c01::67... * Failed to connect to ...
3
votes
1answer
2k views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
3
votes
1answer
391 views

Why is there a very long 6 second delay in start of Curl

I did a full install of Cygwin on my Windows 7 x64 machine. After each machine reboot, Curl can take around 6 seconds to start (even if no command line options were specified). On subsequent uses of ...
6
votes
1answer
554 views

How can I use a command line tool like Wget to log into an OpenID site?

Specifically, I would like to be able to download certain pages from my user profile on the various Stack Exchange sites. I would, however, like to do this automatically (using a cron job), from ...
0
votes
0answers
337 views

Make curl work on ssh connection

On my windows xp machine, I can connect to a remote host via putty and psftp. However I fail to establish a connection with curl. When I enter curl -v --key priv_key.ppk --pubkey pub_key.pub ...
1
vote
1answer
680 views

Have a txt file keep it's line breaks after being send over FTP via Curl

I'm using Curl to send a file over FTP. It is just a generated text file from a screen hardcopy command. When I cat the file, it displays correctly, it looks like this: System up for 14844 1, ...
0
votes
2answers
1k views

tar piped to curl: This does not look like a tar archive

curl -L http://download.transmissionbt.com/files/transmission-2.80.tar.xz -s -o - | tar -xf - The error: tar: This does not look like a tar archive tar: Skipping to next header tar: Exiting with ...
1
vote
1answer
5k views

How can I send a http get request from command line under WinPE

I need to send a http get request under WinPE. My first thought was curl.exe. It works well under Windows but not WinPE. I committed curl.exe to boot.wim but when I tested it under WinPE, it returns ...
1
vote
0answers
187 views

Curl not working on OS X

I recently upgraded XCode and while I'm not sure if that caused the problem, I can't curl out of my computer anymore. This seems to break rvm and homebrew but git still works. I upgraded curl and ...
2
votes
1answer
443 views

What is more transparent, curl or wget in this web crawler project?

I have files on a Windows 7, xampp server (using altered port, 181). I am triggering the script from another computer (linux) using curl http://xxx.xxx.xxx:181/phpproject.php This PHP project is a ...
0
votes
2answers
324 views

Using curl on centos 5.9 for the same server hosted sites giving 404 [closed]

I am using the following command on my ssh terminal- curl -v -L http://mysite/path to request some data. However, its returning a 404 error(the site is hosted on the same server). But using curl ...
0
votes
1answer
263 views

Can't copy website with curl

I want to get all the images from a website behind our company firewall, similar to what was asked here. I can access the site from Firefox on OS-X, but when I try to use curl I get: You ...
2
votes
1answer
541 views

Do curl and wget open gzipped files automatically?

I made two http calls to the same gzipped resource from two different machines - RHEL and OSX. curl http://example.com/file.gz The first call returned an expected binary output: /�IQ���} The ...
0
votes
0answers
37 views

Curl ignores the -r switch when -C is present

I wanted to download a part of a file. (Let's say http://mysite.com/myfile.zip) So I used this command: curl -r 200000-400000 -O http://mysite.com/myfile.zip Unfortunately my Internet ...
4
votes
1answer
3k views

Specifying minor TLS version when using curl

Is there a way to specify curl to use a specific TLS version? Like 1.1 or 1.2? I can see only sslv3 and tlsv1 options in command help. I took latest src and compiled it with openssl 1.0.1e. Still dont ...
1
vote
0answers
141 views

Permissions tar extraction to nfs share from a Perl script

As part of an installation procedure of a piece of software, I use: curl -s <url-to-targz> | tar -p -x -z -C / inside a Perl script (I use qx($command) or system($command), either will ...