curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

0
votes
1answer
13 views

Neither wget nor curl downloads this wikipedia api request

I can perfectly download this link and save it as a text file in a web browser environment. But I can't download and save that wikipedia api request using neither wget nor curl. I use wget -O file.txt ...
0
votes
2answers
195 views

How to POST file contents using cURL?

I want to do an HTTP POST of the contents (as a string) of a local file located at path/to/my-file.txt to a URL endpoint at http://example.com/. For example, I might want to do the following: ...
0
votes
0answers
198 views

Make a curl POST request from scraped data

I scraped a website using the simple DOM php parser : <?php include 'domparse.php'; $html = file_get_html('http://oceanofgames.com/rebel-galaxy-free-download/'); foreach($html->find('form[...
0
votes
1answer
21 views

libcurl version 7.21 to work on Redhat 6

I am having an issue with libcurl version 7.19 and socks proxy on redhat 6. I believe it was fixed in libcurl version 7.21 how do i upgrade libcurl on redhat 6 to 7.21 or greater?
0
votes
2answers
121 views

cURL connecting to localhost (127.0.0.1) instead of destination IP

On Mac OS X I have an issue where calling: curl -v www.google.com results in: * Rebuilt URL to: www.google.com/ * Trying 62.253.72.153... * Connected to www.google.com (127.0.0.1) port 80 (#0) &...
0
votes
3answers
506 views

Is it possible to get the download file name of an mp3 from server using bash?

I am trying to get the final download file name of an mp3 from a server, specifically audiotool.com e.g. http://api.audiotool.com/track/haunt_opaque/mixdown.mp3 When you navigate to this url in a ...
2
votes
0answers
140 views

Retrieving from ftp dirs with wildcards

What is the most efficient way to retrieve all relevant data from ftp if the address of said data can be specified with wildcards as below: ftp://ftp.some.site.gov/data/directory/only/*/these/*/*...
2
votes
0answers
446 views

SSL error when connecting to git from Mac OS X 10.11.3 (El Capitan)

When I try to clone / pull / fetch from a git repository via https from a Mac System I keep getting an error: error: RPC failed; result=56, HTTP code = 200 fatal: The remote end hung up unexpectedly ...
8
votes
2answers
675 views

How can I use a command line tool like Wget to log into an OpenID site?

Specifically, I would like to be able to download certain pages from my user profile on the various Stack Exchange sites. I would, however, like to do this automatically (using a cron job), from ...
3
votes
2answers
171 views

curl check if file is newer and instead of downloading - execute a bash (or python) script

I am having a bit of an issue. I have a file, hosted on a remote server (http://mywebsite/file.zip). I also have a few embedded linux boxes (running openelec OS). The boxes have fairly limited ...
63
votes
7answers
159k views

Run cURL commands from Windows console

Is there a way to install cURL in windows in order to run cURL commands from the command prompt?
1
vote
1answer
57 views

Fetch source file from GitHub using command line tools?

This question is related to a question offsite at FATAL: MemorySanitizer can not mmap the shadow memory. The person who responded said: Do simple tests (e.g. this one) work for you? The link ...
-1
votes
2answers
25 views

Where is my file saved locally?

I've downloaded a file--I see the contents of my text file in the terminal window as it is downloaded in curl, but where is it saved on my local machine: curl -v -O myFile.txt -u username:password "...
0
votes
0answers
33 views

Finding a text string on a website and all its links

Can you help me understand process of finding a certain text string on a website and all its immediate links? I started delving into wget and curl with help of grep but was unable to come up with ...
2
votes
2answers
2k views

How to convert HTML to text?

How it is possible to convert HTML to text file in Linux? For example I want to curl a query to Google, then convert the output html to text and read converted text on my terminal. I am using RHEL6.
1
vote
1answer
58 views

Is there a difference between curl --tlsv1 and --tlsv1.0?

The Curl man page lists -1 --tlsv1 and separately lists --tlsv1.0 along with v1.1 and v1.2. The descriptions are: -1, --tlsv1 (SSL) Forces curl to use TLS version 1 when negotiating ...
0
votes
1answer
76 views

How to echo only 1st line from 'curl' command output?

I'm trying to get only first line of the curl command output. (Sorry if this is confusing) Let's say, for an instance, i run simply: # curl http://localhost <!-- This is the hidden line i want to ...
0
votes
1answer
22 views

How to Check value for an input element after fetching contents of page using cUrl

I am trying to download a page using cUrl. I have saved the page to a file and I then grep on it to extract the value of the text field. The text field appears multiple times and I just need one value....
42
votes
4answers
42k views

How to wget a file with correct name when redirected?

So after some time of searching on Google and Super User (and scanning man pages) I was unable to find an answer to something that (I think) should be simple: If you go here: http://www.vim.org/...
2
votes
1answer
21 views

Using variable in a CURL command

Am trying insert a document into my cloudant as below. @echo OFF SPEECH_TEXT="Call me Ishmael. Some years ago-never mind how long precisely-" curl -X POST -k 'https://<user id>.cloudant.com/...
0
votes
0answers
24 views

Command to fetch a list of urls with connection keep-alive but fail on the first error (404/500)

I am trying to reproduce a potential bug in the Jetty web server with regard to serving static files. I need to repeatedly download a list of urls with "Connection: keep-alive" until one of them ...
9
votes
2answers
26k views

How to get and use certificates with curl

1) Entering the following command curl -k email@mail.com:password https://www.example.com Returns the following error <HTML><HEAD><meta http-equiv="content-type" content="text/...
0
votes
1answer
143 views

How to open cURL in Firefox or other web browser?

I have Firebug installed in Firefox. In the Net panel there are all network requests listed. When I right-click and choose Copy as cURL, how do I use this cURL? How can I repeat my POST request data ...
0
votes
0answers
28 views

OSX terminal : log on on a website

I would like to login/ out on a Web site using the terminal, I currently using a appleScript I wrote but was wondering can you use terminal command line instead (not running my script from terminal) ...
1
vote
3answers
10k views

Upload folder with cURL and FTP using Batch file on Windows?

I am trying to upload a whole folder, including it's sub-folders and files to a folder on my server from a Windows batch (.bat) file. Below is the command I have for the Upload part... curl -T E:\...
0
votes
1answer
138 views

TLS curl POST resulting in SSLv3 error

I am trying to make a POST request via curl to a server that only supports TLS 1.2, TLS 1.1, and TLS 1.0. However, even when I try to specify the need for TLS instead of the default SSL v. 3 (adding --...
20
votes
2answers
10k views

Can cURL send requests to sockets?

I can find nothing about this on Google or the cURL man pages. I have a HTTP server running at /var/run/my-server.sock, and I want to test it by sending a simple request using cURL. Can this be done ...
14
votes
3answers
18k views

how to use curl to verify if a site's certificate has been revoked?

To check if the certificate for google.com has been revoked, I tried the following command: curl https://www.google.com --cacert GeoTrust_Global_CA.pem --crlfile gtglobal.pem -v , but I got the ...
0
votes
0answers
13 views

Indexing file resources from a website

How can I get a list of all (image) resources from a website? Is there a better wait than using wget and parsing the log stream in order to get the list of all images? Worst is, I'm pretty use I ...
3
votes
2answers
2k views

curl: (7) Failed connect to www.google.com:80;

I have installed Git for Windows and got cURL automatically. C:\>curl --version curl 7.30.0 (i386-pc-win32) libcurl/7.30.0 OpenSSL/0.9.8y zlib/1.2.7 Protocols: dict file ftp ftps gopher http https ...
0
votes
0answers
98 views

Using curl with a https proxy

I have setup a squid proxy so that it listens on a HTTPS port instead of the usual HTTP port, in order to have encrypted communication between the client and the proxy itself. While I have been able ...
0
votes
0answers
550 views

Notice: Use of undefined constant CURLOPT_CAINFO

I try to use TwitterOAuth which requires cURL. First of all, I think it is important to point that I use Win 8 64 bits with WampServer 3.0 and PHP 5.6.15 When I run the scripts (given by TwitterOAuth) ...
0
votes
0answers
73 views

Issues installing development tools to Ubuntu: curl: (35) Unknown SSL protocol error

I am attempting to download both Node.js with Linuxbrew, and RVM to an Ubuntu VM. Linuxbrew installs fine, but when I try to install Node.js with it, or RVM, I get this error: curl: (35) Unknown SSL ...
0
votes
0answers
30 views

curl -X POST not working properly?

Here's what I'm running as a test and it is working: curl -X POST http://zone-h.org/notify When I run that, I get the HTML of the page returned. However, when I run this, it isn't returning the ...
0
votes
1answer
122 views

Curl POST syntax work under Windows?

I am trying to issue the following command under Windows 10: D:\>curl -i -X POST -H "Content-Type:application/json" -d '{ "firstName" : "Frodo", "lastName" : "Baggins" }' http://localhost:8080/...
2
votes
2answers
4k views

How do I unzip a compressed file as it's downloading?

I'm downloading a large file that's not an archive, and I want to combine the tasks of downloading and decompressing. How can I do them simultaneously?
1
vote
1answer
2k views

curl 'SSL connect error'

basic steps taken: # echo -n | openssl s_client -showcerts -connect example.com:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/example.cert # curl -v --cacert /tmp/example.cert ...
0
votes
1answer
644 views

PHP CURL SSL CA cert (path? access rights?)

Curl through php is constantly complaining about Problem with the SSL CA cert (path? access rights?). This all happend after trying to fix another problem with curl ssl with pem cert chain not ...
0
votes
0answers
68 views

Slow download over libraries on certain Windows systems

I am using custom libraries for file download (cURL and wget) on Windows client over HTTP from my server. On most of systems the download speed is as it should be - maximum determined by network ...
1528
votes
6answers
1.5m views

What is the cURL command-line syntax to do a POST request?

How can I make a POST request with the cURL command-line tool?
3
votes
0answers
412 views

pipe unlimited curl stdout into chunked files

I've got an HTTP-endpoint that serves an unlimited stream of events. Now I'd like to record that stream using curl, but splitting the recorded files at the same time - preferably based on the line ...
1
vote
0answers
17 views

Curl in bash code - produces wrong status

This code seems to ouput a http status of 000 - which seems to indicate something didn't connect properly but when I do this curl outside of the bash script it works fine and produces a 200 so ...
0
votes
0answers
77 views

curl to test secure web proxy

I have used Chrome to successfully connect to the secure web proxy(squid https_port) Chrome doc , but I want to know any command line tool(such as curl) will connect to it too. Curl's -x option ...
2
votes
2answers
247 views

How can I instruct curl to reuse credentials after it followed a redirect?

I know that curl can follow redirects (the -L argument), and I know that I can provide credentials to a request (-u username:password), but I want to use both all the way down the redirect chain. How ...
0
votes
0answers
89 views

Gmail Automation via Bash (To refresh POP Fetch from other accounts)

Gmail has this option to fetch mails from other accounts. I used this feature to create a central mail box. Works good. Gmail periodically polls other mail box over POP. But the frequency is 1 hour + ...
0
votes
1answer
72 views

cURL in shell vs. knockd

I want to use curl to control features of a IPcam. This works fine in a shell (/bin/sh) but fails when called from knockd. I am trying this on a QNAP NAS (Busybox Linux). The script I am using is: #!...
-2
votes
1answer
157 views

Filter curl header output in verbose mode

Using curl -v ..., i.e. the verbose mode that prints out the input and output headers. However, this info is not piped and I can not grep-out lines which I dont need. How can you filter curl header ...
0
votes
2answers
539 views

Using cURL to post data

I try to post to localhost with some parameters in the URL. Everything just works fine in Fiddler. But cURL doesn't allow me to do it, with the error message below. curl --data "Task=bake" http://...
0
votes
0answers
93 views

WGET not working but CURL works on the same address/host?

WGET give me this error: HTTP request sent, awaiting response... 403 Forbidden 2015-11-23 12:07:06 ERROR 403: Forbidden. If I try the same URL (it’s an internal network Apache server, firewall isn’...
0
votes
0answers
340 views

Download file vis cURL, but not with Wget

I have a link I'm trying to download. Before, we could download with either Wget or cURL because the website (which runs under Tomcat) didn't require a user name or password in order to download. ...