curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

-1
votes
0answers
18 views

monitoring websites using curl

I am looking to monitor about 40 (Linux/Apache) urls hosted in a DMZ. I am thinking of using curl from a Linux box in the DMZ to do this and pass the data to a monitoring tool. My plan is to use a ...
0
votes
0answers
9 views

Bashscript using curl to crawl a website that needs a login

I am looking to crawl a website that has kind of a newsfeed with elements hat have a timestamp. There's only a certain amount of elements in the feed and a "Previous" button at the end of the feed. It ...
0
votes
3answers
27 views

Sending a POST request via curl has issues with double quotes

So, in-order to send a POST request to gists (github), you can do something like this as seen in https://gist.github.com/caspyi... curl --user "user" -X POST --data '{"description":"Created via ...
0
votes
1answer
37 views

curl command not found on OSX El Cap

My understanding is that curl should be installed. I had tried using curl from the Unix command line and got the message it could not be found. I'm wondering If something I did in efforts to clean up ...
1
vote
2answers
95 views

Curl response hangs

I have a site set up in a virtual box. I am able to access it just fine from a browser on my machine, but I have trouble accessing it via CURL while SSH'ed in to the box. When I try, curl hangs ...
0
votes
0answers
7 views

mirror site behind login

I would simply like to mirror all the pages (millions) on a site behind a login. This is for personal use. I have a login for the site in question but I want to scrape it into a database. I ...
-1
votes
1answer
29 views

How to use CURL tool?

I just got introduced to CURL today and I went to their website to download the tool and all I got is .cgi file called "metalink" and I don't know how to use it or how to run it, can someone help ...
0
votes
1answer
22 views

How to use VERB in url

I am trying to create and API which uses VERB PUT,GET,DELETE I can use it in curl by this curl -X DELETE http://localhost:8080/recipes/hi But how can i use similarly in url for eg This works ...
0
votes
1answer
28 views

Dictionary attack in Bash with cURL

I'm fairly new to Bash, but this is a script I've been working on for the past few days. It's meant to run a dictionary attack on an HTTP-POST login for a website, by reading the time module and ...
1
vote
0answers
14 views

“curl” ignores proxy environment variable when using an ip address?

I am using curl to access some web services through a proxy (an instance of polipo running on port 8123, chained to an ssh SOCKS5 proxy). If I provide proxy information using the http_proxy ...
0
votes
1answer
10 views

Neither wget nor curl downloads this wikipedia api request

I can perfectly download this link and save it as a text file in a web browser environment. But I can't download and save that wikipedia api request using neither wget nor curl. I use wget -O file.txt ...
0
votes
2answers
56 views

How to POST file contents using cURL?

I want to do an HTTP POST of the contents (as a string) of a local file located at path/to/my-file.txt to a URL endpoint at http://example.com/. For example, I might want to do the following: ...
0
votes
1answer
18 views

libcurl version 7.21 to work on Redhat 6

I am having an issue with libcurl version 7.19 and socks proxy on redhat 6. I believe it was fixed in libcurl version 7.21 how do i upgrade libcurl on redhat 6 to 7.21 or greater?
4
votes
1answer
188 views

Automatic login to Vodafone Community WiFi (Fon) via Unix script

First I have to say that my question is similar to this one I am using a Vodafone Community Fon AP and it often disconnects. In OS X, which I'm using, there is a tool called Captive Network Assistant ...
2
votes
0answers
138 views

Retrieving from ftp dirs with wildcards

What is the most efficient way to retrieve all relevant data from ftp if the address of said data can be specified with wildcards as below: ...
1
vote
0answers
193 views

SSL error when connecting to git from Mac OS X 10.11.3 (El Capitan)

When I try to clone / pull / fetch from a git repository via https from a Mac System I keep getting an error: error: RPC failed; result=56, HTTP code = 200 fatal: The remote end hung up unexpectedly ...
3
votes
2answers
98 views

curl check if file is newer and instead of downloading - execute a bash (or python) script

I am having a bit of an issue. I have a file, hosted on a remote server (http://mywebsite/file.zip). I also have a few embedded linux boxes (running openelec OS). The boxes have fairly limited ...
1
vote
1answer
33 views

Fetch source file from GitHub using command line tools?

This question is related to a question offsite at FATAL: MemorySanitizer can not mmap the shadow memory. The person who responded said: Do simple tests (e.g. this one) work for you? The link ...
1
vote
0answers
47 views

curl suddenly unable to get local issuer certificate

I've been using Docker 1.7.1 with base Ubuntu 14.04 image and all of a sudden this curl: curl -L -o sbt.tgz https://dl.bintray.com/sbt/native-packages/sbt/0.13.9/sbt-0.13.9.tgz is failing me with ...
-1
votes
2answers
24 views

Where is my file saved locally?

I've downloaded a file--I see the contents of my text file in the terminal window as it is downloaded in curl, but where is it saved on my local machine: curl -v -O myFile.txt -u username:password ...
0
votes
0answers
28 views

Finding a text string on a website and all its links

Can you help me understand process of finding a certain text string on a website and all its immediate links? I started delving into wget and curl with help of grep but was unable to come up with ...
0
votes
2answers
64 views

cURL connecting to localhost (127.0.0.1) instead of destination IP

On Mac OS X I have an issue where calling: curl -v www.google.com results in: * Rebuilt URL to: www.google.com/ * Trying 62.253.72.153... * Connected to www.google.com (127.0.0.1) port 80 (#0) ...
0
votes
1answer
42 views

How to echo only 1st line from 'curl' command output?

I'm trying to get only first line of the curl command output. (Sorry if this is confusing) Let's say, for an instance, i run simply: # curl http://localhost <!-- This is the hidden line i want to ...
0
votes
1answer
22 views

How to Check value for an input element after fetching contents of page using cUrl

I am trying to download a page using cUrl. I have saved the page to a file and I then grep on it to extract the value of the text field. The text field appears multiple times and I just need one ...
2
votes
1answer
20 views

Using variable in a CURL command

Am trying insert a document into my cloudant as below. @echo OFF SPEECH_TEXT="Call me Ishmael. Some years ago-never mind how long precisely-" curl -X POST -k 'https://<user ...
0
votes
0answers
16 views

Command to fetch a list of urls with connection keep-alive but fail on the first error (404/500)

I am trying to reproduce a potential bug in the Jetty web server with regard to serving static files. I need to repeatedly download a list of urls with "Connection: keep-alive" until one of them ...
0
votes
1answer
68 views

How to open cURL in Firefox or other web browser?

I have Firebug installed in Firefox. In the Net panel there are all network requests listed. When I right-click and choose Copy as cURL, how do I use this cURL? How can I repeat my POST request data ...
0
votes
0answers
27 views

OSX terminal : log on on a website

I would like to login/ out on a Web site using the terminal, I currently using a appleScript I wrote but was wondering can you use terminal command line instead (not running my script from terminal) ...
1
vote
1answer
37 views

Is there a difference between curl --tlsv1 and --tlsv1.0?

The Curl man page lists -1 --tlsv1 and separately lists --tlsv1.0 along with v1.1 and v1.2. The descriptions are: -1, --tlsv1 (SSL) Forces curl to use TLS version 1 when negotiating ...
0
votes
1answer
118 views

TLS curl POST resulting in SSLv3 error

I am trying to make a POST request via curl to a server that only supports TLS 1.2, TLS 1.1, and TLS 1.0. However, even when I try to specify the need for TLS instead of the default SSL v. 3 (adding ...
0
votes
0answers
9 views

Indexing file resources from a website

How can I get a list of all (image) resources from a website? Is there a better wait than using wget and parsing the log stream in order to get the list of all images? Worst is, I'm pretty use I ...
0
votes
1answer
44 views

Telling cUrl not to normalize a url

When using curl to call some url, it simplifies the path in the url (for example from /a/../b to /b). How can I tell cUrl not to do so? $ curl -V curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 ...
0
votes
0answers
70 views

Using curl with a https proxy

I have setup a squid proxy so that it listens on a HTTPS port instead of the usual HTTP port, in order to have encrypted communication between the client and the proxy itself. While I have been able ...
0
votes
0answers
379 views

Notice: Use of undefined constant CURLOPT_CAINFO

I try to use TwitterOAuth which requires cURL. First of all, I think it is important to point that I use Win 8 64 bits with WampServer 3.0 and PHP 5.6.15 When I run the scripts (given by TwitterOAuth) ...
0
votes
0answers
52 views

Issues installing development tools to Ubuntu: curl: (35) Unknown SSL protocol error

I am attempting to download both Node.js with Linuxbrew, and RVM to an Ubuntu VM. Linuxbrew installs fine, but when I try to install Node.js with it, or RVM, I get this error: curl: (35) Unknown SSL ...
0
votes
0answers
29 views

curl -X POST not working properly?

Here's what I'm running as a test and it is working: curl -X POST http://zone-h.org/notify When I run that, I get the HTML of the page returned. However, when I run this, it isn't returning the ...
0
votes
1answer
59 views

Curl POST syntax work under Windows?

I am trying to issue the following command under Windows 10: D:\>curl -i -X POST -H "Content-Type:application/json" -d '{ "firstName" : "Frodo", "lastName" : "Baggins" }' ...
0
votes
1answer
385 views

PHP CURL SSL CA cert (path? access rights?)

Curl through php is constantly complaining about Problem with the SSL CA cert (path? access rights?). This all happend after trying to fix another problem with curl ssl with pem cert chain not ...
1
vote
1answer
1k views

curl 'SSL connect error'

basic steps taken: # echo -n | openssl s_client -showcerts -connect example.com:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/example.cert # curl -v --cacert /tmp/example.cert ...
0
votes
0answers
64 views

Slow download over libraries on certain Windows systems

I am using custom libraries for file download (cURL and wget) on Windows client over HTTP from my server. On most of systems the download speed is as it should be - maximum determined by network ...
1
vote
0answers
13 views

Curl in bash code - produces wrong status

This code seems to ouput a http status of 000 - which seems to indicate something didn't connect properly but when I do this curl outside of the bash script it works fine and produces a 200 so ...
0
votes
0answers
59 views

curl to test secure web proxy

I have used Chrome to successfully connect to the secure web proxy(squid https_port) Chrome doc , but I want to know any command line tool(such as curl) will connect to it too. Curl's -x option ...
0
votes
0answers
63 views

Gmail Automation via Bash (To refresh POP Fetch from other accounts)

Gmail has this option to fetch mails from other accounts. I used this feature to create a central mail box. Works good. Gmail periodically polls other mail box over POP. But the frequency is 1 hour + ...
-2
votes
1answer
86 views

Filter curl header output in verbose mode

Using curl -v ..., i.e. the verbose mode that prints out the input and output headers. However, this info is not piped and I can not grep-out lines which I dont need. How can you filter curl header ...
0
votes
0answers
70 views

WGET not working but CURL works on the same address/host?

WGET give me this error: HTTP request sent, awaiting response... 403 Forbidden 2015-11-23 12:07:06 ERROR 403: Forbidden. If I try the same URL (it’s an internal network Apache server, firewall ...
0
votes
2answers
307 views

Using cURL to post data

I try to post to localhost with some parameters in the URL. Everything just works fine in Fiddler. But cURL doesn't allow me to do it, with the error message below. curl --data "Task=bake" ...
0
votes
0answers
24 views

windows curl --compressed : unrecognized command

I found the SAME problem: (si I know it can be duplicated) Curl Compressed Command not Recognized in Windows However no one of these solutions works for me (win7 / 64bits) When I use without the ...
0
votes
1answer
62 views

cURL in shell vs. knockd

I want to use curl to control features of a IPcam. This works fine in a shell (/bin/sh) but fails when called from knockd. I am trying this on a QNAP NAS (Busybox Linux). The script I am using is: ...
0
votes
0answers
63 views

How to interact with submit input type with no name in php using curl

<select name="_limit"> <option value='50' >50</option> <option value='200' >200</option> <option value='300' >300</option> <option value='500' ...
0
votes
0answers
65 views

Curl Could not resolve host

I'm trying to get a webpage after submitting a username and password but i'm receiving an error. This is on Ubuntu server The command i'm using is curl -g -d ...