curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

0
votes
0answers
97 views

Gmail Automation via Bash (To refresh POP Fetch from other accounts)

Gmail has this option to fetch mails from other accounts. I used this feature to create a central mail box. Works good. Gmail periodically polls other mail box over POP. But the frequency is 1 hour + ...
0
votes
1answer
82 views

cURL in shell vs. knockd

I want to use curl to control features of a IPcam. This works fine in a shell (/bin/sh) but fails when called from knockd. I am trying this on a QNAP NAS (Busybox Linux). The script I am using is: #!...
-2
votes
1answer
204 views

Filter curl header output in verbose mode

Using curl -v ..., i.e. the verbose mode that prints out the input and output headers. However, this info is not piped and I can not grep-out lines which I dont need. How can you filter curl header ...
0
votes
2answers
668 views

Using cURL to post data

I try to post to localhost with some parameters in the URL. Everything just works fine in Fiddler. But cURL doesn't allow me to do it, with the error message below. curl --data "Task=bake" http://...
0
votes
0answers
108 views

WGET not working but CURL works on the same address/host?

WGET give me this error: HTTP request sent, awaiting response... 403 Forbidden 2015-11-23 12:07:06 ERROR 403: Forbidden. If I try the same URL (it’s an internal network Apache server, firewall isn’...
0
votes
0answers
351 views

Download file vis cURL, but not with Wget

I have a link I'm trying to download. Before, we could download with either Wget or cURL because the website (which runs under Tomcat) didn't require a user name or password in order to download. ...
0
votes
0answers
42 views

windows curl --compressed : unrecognized command

I found the SAME problem: (si I know it can be duplicated) Curl Compressed Command not Recognized in Windows However no one of these solutions works for me (win7 / 64bits) When I use without the --...
0
votes
0answers
38 views

How can I approximate impact of using proxies on cURL request time using CURLINFO_*_TIME?

Below is a small dataset from which I'm trying to answer two questions: How much time did the proxy take to connect to the API server? How much time did the API request take to return? The basic ...
0
votes
0answers
82 views

How to interact with submit input type with no name in php using curl

<select name="_limit"> <option value='50' >50</option> <option value='200' >200</option> <option value='300' >300</option> <option value='500' >500&...
0
votes
0answers
97 views

Curl Could not resolve host

I'm trying to get a webpage after submitting a username and password but i'm receiving an error. This is on Ubuntu server The command i'm using is curl -g -d “edit[name]=username&edit[pass]=...
46
votes
2answers
21k views

Can I make cURL fail with an exitCode different than 0 if the HTTP status code is not 200?

I was always assuming that when curl got an HTTP 500 response it was returning an exit code that meant failure (!= 0), but that seems to be not the case. Is there a way I can I make cURL fail with an ...
1
vote
1answer
1k views

Apache proxy throws error connecting to a third party

Connection to a certain thirdparty service via php curl throws this error: CURL error: Received HTTP code 403 from proxy after CONNECT The apahe error log on the proxy throws this: Connect to ...
0
votes
1answer
219 views

CURL in bash script works with . (dot space), not ./ (dot slash)

I have a bash script which sends a couple of notifications to Slack via CURL and spawns an expect script. When I run the script via . scriptName.sh it works fine, but when I run it by typing ./...
6
votes
1answer
6k views

How can I a download private repository from GitHub via when I have no access to ‘git’ on my local machine?

What I want to do is to download private repository archive from GitHub, extract it, remove archive file and copy some directories that are inside downloaded project. I tried to use wget but I cannot ...
0
votes
1answer
265 views

Posting XML through cURL using --data-binary

I have this really typical problem. I have an XML file that I have to post to a server. I was told by the network engineer of that site to use the cURL function. The function that he provided to me ...
0
votes
0answers
58 views

Redirection not working for the first time

There are output of curl command. Sometimes my site doesn’t redirect properly, showing error page: no data received from the server. Is there any way to remove this error? C:\Users\sumit>...
1
vote
2answers
104 views

I'm trying to open query.yahooapis.com but curl is trying to use IPV6 even if I only have IPV4 enabled

When I try to call query.yahooapis.com with curl it's trying to use the ipv6 address but IPV6 is disabled on my computer $ curl -v 'http://query.yahooapis.com/' * Hostname was NOT found in DNS cache *...
0
votes
1answer
298 views

Installing and using cURL in Linux and use it in Eclipse as well?

I come from a Windows environment - I used cURL in Eclipse (C). I want to do the same in a Linux environment (Ubuntu). I followed the installation guide on how to install cURL, but how do you use cURL ...
0
votes
0answers
112 views

How to receive sse (server sent events) and execute bash commands then

How do i receive server sent events (sse) from a webserver and execute a bash command with the received data in debian (raspbian)? Is that possible with curl as i can already receive sse. Thanks ...
0
votes
0answers
89 views

Amazon S3 upload error: An exception occurred while uploading parts to a multipart upload

I am trying to upload a 30 GB file to Amazon S3 using the AWS PHP SDK. require('../vendor/autoload.php'); $client = new Aws\S3\S3Client([ 'version' => 'latest', 'region' => 'us-east-1' ]); $...
22
votes
3answers
15k views

Is “curl -u username:password http://example.com” secure?

Is curl -u username:password http://example.com secure? If not, can you give a brief explanation of how someone could obtain your password?
0
votes
0answers
251 views

Sending POST with esp8266 Module

I am trying to send a POST request from my esp8266 wifi module and I am getting errors. Such as: "Your browser sent a request that this server could not understand" I am trying to send: POST http://...
0
votes
1answer
130 views

Delivering a payload to a URL, using cURL

I need help figuring out the cURL request required to send the following data to a URL. {"requestid":"555555", "partnermatchid":"10000-000-0000-0000-000", "usercontext":{"ipaddressmasked":"XXX.XXX....
2
votes
2answers
289 views

Where does cURL store data while it's still downloading?

I'm using cURL on the command line to automate the downloading of many files from a server. I'm wondering where cURL stores its data whilst it's downloading. The reason I need this info is so I can ...
7
votes
2answers
8k views

Specifying minor TLS version when using curl

Is there a way to specify curl to use a specific TLS version? Like 1.1 or 1.2? I can see only sslv3 and tlsv1 options in command help. I took latest src and compiled it with openssl 1.0.1e. Still dont ...
0
votes
0answers
210 views

wget or cURL for dynamic recursive URLs (generated through directories and adding specific same text at the end)

I'm using a wget command with some options in order to refresh the cache module from all URLs of a Prestashop. This is the command: wget -r -D www.domain.com -l3 -P temp -x -R "*cart*","*order*","*My ...
1
vote
1answer
74 views

How can I run the equivalent “curl” command that performs similarly to this “wget” command?

I have a wget command: wget -r --no-parent -nH -nd -q --show-progress --progress=bar:force ftp://ftp.ncdc.noaa.gov/pub/data/swdi/database-csv/v2/ But it only works in newer versions of wget. ...
1
vote
2answers
84 views

Download a list of files and use domain name as filename

I got a bunch of files to download which works great with wget -i list_of_urls but my problem is, that wget uses the filename of the downloaded file. Is there a way (or a different tool) which is ...
5
votes
2answers
290 views

Firebug and cURL

I have been using Firebug and cURL for quite a while. Firebug is amazing to capture HTTPS POST request as opposed to a network analyser as it is aware of SSL/TLS negotiation. And cURL is amazing to ...
0
votes
1answer
70 views

How to extract ip address from no-ip account webpage in the shell?

I've set dynamic dns for my home network(which is behind NAT) using www.noip.com. I use ddclient to send updates of my ip, but sometimes it takes a while to update the ip. Anyways it's not the point. ...
4
votes
2answers
261 views

Get curl to output non-2xx to stderr, but 2xxs to a file

I would like some curl command, possibly with some bash hackery, that: Outputs any 2xx responses to a file Outputs any non-2xx responses to stderr Exits with a nonzero status code when a non-2xx ...
1
vote
1answer
80 views

How to run a remote bat file on a local computer

I need to run a batch file, that is stored on a server, locally on a windows 8.1 machine. I can run a shell script in OS X with a curl: cURL "file path" | sh is it possible to cURL in powershell in ...
0
votes
1answer
641 views

how to make curl re-use existing connection

curl --verbose google.com If I execute this command, after curl does its job, it says "Connection #0 to host google.com left intact". But it doesn't use same connection if I execute same command ...
1
vote
0answers
68 views
0
votes
1answer
162 views

What is the curl equivalent of wget -qO (wget -q -O)?

I need to run the following command in Red Hat Linux: wget -q -O http://somepage/somefile.cgi| bash But I don't have wget installed. I could install it, but I want to know how to do this with curl.
2
votes
1answer
186 views

Bulk SFTP download of files specified by URL preserving directories

I have a list of several hundred files I need to pull via SFTP. They are located in various directories and I need them to be in the same directory setup on my local machine. It is important the files ...
4
votes
1answer
1k views

What units does curl use for bandwidth?

curl on the command-line displays progress like this: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent ...
0
votes
1answer
528 views

Escape semicolon curl with -u option

I can't wrap my head around how to escape semicolon in username while making a GET request. curl -u testclient:sdtest1:pass -XGET -H "Content-type:application/json" \ http://localhost:8081/hello ...
0
votes
1answer
241 views

How to upload all the files from local to host using Curl?

i have written a shell script to upload all files with different names in the specific path from local to host, but i have facing some issue and the files are not getting uploaded #!/bin/sh for file ...
0
votes
2answers
436 views

Curl with ipv6 not working by default

I am trying to curl ipv6 adresses, but without success. If i try lets say: curl https://google.com I wait for a timeout and get network unreachable If i force ipv4, everything is fine. ...
0
votes
0answers
655 views

Curl: How to send custom headers while using proxy

I want to send custom headers to the host while using proxies. Take a look at the examples below. Explaining the situation with examples was easier. This works fine as expected: curl https://www....
2
votes
1answer
2k views

Mac OS X Yosemite and Client SSL Certificates

Ever since I upgraded from Mavericks to Yosemite, my SSL Client Certificates have stopped working in certain places. They still work in web browsers and in wget, but they have stopped working in curl ...
2
votes
2answers
142 views

Fast way to download many small webpages?

I need to download lots of small pages as fast as possible. Average size for one page is 3kb. There are millions of pages. First, I tried curl but it was way too slow, probably because it was waiting ...
1
vote
1answer
1k views

What is the correct way to submit content-type header in curl?

I’m using Mac 10.9.5 with bash shell. I’m trying to submit a request via curl, but I keep getting errors when I try and set the content-type header. Below I try davea$ curl -v -o -H "Content-Type: ...
0
votes
1answer
168 views

How to download the latest version of youtube-dl programmatically?

The latest version of youtube-dl is at: https://yt-dl.org/downloads/latest/ which redirects to (as of today): https://yt-dl.org/downloads/2015.07.04/ I'd like to set up a cron job to download the ...
0
votes
2answers
214 views

Curl post form and get result

I found this website http://www.fonefinder.net/findome.php , that locates phone provider by number (ex: http://www.fonefinder.net/findome.php?npa=562&nxx=866&thoublock=1666) shows VERIZON ...
8
votes
4answers
33k views

Downloading file from FTP using cURL

I'm trying to use a cURL command to download a file from an FTP server to a local drive on my computer. I've tried curl "ftp://myftpsite" --user name:password -Q "CWD /users/myfolder/" -O "myfile....
0
votes
2answers
757 views

use curl to download and yum to install (NTLM proxy issues)

I need to use a wi-fi under a corporate environment with my Fedora 21. It uses NTLM. With curl i've been able to reach sites (yahoo) setting the proxy, the type of proxy, the user and the password, ...
0
votes
1answer
503 views

Using a curl scheduled task in windows

I am using the following command in a scheduled task in windows to update my IP to the free Dynamic DNS service called EntryDNS: curl.exe -k -X PUT -d "" "Their url API string here" The problem I am ...
1
vote
0answers
62 views

Run bash command from link in browser

Is there a way to run an arbitrary command on OSX by clicking a link in the browser? As you know, there are certain types of hyperlinks that launch certain applications with a specified action, for ...