curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

0
votes
0answers
36 views

How can I approximate impact of using proxies on cURL request time using CURLINFO_*_TIME?

Below is a small dataset from which I'm trying to answer two questions: How much time did the proxy take to connect to the API server? How much time did the API request take to return? The basic ...
0
votes
0answers
74 views

How to interact with submit input type with no name in php using curl

<select name="_limit"> <option value='50' >50</option> <option value='200' >200</option> <option value='300' >300</option> <option value='500' >500&...
0
votes
0answers
90 views

Curl Could not resolve host

I'm trying to get a webpage after submitting a username and password but i'm receiving an error. This is on Ubuntu server The command i'm using is curl -g -d “edit[name]=username&edit[pass]=...
43
votes
2answers
20k views

Can I make cURL fail with an exitCode different than 0 if the HTTP status code is not 200?

I was always assuming that when curl got an HTTP 500 response it was returning an exit code that meant failure (!= 0), but that seems to be not the case. Is there a way I can I make cURL fail with an ...
1
vote
1answer
1k views

Apache proxy throws error connecting to a third party

Connection to a certain thirdparty service via php curl throws this error: CURL error: Received HTTP code 403 from proxy after CONNECT The apahe error log on the proxy throws this: Connect to ...
0
votes
1answer
211 views

CURL in bash script works with . (dot space), not ./ (dot slash)

I have a bash script which sends a couple of notifications to Slack via CURL and spawns an expect script. When I run the script via . scriptName.sh it works fine, but when I run it by typing ./...
6
votes
1answer
6k views

How can I a download private repository from GitHub via when I have no access to ‘git’ on my local machine?

What I want to do is to download private repository archive from GitHub, extract it, remove archive file and copy some directories that are inside downloaded project. I tried to use wget but I cannot ...
0
votes
1answer
249 views

Posting XML through cURL using --data-binary

I have this really typical problem. I have an XML file that I have to post to a server. I was told by the network engineer of that site to use the cURL function. The function that he provided to me ...
0
votes
0answers
55 views

Redirection not working for the first time

There are output of curl command. Sometimes my site doesn’t redirect properly, showing error page: no data received from the server. Is there any way to remove this error? C:\Users\sumit>...
1
vote
2answers
97 views

I'm trying to open query.yahooapis.com but curl is trying to use IPV6 even if I only have IPV4 enabled

When I try to call query.yahooapis.com with curl it's trying to use the ipv6 address but IPV6 is disabled on my computer $ curl -v 'http://query.yahooapis.com/' * Hostname was NOT found in DNS cache *...
0
votes
1answer
287 views

Installing and using cURL in Linux and use it in Eclipse as well?

I come from a Windows environment - I used cURL in Eclipse (C). I want to do the same in a Linux environment (Ubuntu). I followed the installation guide on how to install cURL, but how do you use cURL ...
0
votes
0answers
100 views

How to receive sse (server sent events) and execute bash commands then

How do i receive server sent events (sse) from a webserver and execute a bash command with the received data in debian (raspbian)? Is that possible with curl as i can already receive sse. Thanks ...
0
votes
0answers
74 views

Amazon S3 upload error: An exception occurred while uploading parts to a multipart upload

I am trying to upload a 30 GB file to Amazon S3 using the AWS PHP SDK. require('../vendor/autoload.php'); $client = new Aws\S3\S3Client([ 'version' => 'latest', 'region' => 'us-east-1' ]); $...
22
votes
3answers
14k views

Is “curl -u username:password http://example.com” secure?

Is curl -u username:password http://example.com secure? If not, can you give a brief explanation of how someone could obtain your password?
0
votes
0answers
248 views

Sending POST with esp8266 Module

I am trying to send a POST request from my esp8266 wifi module and I am getting errors. Such as: "Your browser sent a request that this server could not understand" I am trying to send: POST http://...
0
votes
1answer
124 views

Delivering a payload to a URL, using cURL

I need help figuring out the cURL request required to send the following data to a URL. {"requestid":"555555", "partnermatchid":"10000-000-0000-0000-000", "usercontext":{"ipaddressmasked":"XXX.XXX....
2
votes
2answers
289 views

Where does cURL store data while it's still downloading?

I'm using cURL on the command line to automate the downloading of many files from a server. I'm wondering where cURL stores its data whilst it's downloading. The reason I need this info is so I can ...
6
votes
2answers
7k views

Specifying minor TLS version when using curl

Is there a way to specify curl to use a specific TLS version? Like 1.1 or 1.2? I can see only sslv3 and tlsv1 options in command help. I took latest src and compiled it with openssl 1.0.1e. Still dont ...
0
votes
0answers
189 views

wget or cURL for dynamic recursive URLs (generated through directories and adding specific same text at the end)

I'm using a wget command with some options in order to refresh the cache module from all URLs of a Prestashop. This is the command: wget -r -D www.domain.com -l3 -P temp -x -R "*cart*","*order*","*My ...
1
vote
1answer
73 views

How can I run the equivalent “curl” command that performs similarly to this “wget” command?

I have a wget command: wget -r --no-parent -nH -nd -q --show-progress --progress=bar:force ftp://ftp.ncdc.noaa.gov/pub/data/swdi/database-csv/v2/ But it only works in newer versions of wget. ...
1
vote
2answers
80 views

Download a list of files and use domain name as filename

I got a bunch of files to download which works great with wget -i list_of_urls but my problem is, that wget uses the filename of the downloaded file. Is there a way (or a different tool) which is ...
5
votes
2answers
267 views

Firebug and cURL

I have been using Firebug and cURL for quite a while. Firebug is amazing to capture HTTPS POST request as opposed to a network analyser as it is aware of SSL/TLS negotiation. And cURL is amazing to ...
0
votes
1answer
62 views

How to extract ip address from no-ip account webpage in the shell?

I've set dynamic dns for my home network(which is behind NAT) using www.noip.com. I use ddclient to send updates of my ip, but sometimes it takes a while to update the ip. Anyways it's not the point. ...
4
votes
2answers
258 views

Get curl to output non-2xx to stderr, but 2xxs to a file

I would like some curl command, possibly with some bash hackery, that: Outputs any 2xx responses to a file Outputs any non-2xx responses to stderr Exits with a nonzero status code when a non-2xx ...
1
vote
1answer
76 views

How to run a remote bat file on a local computer

I need to run a batch file, that is stored on a server, locally on a windows 8.1 machine. I can run a shell script in OS X with a curl: cURL "file path" | sh is it possible to cURL in powershell in ...
0
votes
1answer
568 views

how to make curl re-use existing connection

curl --verbose google.com If I execute this command, after curl does its job, it says "Connection #0 to host google.com left intact". But it doesn't use same connection if I execute same command ...
1
vote
0answers
63 views
0
votes
1answer
152 views

What is the curl equivalent of wget -qO (wget -q -O)?

I need to run the following command in Red Hat Linux: wget -q -O http://somepage/somefile.cgi| bash But I don't have wget installed. I could install it, but I want to know how to do this with curl.
2
votes
1answer
170 views

Bulk SFTP download of files specified by URL preserving directories

I have a list of several hundred files I need to pull via SFTP. They are located in various directories and I need them to be in the same directory setup on my local machine. It is important the files ...
4
votes
1answer
1k views

What units does curl use for bandwidth?

curl on the command-line displays progress like this: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent ...
0
votes
1answer
475 views

Escape semicolon curl with -u option

I can't wrap my head around how to escape semicolon in username while making a GET request. curl -u testclient:sdtest1:pass -XGET -H "Content-type:application/json" \ http://localhost:8081/hello ...
0
votes
1answer
216 views

How to upload all the files from local to host using Curl?

i have written a shell script to upload all files with different names in the specific path from local to host, but i have facing some issue and the files are not getting uploaded #!/bin/sh for file ...
0
votes
2answers
423 views

Curl with ipv6 not working by default

I am trying to curl ipv6 adresses, but without success. If i try lets say: curl https://google.com I wait for a timeout and get network unreachable If i force ipv4, everything is fine. ...
0
votes
0answers
599 views

Curl: How to send custom headers while using proxy

I want to send custom headers to the host while using proxies. Take a look at the examples below. Explaining the situation with examples was easier. This works fine as expected: curl https://www....
2
votes
1answer
2k views

Mac OS X Yosemite and Client SSL Certificates

Ever since I upgraded from Mavericks to Yosemite, my SSL Client Certificates have stopped working in certain places. They still work in web browsers and in wget, but they have stopped working in curl ...
2
votes
2answers
139 views

Fast way to download many small webpages?

I need to download lots of small pages as fast as possible. Average size for one page is 3kb. There are millions of pages. First, I tried curl but it was way too slow, probably because it was waiting ...
1
vote
1answer
1k views

What is the correct way to submit content-type header in curl?

I’m using Mac 10.9.5 with bash shell. I’m trying to submit a request via curl, but I keep getting errors when I try and set the content-type header. Below I try davea$ curl -v -o -H "Content-Type: ...
0
votes
1answer
164 views

How to download the latest version of youtube-dl programmatically?

The latest version of youtube-dl is at: https://yt-dl.org/downloads/latest/ which redirects to (as of today): https://yt-dl.org/downloads/2015.07.04/ I'd like to set up a cron job to download the ...
0
votes
2answers
206 views

Curl post form and get result

I found this website http://www.fonefinder.net/findome.php , that locates phone provider by number (ex: http://www.fonefinder.net/findome.php?npa=562&nxx=866&thoublock=1666) shows VERIZON ...
8
votes
4answers
33k views

Downloading file from FTP using cURL

I'm trying to use a cURL command to download a file from an FTP server to a local drive on my computer. I've tried curl "ftp://myftpsite" --user name:password -Q "CWD /users/myfolder/" -O "myfile....
0
votes
2answers
680 views

use curl to download and yum to install (NTLM proxy issues)

I need to use a wi-fi under a corporate environment with my Fedora 21. It uses NTLM. With curl i've been able to reach sites (yahoo) setting the proxy, the type of proxy, the user and the password, ...
0
votes
1answer
450 views

Using a curl scheduled task in windows

I am using the following command in a scheduled task in windows to update my IP to the free Dynamic DNS service called EntryDNS: curl.exe -k -X PUT -d "" "Their url API string here" The problem I am ...
1
vote
0answers
61 views

Run bash command from link in browser

Is there a way to run an arbitrary command on OSX by clicking a link in the browser? As you know, there are certain types of hyperlinks that launch certain applications with a specified action, for ...
2
votes
1answer
614 views

Site displays with Curl but not within the browser

I've set up a dev-enviroment on Debian 7 and installed the following services: Apache Bind9 I have one site that is reachable through "asix.debian.local". I've set up the zone and added NS and A ...
0
votes
1answer
118 views

curl ignore --data starting with @ sign: don't read from file

In slack you can script slackbot to post messages to a channel like this: curl --data "$msg" $'https://<yourteam>.slack.com/services/hooks/slackbot?token=<yourtoken>&channel=#random' ...
3
votes
1answer
790 views

Why does “curl” command work normally while “ping” not?

I find a strange phenomenon in my corporate network: I can execute curl google.com command successfully: [root@localhost ~]# curl google.com <HTML><HEAD><meta http-equiv="content-...
0
votes
3answers
104 views

Why can't the guest of VirtualBox connect to network?

I have installed CentOS 7 on virtualbox, and network setting is NAT. The guest can't connect to network: systemctl -l status network output likes this: ip addr output likes this: Could anyone ...
0
votes
0answers
218 views

Why do wget and curl fail to download page source?

Via a shell script, I would like to download the source for a web page with air quality data on aqicn.org. Here is the URL for the page with Beijing air quality data: http://aqicn.org/city/beijing/m/ ...
1
vote
1answer
267 views

curl capturing http status and timing the request

I have a loop in wich I check urls from file one-by-one and want to get the status header and the execution time of every request as result. I have two curl commands: This one outputs the header ...
1
vote
2answers
689 views

What's the exit code for “curl -I” when not HTTP 200?

I want to check what HTTP status code is returned for an HTTP(S) URL. I don't care about content, so I just request head with curl -I $url or curl --head $url But what's the exit code I should ...