curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

0
votes
0answers
5 views

Can someone explain the differences of file_get_contents vs guzzle vs curl?

I've been getting data from api websites by using file_get_contents or curl, but I never knew whats the difference from them. They seem to have the same results, but now there is Guzzle for php. What ...
1
vote
2answers
4k views

CURL Error version OPENSSL_1.0.1 'not found …'

I have the last few days, a concern with CURL. I get this error: curl: / lib/i386-linux-gnu/libssl.so.1.0.0: version `OPENSSL_1.0.1 'not found (required by / usr/lib/i386-linux-gnu / libcurl.so.4) ...
3
votes
1answer
2k views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
2
votes
3answers
4k views

curl http_code of 000

I have a shell script that I use to monitor loading times and response codes on my live server cluster. It runs a total of 250 iterations every 5 minutes, distributed across 10 servers and 6 sites. It ...
-2
votes
0answers
20 views

wget to upload file to HTTPS server

Is there any way to upload files to HTTPS server using wget or can we use curl to upload to HTTPs with certification. Any help would be appreciated
0
votes
1answer
25 views

Curl command to download a file over HTTPS

I need to download a particular file from https://file_path. Here is the curl command I use for that: curl --user user:password "https://file_path?raw" > location I wish to download the real ...
2
votes
1answer
41 views

GIT from source: fatal: Unable to find remote helper for 'https'

Due to me not having root rights, I compiled git from source locally. My distro is OpenSUSE 13.1. Googling the error fatal: Unable to find remote helper for 'https' It was suggested that curl is not ...
0
votes
1answer
4k views

curl failed setting cipher list

I'm trying to make curl use GOST2001-GOST89-GOST89 cipher which is available and usable by OpenSSL but keep getting failed setting cipher list error despite the fact that curl sees gost engine and can ...
0
votes
2answers
30 views

How can I archive someone's Facebook profile page?

I thought about using archive.is or archive.org. I tried to view the page in a private browser, but it asks to log in, so that site will see the same thing. I tried to open the developer console and ...
2
votes
2answers
3k views

libcurl or cURL not working OSX Mavericks

Before I start let me say I have uninstalled, reinstalled, compiled from source, used macports and generally tried for the last 6 hours to get curl working on my Mac OSX Mavericks but I still haven't ...
59
votes
6answers
63k views

PowerShell equivalent of curl

Is there an equivalent of curl in PowerShell? Does it have some similar built-in capability or is there a 3rd party cmdlet?
0
votes
1answer
22 views

Sending gist to github via cURL and issues with new lines inside file

So, let's say I have an index.php file containing this: <?= "Hello" ?> <?= echo "WORLD" ?> And I wanted upload the contents of this file to my gists in github, which I am doing via ...
1
vote
1answer
45 views

Can't update meteor, because it can't download manifest

I've meteor installed via npm and I'd like to update it, but I can't. I've tried: $ meteor update Failed to download manifest. but it fails with: Failed to download manifest. So then I've ...
0
votes
0answers
17 views

curl lookup time is 5.5 seconds when accessing address in /etc/hosts

In my /etc/hosts file I have a custom entry since the name doesn't actually exist like this: 5.5.5.5 someinternalserver.thing.net #thats a fake IP for this post Using curl like so: curl -o ...
0
votes
1answer
33 views

How can you “press” a button of a webpage from the command line to restart a cable modem?

I'm trying to reboot my cable modem programmatically. I used to be able to do this by using wget: wget http://192.168.100.1/reset.htm?reset_modem=Restart+Cable+Modem However I changed my cable ...
1
vote
1answer
7k views

How can I send an HTTP GET request from the command line under WinPE?

I need to send an HTTP GET request under Windows Preinstallation Environment (WinPE). My first thought was curl.exe. It works well under Windows, but not WinPE. I committed curl.exe to boot.wim, but ...
0
votes
1answer
19 views

cURL command units of Dload speed?

When I run curl I sometimes see a progress meter like this: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total ...
0
votes
1answer
22 views

Psexec with curl or wget

I recently found out about psexec and was wondering if this program could help me run wget or curl commands on a remote computer. is this even possible?
24
votes
8answers
56k views

Wget/cURL alternative native to Windows?

Is there a Wget or cURL type command line utility native to Windows Vista? How does it work?
0
votes
1answer
7k views

Curl returns “Connection Refused” when trying to access XAMPP via localhost

I am running XAMPP (Apache, MySQL and PHP) on my Windows Vista computer. I have curl installed on my machine. When I try to curl my localhost in the command prompt, I get the following: curl -v -i ...
6
votes
1answer
1k views

How did Google get on my Mac?

I am running a MacBook Pro, and have never installed Chrome, Google Earth, or anything blatantly Google. Just installed Little Snitch (are there no good free firewalls for Mac?) and see that CURL is ...
4
votes
1answer
254 views

Automatic login to Vodafone Community WiFi (Fon) via Unix script

First I have to say that my question is similar to this one I am using a Vodafone Community Fon AP and it often disconnects. In OS X, which I'm using, there is a tool called Captive Network Assistant ...
0
votes
0answers
26 views

Bashscript using curl to crawl a website that needs a login

I am looking to crawl a website that has kind of a newsfeed with elements hat have a timestamp. There's only a certain amount of elements in the feed and a "Previous" button at the end of the feed. It ...
0
votes
1answer
55 views

Telling cUrl not to normalize a url

When using curl to call some url, it simplifies the path in the url (for example from /a/../b to /b). How can I tell cUrl not to do so? $ curl -V curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 ...
0
votes
3answers
59 views

Sending a POST request via curl has issues with double quotes

So, in-order to send a POST request to gists (github), you can do something like this as seen in https://gist.github.com/caspyi... curl --user "user" -X POST --data '{"description":"Created via ...
0
votes
1answer
94 views

curl command not found on OSX El Cap

My understanding is that curl should be installed. I had tried using curl from the Unix command line and got the message it could not be found. I'm wondering If something I did in efforts to clean up ...
1
vote
2answers
117 views

Curl response hangs

I have a site set up in a virtual box. I am able to access it just fine from a browser on my machine, but I have trouble accessing it via CURL while SSH'ed in to the box. When I try, curl hangs ...
209
votes
9answers
229k views

Getting curl to output HTTP status code?

I'm using curl at the command line on Linux to issue HTTP requests. The response bodies are printed to standard out, which is fine, but I can't see from the man page how to get curl to print the ...
0
votes
0answers
9 views

mirror site behind login

I would simply like to mirror all the pages (millions) on a site behind a login. This is for personal use. I have a login for the site in question but I want to scrape it into a database. I ...
-1
votes
1answer
30 views

How to use CURL tool?

I just got introduced to CURL today and I went to their website to download the tool and all I got is .cgi file called "metalink" and I don't know how to use it or how to run it, can someone help ...
1
vote
0answers
109 views

curl suddenly unable to get local issuer certificate

I've been using Docker 1.7.1 with base Ubuntu 14.04 image and all of a sudden this curl: curl -L -o sbt.tgz https://dl.bintray.com/sbt/native-packages/sbt/0.13.9/sbt-0.13.9.tgz is failing me with ...
0
votes
1answer
26 views

How to use VERB in url

I am trying to create and API which uses VERB PUT,GET,DELETE I can use it in curl by this curl -X DELETE http://localhost:8080/recipes/hi But how can i use similarly in url for eg This works ...
0
votes
1answer
41 views

Dictionary attack in Bash with cURL

I'm fairly new to Bash, but this is a script I've been working on for the past few days. It's meant to run a dictionary attack on an HTTP-POST login for a website, by reading the time module and ...
27
votes
3answers
14k views

Curl local host names on Mac OS X Yosemite

I just upgraded from Mavericks to Yosemite, and now curl can't see loopback host names. Set up a simple http server to test: $ python -m SimpleHTTPServer Serving HTTP on 0.0.0.0 port 8000 ... Now ...
1
vote
0answers
594 views

How to use .pfx in certificate store in CURL for Windows 7

I want to use cURL to access a remote server, and I need to identify myself using a private key or the remote server gives a 401 "Unauthorized". I'm on Windows 7 (work machine, not my choice) and my ...
1
vote
0answers
15 views

“curl” ignores proxy environment variable when using an ip address?

I am using curl to access some web services through a proxy (an instance of polipo running on port 8123, chained to an ssh SOCKS5 proxy). If I provide proxy information using the http_proxy ...
2
votes
1answer
3k views

Upload a file with a comma in its name with curl

How can I upload a file named yes, this filename has a comma.txt with the cli curl? You would normally do this to upload a file with curl: curl --progress-bar -F "fileUpload=@filename.txt" However ...
0
votes
1answer
12 views

Neither wget nor curl downloads this wikipedia api request

I can perfectly download this link and save it as a text file in a web browser environment. But I can't download and save that wikipedia api request using neither wget nor curl. I use wget -O file.txt ...
0
votes
2answers
103 views

How to POST file contents using cURL?

I want to do an HTTP POST of the contents (as a string) of a local file located at path/to/my-file.txt to a URL endpoint at http://example.com/. For example, I might want to do the following: ...
0
votes
0answers
180 views

Make a curl POST request from scraped data

I scraped a website using the simple DOM php parser : <?php include 'domparse.php'; $html = file_get_html('http://oceanofgames.com/rebel-galaxy-free-download/'); ...
0
votes
1answer
19 views

libcurl version 7.21 to work on Redhat 6

I am having an issue with libcurl version 7.19 and socks proxy on redhat 6. I believe it was fixed in libcurl version 7.21 how do i upgrade libcurl on redhat 6 to 7.21 or greater?
0
votes
2answers
87 views

cURL connecting to localhost (127.0.0.1) instead of destination IP

On Mac OS X I have an issue where calling: curl -v www.google.com results in: * Rebuilt URL to: www.google.com/ * Trying 62.253.72.153... * Connected to www.google.com (127.0.0.1) port 80 (#0) ...
0
votes
3answers
493 views

Is it possible to get the download file name of an mp3 from server using bash?

I am trying to get the final download file name of an mp3 from a server, specifically audiotool.com e.g. http://api.audiotool.com/track/haunt_opaque/mixdown.mp3 When you navigate to this url in a ...
2
votes
0answers
139 views

Retrieving from ftp dirs with wildcards

What is the most efficient way to retrieve all relevant data from ftp if the address of said data can be specified with wildcards as below: ...
2
votes
0answers
308 views

SSL error when connecting to git from Mac OS X 10.11.3 (El Capitan)

When I try to clone / pull / fetch from a git repository via https from a Mac System I keep getting an error: error: RPC failed; result=56, HTTP code = 200 fatal: The remote end hung up unexpectedly ...
8
votes
2answers
669 views

How can I use a command line tool like Wget to log into an OpenID site?

Specifically, I would like to be able to download certain pages from my user profile on the various Stack Exchange sites. I would, however, like to do this automatically (using a cron job), from ...
3
votes
2answers
134 views

curl check if file is newer and instead of downloading - execute a bash (or python) script

I am having a bit of an issue. I have a file, hosted on a remote server (http://mywebsite/file.zip). I also have a few embedded linux boxes (running openelec OS). The boxes have fairly limited ...
61
votes
7answers
156k views

Run cURL commands from Windows console

Is there a way to install cURL in windows in order to run cURL commands from the command prompt?
1
vote
1answer
44 views

Fetch source file from GitHub using command line tools?

This question is related to a question offsite at FATAL: MemorySanitizer can not mmap the shadow memory. The person who responded said: Do simple tests (e.g. this one) work for you? The link ...
-1
votes
2answers
24 views

Where is my file saved locally?

I've downloaded a file--I see the contents of my text file in the terminal window as it is downloaded in curl, but where is it saved on my local machine: curl -v -O myFile.txt -u username:password ...