curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

3
votes
0answers
420 views

pipe unlimited curl stdout into chunked files

I've got an HTTP-endpoint that serves an unlimited stream of events. Now I'd like to record that stream using curl, but splitting the recorded files at the same time - preferably based on the line ...
2
votes
0answers
140 views

Retrieving from ftp dirs with wildcards

What is the most efficient way to retrieve all relevant data from ftp if the address of said data can be specified with wildcards as below: ftp://ftp.some.site.gov/data/directory/only/*/these/*/*...
2
votes
0answers
540 views

SSL error when connecting to git from Mac OS X 10.11.3 (El Capitan)

When I try to clone / pull / fetch from a git repository via https from a Mac System I keep getting an error: error: RPC failed; result=56, HTTP code = 200 fatal: The remote end hung up unexpectedly ...
2
votes
0answers
284 views

Posting XML wrapped in SOAP posted using Curl results in invalid XML character was found exceptions

Sending an XML file using curl results in: Invalid XML character (Unicode : 0x<integer>) Attempt to solve the issue According to this SO answer the issue will be solved if the XML version is ...
2
votes
0answers
311 views

Why doesn’t curl install and run on my 64-bit Windows 8 system but the 32-bit version works fine?

I almost gave up installing cURL on my 64-bit Windows 8 system. I tried almost every single .zip, .exe, etc… on this site, but none of them worked. I got this message every single time: The ...
2
votes
0answers
40 views

How to add a 'greek' character in the path using cURL?

I have a Makefile where one of many tasks is to upload a file using cURL, one of these files, has a greek character in the name and when I try to upload using cURL, I receive this error: curl: (43) A ...
2
votes
0answers
554 views

loop curl could not resolve host

I've had working script, which suddenly doesn't works. It contains something like this: while read url; do curl $url done < links links file: http://duckduckgo.com http://google.com And I ...
2
votes
0answers
191 views

Curl --insecure error in Cygwin (works in Linux)

Curl is returning an error in Cygwin. How do I eliminate this error? curl --insecure -X POST -d '<run><log encoding="hexBinary">4142430A</log><result>0</result><...
2
votes
0answers
972 views

Libcurl-devel on Mac OS X

I am trying to build an application that depends on libcurl version 7.21 or newer on Mac OS X 10.5.8. I tried downloading and installing the latest version from libcurl's website, but the application ...
2
votes
0answers
212 views

curl works but ping/urllib2 don't, in a virtual machine

I have a Crunchbang 10 VM running on VirtualBox on a Windows 7 Home host. I have a python script that uses urllib2 that was failing, and discovered that: ordinary browsing and curl work normally on ...
2
votes
0answers
530 views

PHP : curl is and Not working in ubuntu10.04 and Nginx

I am running Nginx server on ubuntu 10.04 I am using the php version 5.3.2 .I installed php5-curl using apt-get install php5-curl. Unfortunately the curl package is not displaying in phpinfo() . ...
2
votes
0answers
913 views

Restore default CA Certificates on Mac OSX 10.7

I've been screwing around with my Mac, trying to install self-signed certificate as trusted for our WAN site (for which we can't justify paying for a real signed certificate) I've broken something ...
1
vote
0answers
28 views

can't connect to https API using cURL?

I have API endpoint URL, certificate file ( cert.p12 ) & password. Here is the steps I try to connect to this API. ( IN Windows 7 ) First I type the URL in chrome browser and it didn't give me ...
1
vote
0answers
36 views

Centos cURL connection refused on Its own IP

I have two services on one server (with different ports to access them). One of my services is trying to connect to the other one with the IP of server and its specific port. But I got some errors ...
1
vote
0answers
20 views

how to upload files to https server with certificates using curl or any other commands

I am Trying to upload to my HTTPS server using curl .I am using curl for that ,the code is curl -v --cacert /tmp/WIN-CLS.vdi.com.pem -T /etc/hosts \ --user administrator:pass_123 https://xxx.xxx....
1
vote
0answers
122 views

Can someone explain the differences of file_get_contents vs guzzle vs curl?

I've been getting data from api websites by using file_get_contents or curl, but I never knew whats the difference from them. They seem to have the same results, but now there is Guzzle for php. What ...
1
vote
0answers
21 views

“curl” ignores proxy environment variable when using an ip address?

I am using curl to access some web services through a proxy (an instance of polipo running on port 8123, chained to an ssh SOCKS5 proxy). If I provide proxy information using the http_proxy ...
1
vote
0answers
219 views

curl suddenly unable to get local issuer certificate

I've been using Docker 1.7.1 with base Ubuntu 14.04 image and all of a sudden this curl: curl -L -o sbt.tgz https://dl.bintray.com/sbt/native-packages/sbt/0.13.9/sbt-0.13.9.tgz is failing me with ...
1
vote
0answers
18 views

Curl in bash code - produces wrong status

This code seems to ouput a http status of 000 - which seems to indicate something didn't connect properly but when I do this curl outside of the bash script it works fine and produces a 200 so ...
1
vote
0answers
67 views

How do you download a zipped build from a jenkins job using “jenkinsapi” module in Python?

Has anyone tried download a build using jenkinsapi module in Python?
1
vote
0answers
61 views

Run bash command from link in browser

Is there a way to run an arbitrary command on OSX by clicking a link in the browser? As you know, there are certain types of hyperlinks that launch certain applications with a specified action, for ...
1
vote
0answers
162 views

FTP: wildcard downloading

I am trying to download a file with name ending with *.csv The problem is complicated by the fact that this *.csv contains in directory of unknown name: eg. Sometimes it would be 2015->Date 29:10:...
1
vote
0answers
293 views

I destroyed my Mac's PHP setup: SSL certification fails

Long story short, I deleted the PHP symlink on my Mac, and replaced it with a symlink from XAMPP. Now it seems that there's a problem with my SSL configuration or what seems to me, a lack thereof. ...
1
vote
0answers
13 views

Sporadic loss of access to local server

For some reason, I lose access to a local server called lux-app04 seemingly at random, but only in some applications. As you can see in the image below, Postman (top) can connect to it, but curl (...
1
vote
0answers
124 views

curl won't shut up (--silent does nothing)

I'm writing a quick shell script, and I cannot for the life of me get curl to suppress the progress bar. The relevant line looks like this: curl http://website -o $temp >/dev/null 2>&1 ...
1
vote
0answers
373 views

Curl to download large CSV

I am using curl through Cygwin on Windows 7 to download content from Apache Solr in CSV format. This is my command which is running in a for loop curl -o ret-key-data-$i.csv $URL As the content is ...
1
vote
0answers
33 views

Bug or syntax change on Debian's “curl” 7.36? Mess when trying to save sequential files with brackets/hash

(Perhaps specifically 7.36.0-1, which is shown as the most recent available on my package manager). I think there was such a bug some time ago, but perhaps it was back in curl 7.33, I can't remember....
1
vote
0answers
60 views

SOAP-request realization in DOS 6.22 16-bit

I have a task to develop some application providing SOAP-requests realization for DOS. I have already developed a similar application under Linux, using C++/STL and curl and openssl libraries. Is ...
1
vote
0answers
105 views

How do I log in to “The Cloud” wifi hotspots using curl?

In England at least, there's this accursèd thing called 'The Cloud', which increasingly provides all public wifi hotspots. Has anyone already knocked up a curl command/bash script/etc to log in to it ...
1
vote
0answers
363 views

Curl not working on OS X

I recently upgraded XCode and while I'm not sure if that caused the problem, I can't curl out of my computer anymore. This seems to break rvm and homebrew but git still works. I upgraded curl and ...
1
vote
0answers
168 views

Permissions tar extraction to nfs share from a Perl script

As part of an installation procedure of a piece of software, I use: curl -s <url-to-targz> | tar -p -x -z -C / inside a Perl script (I use qx($command) or system($command), either will do....
1
vote
0answers
145 views

No output returned when curl ing any URL in Red Hat 6 but the same url works in lynx browser

I am trying to install some packages in Python using easy install. But i get a 'Download Error' for any package i try to install. So i tried to test if internet is accessible at all from the machine (...
1
vote
0answers
222 views

the right options to traverse/download the pages/directories of a subdomain

Let's suppose exist a site with the following directories (subdomain) index.php |-sub1 |-index.php |-sub1sub1 |-index.php |-other.php |-sub1sub1sub1 |-sub2 |-index.php |- …....
1
vote
0answers
142 views

Download web server structure with empty files

I want to make a mirror of a Web server, but downloading the actual files will take too long. So I thought of having just the directory and file structure, and when I need the actual contents of the ...
1
vote
0answers
678 views

VirtualBox Ubuntu guest on OS X Lion host: Curl won't work

I'm trying to install some gems on an Ubuntu VM on OS X Lion using VirtualBox. I've narrowed my problem down to curl which is not responding at all. It seems to be going through some proxy local host ...
1
vote
0answers
217 views

how to use to proxy in the intranet?

i have an intranet site which is inside only our company and feeds information. i download an weather API from http://www.foreca.com/Philippines/ and hook to my site. the API has a search engine and ...
0
votes
0answers
6 views

wget keeps consuming space on C drive when run from another drive

I have wget download a 200gb file with the -c setting, run from cmd. Yet it keeps making a ghost copy to C drive, a copy I cannot even locate. Have to abort each time it completely eats up my SSD or ...
0
votes
0answers
9 views

use curl to download media files

I would like to download a music from a HTTPS website. Now they don't offer a specific download button, so I imagined using curl in Terminal. Sth like that: curl -O https://urlhere This normally ...
0
votes
0answers
32 views

Schedule on crontab to reboot router

I would like your much aprreciated help. I have a ISP router (gateway) that doesen't have a schedule reboot on the web interface. I have a raspbian and I would like to schedule on crontab to reboot ...
0
votes
0answers
21 views

jq filtering curl- 'Cannot index number with string'

I am trying to filter a curl request with jq to get specific fields and put them in variables. Initially, I was using a basic curl request such as this: myResponse=$(curl -u myUsername:myPassword -...
0
votes
0answers
14 views

Use curl to download an LDAP attribute value only

I am using the following curl command to download a value from an LDAP directory: curl ldap://example.com/CN=ACME-Test-CRL,O=ACME?certificateRevocationList;binary -o crl.crl The problem is it ...
0
votes
0answers
105 views

curl: (56) Recv failure: Connection reset by peer

I am trying to download one file from my local server using curl.i am using command curl -L -O --retry 999 --retry-max-time 0 -C - http://192.168.1.74/gio-v2-9515.dd -u ajay:aj123 For experiment ...
0
votes
0answers
28 views

curl lookup time is 5.5 seconds when accessing address in /etc/hosts

In my /etc/hosts file I have a custom entry since the name doesn't actually exist like this: 5.5.5.5 someinternalserver.thing.net #thats a fake IP for this post Using curl like so: curl -o /dev/...
0
votes
0answers
99 views

Bashscript using curl to crawl a website that needs a login

I am looking to crawl a website that has kind of a newsfeed with elements hat have a timestamp. There's only a certain amount of elements in the feed and a "Previous" button at the end of the feed. It ...
0
votes
0answers
15 views

mirror site behind login

I would simply like to mirror all the pages (millions) on a site behind a login. This is for personal use. I have a login for the site in question but I want to scrape it into a database. I ...
0
votes
0answers
34 views

Finding a text string on a website and all its links

Can you help me understand process of finding a certain text string on a website and all its immediate links? I started delving into wget and curl with help of grep but was unable to come up with ...
0
votes
0answers
27 views

Command to fetch a list of urls with connection keep-alive but fail on the first error (404/500)

I am trying to reproduce a potential bug in the Jetty web server with regard to serving static files. I need to repeatedly download a list of urls with "Connection: keep-alive" until one of them ...
0
votes
0answers
29 views

OSX terminal : log on on a website

I would like to login/ out on a Web site using the terminal, I currently using a appleScript I wrote but was wondering can you use terminal command line instead (not running my script from terminal) ...
0
votes
0answers
13 views

Indexing file resources from a website

How can I get a list of all (image) resources from a website? Is there a better wait than using wget and parsing the log stream in order to get the list of all images? Worst is, I'm pretty use I ...
0
votes
0answers
112 views

Using curl with a https proxy

I have setup a squid proxy so that it listens on a HTTPS port instead of the usual HTTP port, in order to have encrypted communication between the client and the proxy itself. While I have been able ...