curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

1
vote
0answers
176 views

the right options to traverse/download the pages/directories of a subdomain

Let's suppose exist a site with the following directories (subdomain) index.php |-sub1 |-index.php |-sub1sub1 |-index.php |-other.php |-sub1sub1sub1 |-sub2 |-index.php |- ...
1
vote
4answers
820 views

How does ping figure out the IP address of a website?

I'm trying to understand better the different network-query tools such as nc, curl, whois, dig, nslookup and at the same time a bit more about the architecture of internet requests, servers, and the ...
2
votes
0answers
152 views

Curl --insecure error in Cygwin (works in Linux)

Curl is returning an error in Cygwin. How do I eliminate this error? curl --insecure -X POST -d '<run><log ...
0
votes
1answer
352 views

Error with curl

I'm having this error with curl: curl https://www.google.com -v * About to connect() to www.google.com port 443 (#0) * Trying 2607:f8b0:400e:c01::67... * Failed to connect to ...
3
votes
1answer
1k views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
3
votes
1answer
348 views

Why is there a very long 6 second delay in start of Curl

I did a full install of Cygwin on my Windows 7 x64 machine. After each machine reboot, Curl can take around 6 seconds to start (even if no command line options were specified). On subsequent uses of ...
1
vote
1answer
1k views

How do I install hiphop-php in CentOS 6?

I've been trying to install hiphop-php on CentOS 6 with no luck. I found an .rpm for it but it fails dependency checks on boost which fails to install because it depends on curl which fails to ...
0
votes
0answers
267 views

Make curl work on ssh connection

On my windows xp machine, I can connect to a remote host via putty and psftp. However I fail to establish a connection with curl. When I enter curl -v --key priv_key.ppk --pubkey pub_key.pub ...
1
vote
1answer
618 views

Have a txt file keep it's line breaks after being send over FTP via Curl

I'm using Curl to send a file over FTP. It is just a generated text file from a screen hardcopy command. When I cat the file, it displays correctly, it looks like this: System up for 14844 1, ...
0
votes
2answers
1k views

tar piped to curl: This does not look like a tar archive

curl -L http://download.transmissionbt.com/files/transmission-2.80.tar.xz -s -o - | tar -xf - The error: tar: This does not look like a tar archive tar: Skipping to next header tar: Exiting with ...
2
votes
0answers
124 views

curl works but ping/urllib2 don't, in a virtual machine

I have a Crunchbang 10 VM running on VirtualBox on a Windows 7 Home host. I have a python script that uses urllib2 that was failing, and discovered that: ordinary browsing and curl work normally on ...
1
vote
1answer
4k views

How can I send a http get request from command line under WinPE

I need to send a http get request under WinPE. My first thought was curl.exe. It works well under Windows but not WinPE. I committed curl.exe to boot.wim but when I tested it under WinPE, it returns ...
1
vote
0answers
161 views

Curl not working on OS X

I recently upgraded XCode and while I'm not sure if that caused the problem, I can't curl out of my computer anymore. This seems to break rvm and homebrew but git still works. I upgraded curl and ...
2
votes
1answer
416 views

What is more transparent, curl or wget in this web crawler project?

I have files on a Windows 7, xampp server (using altered port, 181). I am triggering the script from another computer (linux) using curl http://xxx.xxx.xxx:181/phpproject.php This PHP project is a ...
0
votes
2answers
298 views

Using curl on centos 5.9 for the same server hosted sites giving 404 [closed]

I am using the following command on my ssh terminal- curl -v -L http://mysite/path to request some data. However, its returning a 404 error(the site is hosted on the same server). But using curl ...
0
votes
1answer
233 views

Can't copy website with curl

I want to get all the images from a website behind our company firewall, similar to what was asked here. I can access the site from Firefox on OS-X, but when I try to use curl I get: You ...
0
votes
0answers
34 views

Curl ignores the -r switch when -C is present

I wanted to download a part of a file. (Let's say http://mysite.com/myfile.zip) So I used this command: curl -r 200000-400000 -O http://mysite.com/myfile.zip Unfortunately my Internet ...
2
votes
1answer
469 views

Do curl and wget open gzipped files automatically?

I made two http calls to the same gzipped resource from two different machines - RHEL and OSX. curl http://example.com/file.gz The first call returned an expected binary output: /�IQ���} The ...
1
vote
0answers
129 views

Permissions tar extraction to nfs share from a Perl script

As part of an installation procedure of a piece of software, I use: curl -s <url-to-targz> | tar -p -x -z -C / inside a Perl script (I use qx($command) or system($command), either will ...
0
votes
2answers
527 views

cURL or xargs: show progress

I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into ...
0
votes
1answer
181 views

Login webpage using wget

I have problems with logging in to this webpage using wget: http://dealers.volare-kinderfietsen.nl/index.php?dispatch=auth.login_form Normally this is no problem for me. I just want to store the ...
1
vote
1answer
1k views

cURL SFTP upload speed only 50% compared to command line SFTP

I need to use cURL to upload some files to an SFTP server, it has to be using cURL because I'll be later using cURL_multi etc to transfer files simultaneously (unless anyone knows a command line ...
2
votes
2answers
672 views

I am not able access certain website [closed]

I am trying to access www.psychologytoday.com and it fails every time. I tried proxysites and still but every time connection got refused. I've checked Google cached page of the same. Only a few ...
0
votes
1answer
502 views

How to add self signed certificate to certificate bundle?

How to add self signed certificate to certificate bundle so that the Curl http client can verify the self signed certificate as valid one?
1
vote
1answer
2k views

Upload a file with a comma in its name with curl

How can I upload a file named yes, this filename has a comma.txt with the cli curl? You would normally do this to upload a file with curl: curl --progress-bar -F "fileUpload=@filename.txt" However ...
-1
votes
1answer
329 views

cURL for Mac OS X Lion

I am looking for cURL for Mac OS X 10. I have found versions that go up to OS X 8 (Link), but no later. Is there a version for OS X 10 (Mac OS X Lion 10.7.5 (11G63)) or another tool?
0
votes
3answers
775 views

Batch - Get the url of to a file from html document

I want to use curl to download the latest version of this file. The site has a robots.txt, however, which is what I think is stopping me from just using curl -L -z WorldGuard.zip ...
0
votes
1answer
106 views

Why doesn't my script work?

The script I am using is FOR /F "tokens=91 delims=<>" %%v IN ('curl http://ci.onarandombox.com/job/Multiverse-Core/lastStableBuild/api/xml') DO ECHO %%v What I want it to do is search the output ...
3
votes
1answer
4k views

Download a file from the latest stable Jenkins build

Using a batch file and wget/curl, how can I download the Multiverse-Core-.jar from here? http://ci.onarandombox.com/job/Multiverse-Core/lastStableBuild/artifact/target/ I want to use the timestamping ...
0
votes
0answers
147 views

cURL wont download for me?

OS X 10.8.3, from a terminal I am executing: $ curl -OCv http://<URL>/<file>.dmg I get (for 2 seconds) % Total % Received % Xferd Average Speed Time Time Time Current ...
1
vote
1answer
302 views

curl ifconfig.me and ifconfig showing different results

If I run curl ifconfig.me I get one IP, and if I run ifconfig, I get other IP's. Why is it so ?
0
votes
1answer
697 views

curl (48) An unknown option was passed in to libcurl, cannot install composer

I'm trying to install composer via cURL but I cannot find how to download the phar archive. I've installed cURL from source activating in it's build ssl and ssh compatibility; I need composer to have ...
0
votes
2answers
856 views

Formatting cURL Output in the Windows Terminal

When I try to perform the cURL request here, Windows 7 (x64) returns an output with no line breaks: C:\Users\kiwi>curl http://www.documentcloud.org/api/search.json?q=group:nytimes ...
0
votes
2answers
828 views

What should I do in order to build curl without an error?

This failed when I run ./buildconf The error information is as follows: [mirror@home curl]$ ls acinclude.m4 CMakeLists.txt GIT-INFO MacOSX-Framework packages TODO-RELEASE ...
1
vote
2answers
488 views

How to stop cURL from writing over downloaded files

I'm using $ xargs -n 1 curl -O < gwurls.txt to grab a long list of files. Unfortunately, the site I'm grabbing from relies on the path to provide uniqueness, so -O doesn't know the difference ...
1
vote
0answers
554 views

Libcurl-devel on Mac OS X

I am trying to build an application that depends on libcurl version 7.21 or newer on Mac OS X 10.5.8. I tried downloading and installing the latest version from libcurl's website, but the application ...
1
vote
1answer
119 views

Is it possible to send the output of -w and -o Curl streams to different files? How?

I'd like to run a curl command, grep the output (response payload) for some keywords, then also print some metrics via the -w option. I think this would be easy to do if I could direct the -o ...
2
votes
0answers
345 views

PHP : curl is and Not working in ubuntu10.04 and Nginx

I am running Nginx server on ubuntu 10.04 I am using the php version 5.3.2 .I installed php5-curl using apt-get install php5-curl. Unfortunately the curl package is not displaying in phpinfo() . ...
0
votes
1answer
170 views

Get Directory/file structure without payload

I need to recursively retreive the file/directory structure of an FTP server, getting only file names (without payload). Can you help me how to do this (curl, wget, ...)? The result should be the ...
4
votes
1answer
2k views

Why does sudo not work with curl?

I ran a command to download something with curl and it threw a permission denied error. So I did sudo curl and I got the same error... Why is that? I was able to run the install command without ...
0
votes
0answers
468 views

Why can't I proxy SSL connections in Charles under Linux (ubuntu)?

I want to proxy some API requests over SSL from a PHP application using Charles Proxy which should allow me to view unencrytped requests. I've done this on Windows before but on Linux, I'm seeing the ...
0
votes
1answer
760 views

piping output from curl to rpm2cpio

I download alot of srpms to rummage through, I used to go through a long winded approach, root@server$ curl rpm -o file.rpm root@server$ rpm2cpio file.rpm | cpio -id but that leaves behind an ugly ...
0
votes
1answer
486 views

How to check for a value after curl or wget?

I need to curl/wget a page and make sure it returned a particular value (else return 1), something like this: curl http://example.com/something/run | grep -e '^success!$' || return 1 Which flags ...
2
votes
2answers
145 views

Use of curl command via HTTP

Using curl command, how to transfer a file from one location to other via HTTP request?
3
votes
1answer
373 views

Unable to resolve host using curl/wget while page loads in browser

The problem I am facing is that while I open a link in my browser it loads quickly but whenever I try using the link in terminal with for e.g. curl -0 www.google.com or wget ...
0
votes
4answers
125 views

Simple script parsing text, what is wrong here?

I'm a big user of https://www.grc.com/passwords.htm to get strong passwords. However, having to go to the site and manually copy the password every time gets old fast, so I decided to do a little ...
2
votes
1answer
1k views

using cURL to download all files in certain folder, except 404 error page?

I'm using cURL in ubuntu to download some files like pic001.jpg pic002.jpg pic003.jpg ... pic999.jpg While some files may be missing from this sequence but when I just use ...
1
vote
0answers
125 views

No output returned when curl ing any URL in Red Hat 6 but the same url works in lynx browser

I am trying to install some packages in Python using easy install. But i get a 'Download Error' for any package i try to install. So i tried to test if internet is accessible at all from the machine ...
0
votes
1answer
169 views
0
votes
1answer
1k views

Why can't I fetch this sites contents with curl?

I'm trying to download the contents of https://aarhustech.itslearning.com/elogin/default.aspx with curl, but I get this error: $ curl https://aarhustech.itslearning.com/elogin/default.aspx curl: (35) ...