curl is a data transfer utility used for a multitude of network=related applications (such as scraping data).

learn more… | top users | synonyms

3
votes
3answers
774 views

What's the equivalent of “wget -i FILE” in Curl?

What's the equivalent of "wget -i FILE" in Curl?
4
votes
3answers
3k views

Downloading XCode using wget or curl [closed]

Is there a way for downloading XCode using curl or wget? EDIT: Why is the following curl snippet not working? First, I login to Apple's site: curl -d "theAccountName=USERNAME&theAccountPW=PASS" ...
10
votes
4answers
4k views

How “legal” is site-scraping using cURL? [closed]

Recently I was experimenting with the cURL, and I found lot is possible with it. I built a small script that crawls a musical site, which plays online songs. On the way of my experiment, I found that ...
0
votes
1answer
71 views

how can i check the size on a server?

how can I check the size of file on server in bytes? i need to do it for following
0
votes
2answers
408 views

How to automatically get images from an SSL Site that uses Java Servlets

This is tricky. I want to use curl, wget or any other tool to login into a SSL Website which provides a Login Form. Then I want to visit several Links within that Domain and fetch certain images. I ...
25
votes
4answers
32k views

cURL: how to suppress strange output when redirecting?

I'm trying to print just the verbose sections of a cURL request (which are sent to stderr) from the bash shell. But when I redirect stdout like this: curl -v http://somehost/somepage > /dev/null ...
0
votes
3answers
3k views

CURL works only as root

I'm on an Ubuntu 10.04 installation and somehow curl stopped working with my user. I always receive: curl: (7) couldn't connect to host. It only works as root. Can anyone help me?
0
votes
1answer
229 views

cURL randomly stopped working

In the middle of writing an application that heavily relies on cURL I came across an error. The error cURL returns is <HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY&...
1
vote
1answer
107 views

screen scraper templates for various websites

I'm looking specifically for a convenient way to locally archive posts from this and other similar sites. I'd like to separate the question itself from the answers, or maybe crop the question and ...
3
votes
2answers
2k views

Curl: Difference between 2 ssl commands

I cant tell the difference or when to use one over the other. (Should this be on serverfault?) -E/--cert (SSL) Tells curl to use the specified certificate file when getting a file with ...
2
votes
1answer
812 views

PHP/cURL/Fedora issues resolving hostname possibly a DNS problem?

I have the following simple test script for cURL/php on a FC12/apache server that has PHP 5.3.2, cURL v7.19.7 (according to phpinfo()): $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, 'http://www....
0
votes
3answers
4k views

unauthenticated http proxy, wget fails, curl works

I'm running a ubuntu 10.04 i386 desktop edition. I have configured the http proxy via the http_proxy variable. When I use wget I get a connection refused but, curl, apt-get etc. work just fine. Any ...
2
votes
1answer
3k views

Using cURL through SSH tunnel or VPN

I would like to set my CURL to use SSH tunneling for certain domains. How can I accomplish that? I can also set up VPN or SOCKS or whatever, but I need to use CURL on local machine, but use the IP of ...
3
votes
1answer
2k views

Use cURL with multiple POSTs

I'm trying to use cURL to download the contents of webpages that require forms to get to. In a browser it looks something like this 1.) Login using POST 2.) Pick which page to go to using another ...
7
votes
2answers
6k views

Persistent retrying resuming downloads with curl

I'm on a mac and have a list of files I would like to download from an ftp server. The connection is a bit buggy so I want it to retry and resume if connection is dropped. I know I can do this with ...
63
votes
7answers
160k views

Run cURL commands from Windows console

Is there a way to install cURL in windows in order to run cURL commands from the command prompt?
1
vote
1answer
538 views

help with Outlook Exchange server and curl

I work on a mac in a building full of PCs, and the IT department here doesn't have IMAP access turned on on the exchange servers. So I miss a lot of meetings because I don't get reminders because I ...
0
votes
2answers
911 views

Make environment variables available when running command as sudo

I am using Curl, and am having a problem trying to get it to recognize the $http_proxy environment variable when using sudo curl. I tried putting export http_proxy=.... in my /etc/profile and ...
9
votes
2answers
2k views

How to download parts of the same file from different sources with curl/wget?

I have a quite large file hosted on five different servers. I would like to be able to download different parts of the file from each server and subsequently concatenate the parts, in order to ...
5
votes
1answer
2k views

cURL hangs trying to upload file from stdin

I'm trying to PUT a file with cURL. This hangs: curl -vvv --digest -u user -T - https://example.com/file.txt < file This does not: curl -vvv --digest -u user -T file https://example.com/file.txt ...
1
vote
3answers
770 views

How to retrieve all *.html files from website using Unix command line tools and regular expression

I would like to retrieve all .html files from a website that has a certain text on its name: eg. this_is_good_site.html So, I would like to download .html files with a word "good" on its name. I ...
1
vote
2answers
1k views

Download Speed cURL

I am having a bit of problem using cURL to download using FTP, that is to say I think it's being rather slow if working at all. I usually use wget an this one file which I downloaded earlier with wget ...
0
votes
1answer
1k views

piping curl's response headers

I'm getting weird results when trying to capture curl's response headers in a variable: pattern="< Content-Length: " val=$(curl --verbose \ --request POST --data "desc=hello world" \ ...
4
votes
3answers
16k views

URL to reboot a WebSTAR DPC2100R2 cable modem with curl?

Does anyone know what the magic URL is for rebooting a WebSTAR DPC2100R2 with curl? I used to have a SurfBoard, and the curl command: curl http://192.168.100.1/reset.htm?reset_modem=Restart%20Cable%...
3
votes
1answer
1k views

Force curl to download an empty file

In a bash script, I am using curl command on linux to download files from a FTP server. The command I am using is the following (the information about the FTP server connexion is stored in a .netrc ...
1532
votes
6answers
1.6m views

What is the cURL command-line syntax to do a POST request?

How can I make a POST request with the cURL command-line tool?