0
votes
0answers
23 views

Downloading an authenticated large file with command line tools

I need to download a file sized around 1.6GB. I's like to get it download via a command line tool like axel, wget or curl However, how can I download a file which needs to be logged in to the site ...
0
votes
0answers
25 views

How can I get all urls of pattern x from a paged list of search results?

I'm using Windows 7 SP1, and I'm willing to use powershell or wget or whatever will work. I downloaded powershell today (ps 4) and I've got some scripts working, but I'm having trouble with this one. ...
0
votes
0answers
43 views

Parallel download for a list of urls and renaming

I have a list of tab-separated urls, and target file names, urls_to_download.txt, for example: first_file.jpg\thttps://www.google.co.il/images/srpr/logo11w.png ...
0
votes
1answer
37 views

How to automatically save a list of files with different names?

I have a text file of URLs, each in the form of: http://domain.com/a/b/c/d.jpg I want to download all of these, but save each file under the name: c_d.jpg In other words, for each file, I want to ...
0
votes
0answers
83 views

How can I use Wget / Curl / Plowshare to download from 1fichier.com as a premium member?

How can I download files from 1fichier.com (I am a premium member) with the help of Wget / Curl / Plowshare (Plowdown) or other CLI / command line tool / script?
1
vote
1answer
52 views

What is the curl equivalent to wget -x?

I want to use curl to save multiple files with the same local paths as would be produced by wget -x (ie, cleaned up versions of the full urls). But I can't find the right option in the curl docs. Is ...
0
votes
0answers
46 views

wget sometimes fails, curl succeeds

I am continuously downloading some packets via wget. Once in a while, the wget fails. I have used retries as 2 and timeout as 10 seconds. When the failure occurs, I try the same URL again, but with ...
0
votes
0answers
31 views

Gzipped file arrives unzipped

I've recently set up an ArchLinux system at work and added the ArchHaskell repo. This is far from the first time I do this. I have a number of systems where it's worked just fine, but this last time ...
2
votes
1answer
496 views

How can I download private repo from github?

What I want to do is to download private repository archive from GitHub, extract it, remove archive file and copy some directories that are inside downloaded project. I tried to use wget but I cannot ...
2
votes
1answer
160 views

How can I get a list of all messages in my gmail account in linux command line?

How can I get a list of all messages in my gmail account in linux command line? Ideally I would want to see something similar to: from:sender1@domain.com to:me@gmail.com Date:11/22/33 subj:Foo1 ...
0
votes
0answers
83 views

Pinging local machine with external IP

On a Gentoo server, I am trying to wget or curl a page from its url: wget http://example.com/test.html (I can get result for this url from my browser) All I receive is Timeout. With cUrl I get ...
0
votes
0answers
18 views

Can One Extract a Hyphenated Filename File From a Server Using Curl?

The images I'm dealing with have file names like part-1.jpg, object-a.jpg, Aspect-0.jpg. What I'm wondering is what I put in place of ???: curl -O http://server/[???].jpg
3
votes
1answer
854 views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
4
votes
1answer
369 views

How can I use a command line tool like wget to log into an OpenID site?

Specifically, I would like to be able to download certain pages from my user profile on the various SE sites. I would, however, like to do this automatically (using a cron job), from the command line ...
2
votes
1answer
292 views

What is more transparent, curl or wget in this web crawler project?

I have files on a Windows 7, xampp server (using altered port, 181). I am triggering the script from another computer (linux) using curl http://xxx.xxx.xxx:181/phpproject.php This PHP project is a ...
1
vote
1answer
304 views

Do curl and wget open gzipped files automatically?

I made two http calls to the same gzipped resource from two different machines - RHEL and OSX. curl http://example.com/file.gz The first call returned an expected binary output: /�IQ���} The ...
0
votes
1answer
159 views

Login webpage using wget

I have problems with logging in to this webpage using wget: http://dealers.volare-kinderfietsen.nl/index.php?dispatch=auth.login_form Normally this is no problem for me. I just want to store the ...
3
votes
1answer
3k views

Download a file from the latest stable Jenkins build

Using a batch file and wget/curl, how can I download the Multiverse-Core-.jar from here? http://ci.onarandombox.com/job/Multiverse-Core/lastStableBuild/artifact/target/ I want to use the timestamping ...
0
votes
1answer
137 views

Get Directory/file structure without payload

I need to recursively retreive the file/directory structure of an FTP server, getting only file names (without payload). Can you help me how to do this (curl, wget, ...)? The result should be the ...
0
votes
1answer
371 views

How to check for a value after curl or wget?

I need to curl/wget a page and make sure it returned a particular value (else return 1), something like this: curl http://example.com/something/run | grep -e '^success!$' || return 1 Which flags ...
0
votes
2answers
430 views

cURL or xargs: show progress

I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into ...
2
votes
1answer
279 views

Unable to resolve host using curl/wget while page loads in browser

The problem I am facing is that while I open a link in my browser it loads quickly but whenever I try using the link in terminal with for e.g. curl -0 www.google.com or wget ...
1
vote
0answers
134 views

the right options to traverse/download the pages/directories of a subdomain

Let's suppose exist a site with the following directories (subdomain) index.php |-sub1 |-index.php |-sub1sub1 |-index.php |-other.php |-sub1sub1sub1 |-sub2 |-index.php |- ...
1
vote
0answers
112 views

Download web server structure with empty files

I want to make a mirror of a Web server, but downloading the actual files will take too long. So I thought of having just the directory and file structure, and when I need the actual contents of the ...
2
votes
2answers
2k views

How do I unzip a compressed file as it's downloading?

I'm downloading a large file that's not an archive, and I want to combine the tasks of downloading and decompressing. How can I do them simultaneously?
0
votes
1answer
250 views

What tool does yum use to download rpm packages from remote server?

My system is centos, and now I am installing packages using yum command. Updating for dependencies: apr-util i386 1.3.9-1.el5 lxcenter-updates 203 k gcc-gfortran ...
1
vote
1answer
1k views

How do I remotely fetch files from redirected URLs from a terminal?

I want to fetch a tarball of this python library from terminal. https://github.com/simplegeo/python-oauth2/downloads However, I cannot simply call wget ...
1
vote
1answer
651 views

Grab source from Google Code using wget or curl?

Is it possible to grab the source-code from a Google Code repository (hg) using wget or curl? Else using Python? Thanks for all suggestions FYI: I'm on a shared WebFaction host, so can't install hg ...
0
votes
2answers
1k views

How to use wget to grab copy of Google Code site documents?

I have a Google Code project which has a lot of wiki'ed documentation. I would like to create a copy of this documentation for offline browsing. I would like to use wget or a similar utility. I have ...
1
vote
1answer
6k views

Downloading only images using curl or wget?

UPDATED: I've found using this Bash script fixes the problem of having GIF files with a .jpg extension. I'm attempting to download images from a forum who's url uses the following format: ...
1
vote
1answer
617 views

How to use delicious API from command line

I'm trying to add a delicious bookmark from a shell script, but keep getting the mysterious "something went wrong" error. I've checked the official spec and several examples and everything looks ...
0
votes
1answer
375 views

curl (7) permission denied on cgi script

Can't understand which permissions could be denied. Running cgi script from command line works fine. http_proxy="" curl http://url/ -o file Produces curl(7): permission denied (same issues with ...
1
vote
1answer
1k views

Submitting a form using curl

Forgive me for the simple question... I haven't used curl before. I'm trying to automatically download some data from this form: http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500 which ...
1
vote
1answer
439 views

Scrape a site with curl/wget appending a parameter to each site

I need to scrape a website for all sites and append an Paramater, e.g. www.theSite.com/allpages?myPara=us Now what I got is wget --recursive to get all sites, but that doesn't append the parameter ...
1
vote
1answer
993 views

curl can't connect; wget can

trying to connect to my script on a remote server with curl but: col-pos-spd# curl -v http://***:80/monitor/reply.php * About to connect() to *** port 80 (#0) * Trying 83.xxx.xxx.2... Connection ...
0
votes
1answer
549 views

Using wget on a site with cgi

I need to mirror a website. However, each of the links on the site's webpage is actually a 'submit' to a cgi script that shows up the resulting page. AFAIK wget should fail on this since it needs ...
2
votes
1answer
930 views

Download a file from a website in command line in Linux

I need to download a file from a website which has a URL formatted like: http://www.server.com/downloads/1234/ This redirects to a .zip file which has to be saved. There is also a need to ...
3
votes
3answers
2k views

HTTP responses curl and wget different results

To check HTTP response header for a set of urls I send with curl the following request headers foreach ( $urls as $url ) { // Setup headers - I used the same headers from Firefox version 2.0.0.6 ...
21
votes
3answers
20k views

How to wget a file with correct name when redirected?

So after some time of searching on google and superuser (and scanning man pages) I was unable to find an answer to something that (I think) should be simple: If you go here: ...
11
votes
8answers
25k views

wget/cURL alternative native to Windows?

Is there a wget or cURL type command line utility native to Windows Vista? How does it work?
0
votes
2answers
348 views

remote backup with curl/wget

I need to backup a clients server (source) from our mediatemple server (destination) and then add it into a git repository while ssh'd in. What is the most efficient/fastest way to do so? wget? or ...
1
vote
1answer
795 views

Load multiple files simulatenously with cURL

I am looking to download multiple files from a server using cURL from the Terminal on a Mac. I know that using the {} operator, I can load one file after another but I am looking to download all (10) ...
1
vote
1answer
2k views

How do I setup curl / wget with FTP proxy?

I'm having a good deal of trouble getting curl or wget to download files from FTP servers through an FTP proxy. I have setup GUI FTP programs to work... gFTP works with these settings: I get ...
0
votes
2answers
3k views

How to download a file behind authentication via wget or curl [duplicate]

Possible Duplicate: WGet a Page that Requires Logging in I wish to download file from http://mbr.nlm.nih.gov/Download/MetaMapped_Medline/2011/ however in order to download the file, i need ...
0
votes
3answers
2k views

Wget hangs in the middle of get request

Sometimes wget or curl randomly hangs in the middle of get request in Ubuntu 10.10: curl "http://api.indeed.com/ads/apisearch?publisher=7570038743238473" wget ...
3
votes
3answers
696 views

What's the equivalent of “wget -i FILE” in Curl?

What's the equivalent of "wget -i FILE" in Curl?
4
votes
3answers
2k views

Downloading XCode using wget or curl [closed]

Is there a way for downloading XCode using curl or wget? EDIT: Why is the following curl snippet not working? First, I login to Apple's site: curl -d "theAccountName=USERNAME&theAccountPW=PASS" ...
0
votes
2answers
338 views

How to automatically get images from an SSL Site that uses Java Servlets

This is tricky. I want to use curl, wget or any other tool to login into a SSL Website which provides a Login Form. Then I want to visit several Links within that Domain and fetch certain images. I ...
0
votes
3answers
3k views

unauthenticated http proxy, wget fails, curl works

I'm running a ubuntu 10.04 i386 desktop edition. I have configured the http proxy via the http_proxy variable. When I use wget I get a connection refused but, curl, apt-get etc. work just fine. Any ...
3
votes
2answers
2k views

Persistent retrying resuming downloads with curl

I'm on a mac and have a list of files I would like to download from an ftp server. The connection is a bit buggy so I want it to retry and resume if connection is dropped. I know I can do this with ...