GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

learn more… | top users | synonyms

179
votes
15answers
394k views

How to download files from command line in Windows, like Wget is doing?

How can I download something from the web directly without Internet Explorer or Firefox opening Acrobat Reader/Quicktime/MS Word/whatever? I'm using Windows, so a Windows version of Wget would ...
26
votes
8answers
60k views

Wget/cURL alternative native to Windows?

Is there a Wget or cURL type command line utility native to Windows Vista? How does it work?
52
votes
4answers
66k views

Save a single web page (with background images) with Wget

I want to use Wget to save single web pages (not recursively, not whole sites) for reference. Much like Firefox's "Web Page, complete". My first problem is: I can't get Wget to save background images ...
9
votes
3answers
24k views

How to download list of files from a file server?

How would I download a list of files from a file server like this one http://www.apache.org/dist/httpd/binaries/ ? I suppose I could use wget but then it tries to get all the links and the html file ...
6
votes
5answers
16k views

How do you use WGET to mirror a site 1 level deep, recovering JS, CSS resources including CSS images?

Pretend I wanted a simple page copy to be downloaded to my HD for permanent keeping. I'm not looking for a deep recursive get, just a single page, but also any resources loaded by that page to be also ...
12
votes
5answers
16k views

How can I make wget rename downloaded files to not include the query string?

I'm downloading a site with wget and a lot of the links have queries attached to them, so when I do this: wget -nv -c -r -H -A mp3 -nd http://url.to.old.podcasts.com/ I end up with a lot of files ...
7
votes
10answers
22k views

Where on my computer did Wget download this image?

I am using Windows XP. I just downloaded wget and put it into my C:\Windows folder. I then ran this command line (trying to download an image from a website): wget -r -A ".jpg,.gif,.png" http://...
13
votes
1answer
16k views

Make wget convert HTML links to relative after download if -k wasn't specified

The -k option (or --convert-link) will convert links in your web pages to relative after the download finishes, such as the man page says: After the download is complete, convert the links in ...
1
vote
3answers
544 views

Wget a page that requires logging in

I’m trying to figure out a way to use Wget or a similar tool so that I can schedule a web page to be downloaded regularly as a sort of updating log. The problem is that the page requires that I be ...
45
votes
4answers
44k views

How to wget a file with correct name when redirected?

So after some time of searching on Google and Super User (and scanning man pages) I was unable to find an answer to something that (I think) should be simple: If you go here: http://www.vim.org/...
4
votes
3answers
35k views

Download ALL Folders, SubFolders, and Files using Wget

I have been using Wget, and I have run across an issue. I have a site,that has several folders and subfolders within the site. I need to download all of the contents within each folder and subfolder. ...
9
votes
3answers
27k views

How to set http proxy address for wget under windows?

If ran without parameters my wget prints: D:\>wget SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc syswgetrc = c:/progra~1/wget/etc/wgetrc D:\Apps\Util\wget: missing URL Usage: D:\Apps\Util\wget [...
7
votes
2answers
6k views

Persistent retrying resuming downloads with curl

I'm on a mac and have a list of files I would like to download from an ftp server. The connection is a bit buggy so I want it to retry and resume if connection is dropped. I know I can do this with ...
0
votes
2answers
5k views

How to download a file behind authentication via Wget or cURL [duplicate]

Possible Duplicate: Wget a page that requires logging in I wish to download file from http://mbr.nlm.nih.gov/Download/MetaMapped_Medline/2011/. However, in order to download the file, I need ...
4
votes
3answers
8k views

How to force redownload with wget?

I had the bad surprise that wget doesn't redownload when a file of the same name already exist. Is there an option to force him to redownload without deleting the file first on linux ?
2
votes
2answers
7k views

crontab - /bin/sh: wget: command not found

I want to set up a crontab on my Macbook Pro running Lion. I run the usual crontab -e and provide a quick job to see if everything is working ok --- */10 * * * * wget -O - -q -t 1 http://site.local/...
31
votes
6answers
50k views

how to download dropbox files using wget command?

Seems I can only download dropbox files using explorer such as chrome and firefox. If I use wget to download, then I would get a file which is in html format. Why? For example you can open this link ...
10
votes
4answers
31k views

Using Wget to Recursively Crawl a Site and Download Images

How do you instruct wget to recursively crawl a website and only download certain types of images? I tried using this to crawl a site and only download Jpeg images: wget --no-parent --wait=10 --...
9
votes
5answers
6k views

Trouble using wget or httrack to mirror archived website

I am trying to use wget to create a local mirror of a website. But I am finding that I am not getting all the linking pages. Here is the website http://web.archive.org/web/20110722080716/http://cst-...
4
votes
4answers
11k views

How can I download an entire (active) phpbb forum?

One of the forums that I frequent (and have added a LOT of quality content too) seems to be having problems with their server. I am not confident in their ability to sort out the problems they are ...
9
votes
2answers
2k views

How to download parts of the same file from different sources with curl/wget?

I have a quite large file hosted on five different servers. I would like to be able to download different parts of the file from each server and subsequently concatenate the parts, in order to ...
22
votes
1answer
3k views

Why does wget give an error when executed with sudo, but works fine without?

I tried the following command: $ wget -q --tries=10 --timeout=20 --spider http://google.com (From this SO post. I want to check my internet connection in bash.) I get following output: Spider ...
5
votes
1answer
3k views

Downloading multiple files, and specifying output filenames with wget

I'm using wget with the -i option to download a list of files from a URL. However, I want to specify the names that these files will be saved with as well. I see you can do that with a single file ...
3
votes
7answers
11k views

How to recursively download an entire web directory?

i have a web directory that has many folders and many sub folders containing files. i need to download everything using wget or bash.
7
votes
2answers
3k views

Call but not download a page with wget

I was wondering if there was any argument that allowed me to use wget and "call" a page, but without downloading the page. The problem is that when you call wget on a page, it downloads it to the ...
7
votes
4answers
3k views

Wget is silent, but it displays error messages

I want to download a file with Wget, but per the usual UNIX philosophy, I don't want it to output anything if the download succeeds. However, if the download fails, I want an error message. The -q ...
6
votes
3answers
3k views

Make wget download page resources on a different domain

How do you use wget to download an entire site (domain A) when its resources are on another domain, (domain B)? I've tried: wget -r --level=inf -p -k -E --domains=domainA,domainB http://www.domainA
5
votes
5answers
2k views

Why does wget'ing an image give me a file, not an image?

I do wget http://services.runescape.com/m=itemdb_rs/3809_obj_sprite.gif?id=2 and it downloads the following file: 3809_obj_sprite.gif?id=2 How can I make it save as the following? 2.png
5
votes
1answer
5k views

Exclude list of specific files in wget

I am trying to download a lot of pages from a website on dial-up and it can be brutally slow. I have almost got the perfect wget command, but because I'm downloading pages from the same site wget ...
2
votes
1answer
4k views

wget - only getting .listing file in every sub-dir

if I use the command "wget --no-remove-listing -P ...../debugdir/gnu// ftp:/gnu//" I will get the .listing file of that directory. But I have to step through each subsequent sub-directories to get the ...
2
votes
2answers
9k views

How can I scrape specific data from a website

I'm trying to scrape data from a website for research. The urls are nicely organized in an example.com/x format, with x as an ascending number and all of the pages are structured in the same way. I ...
0
votes
1answer
1k views

How to download images using wget from a txt file that contains links

I can download images using wget if I download from a website. But I have several links and I have saved them in a text file. For example: wget -r -A.jpg -np www.fragrancenet.com There are so many ...
5
votes
4answers
3k views

wget not converting links

I am trying to mirror a fairly large site (20,000+ pages) prior to a major overhaul. Basically, I need a backup before cutting over to the new one in case we forgot something we need (we'll have ...
4
votes
1answer
6k views

What units does wget use for bandwidth?

When downloading, wget reports speed in "K/s". K...what? kilobits? kilobytes? 1024 or 1000? Update: wget -O /dev/null http://newark1.linode.com/100MB-newark.bin produces "348 K/s". Meanwhile:...
3
votes
4answers
12k views

Copy entire website including PHP and SQL?

Is it possible when copying an entire website and template using wget (httrack or other program) to also get the PHP code and the SQL database? What software would I need to copy/mirror a site like ...
2
votes
1answer
956 views

wget throws a 'missing url' when trying to install a repository signature key

I am installing INetSim on Ubuntu and when running this command to install the signature key for the program: sudo wget -O - http://www.inetsim.org/inetsim.org-archive-signing-key.asc | apt-key add - ...
1
vote
1answer
248 views

Wget - Difficulty Excluding Domains and Directories

I am using Wget in Windows 7. I believe this restricts me to Wget 1.11.4. For anyone interested in the details, I have posted a sample wgetrc file. This sample is actually the one I am currently ...
1
vote
2answers
2k views

curl can't connect; wget can

trying to connect to my script on a remote server with curl but: col-pos-spd# curl -v http://***:80/monitor/reply.php * About to connect() to *** port 80 (#0) * Trying 83.xxx.xxx.2... Connection ...
0
votes
1answer
552 views

using wget to download from website with case-sensitive URLs to Windows

I used Gnuwin32 wget to download emacs manuals with this command (takes about 30 minutes): wget --mirror --page-requisites --convert-links --no-parent --accept .html,.htm,.css,.js http://www.gnu.org/...
0
votes
1answer
6k views

How to mirror ENTIRELY a website with wget [duplicate]

I really need the command to download everything from a website. It is arnos.gr and has a lot of content on it which I want to browse locally without Internet I have an account for login
0
votes
1answer
1k views

wget - save all data (images) from given directory?

I want wget to scan for sub and sub-subdirectories of: domain.com/profile/username/albums/ Then grab every .jpg file from their sources. Wget should get files like: domain.com/profile/...
0
votes
1answer
2k views

Downloading multiple files with wget and handling parameters

How can I download multiple files using wget? I also want to rename the files. Here are the commands I'm running one by one (copy/paste on terminal): wget -c --load-cookies cookies.txt http://www....