GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

learn more… | top users | synonyms

1
vote
1answer
442 views

Download directory & subdirectories via wget

I have a folder with folder view on the web (http://example.com/folder1/folder2/) /folder2 has multiple folders with pdf files within them. I want download to my server via ssh all the content of /...
2
votes
2answers
328 views

Wget download 403 Forbidden or 503 service unavaible

First of all I need to say that there are a lot of question about this error but I almost tried all but still I have. url="https://cdn59.my.mail.ru/hv/55045204.mp4?sign=...
0
votes
1answer
204 views

How to download list of files and rename them incrementally using wget?

I have a list which contains imgur links, I want to download the links from it and auto rename them with wget. For example, these are the links in the list: http://i.imgur.com/YSLB7eh.jpg http://i....
2
votes
0answers
285 views

403 when downloading from cloudfront with wget (after 302)

I'm trying to download file from cloudfront. It is a file archive from scala-sbt website. When using browser (Chrome) everything is ok. Basically the flow looks like: https://dl.bintray.com/sbt/...
0
votes
0answers
104 views

Not able to download all files from nexus to git repository folder with wget

I am downloading files from the nexus repo using wget command. Using the command, I am able to download all the files in the C:/ area, but when I am using the same command inside C:/users/Myname/...
0
votes
0answers
361 views

wget through proxy in virtual box (cloudera quickstart VM - Cent OS)

I have installed the Cloudera Quickstart VM 5.3.0 running on virtual box for windows 7 and I would like to run a wget command but I have troubles with my company proxy (which needs authentification). ...
0
votes
1answer
32 views

apple store showing different page in wget

istore.apple.com live url is different that url which is downloaded using wget Open this url in browser https://itunes.apple.com/us/genre/ios-games-action/id7001?mt=8&letter=A You will see ...
1
vote
0answers
104 views

Make static site from wordpress site using bash

I have tried a few other scripts that can make a static version of a wordpress site but none of them are designed for bulk sites and I haven't found any that will just serve a static version while ...
2
votes
2answers
139 views

Fast way to download many small webpages?

I need to download lots of small pages as fast as possible. Average size for one page is 3kb. There are millions of pages. First, I tried curl but it was way too slow, probably because it was waiting ...
2
votes
0answers
405 views

How do I download an entire folder with aria2?

Aria2 is a great wget replacement, having a lot of features like multi-threading, split download,download resume, etc But having all this features and option can make it hard to do some specific ...
0
votes
0answers
29 views

wget a whole website, no sound

My friends showed me a site and I want to download this site for offline use, in basic .html format Using the command: wget -r -k -p "www.flappyplane.net" I was able to obtain the site for offline ...
1
vote
1answer
1k views

I can ping but cannot download from the internet

When using the system proxy, firefox has no access to internet. After changing the proxy to auto, firefox downloads fine. In the terminal, when I tried to download using wget, the ternimal stopped at ...
4
votes
1answer
173 views

Wget - Saving a file to a local directory named after link

I'm new to using wget, so I hope this is not a dumb question : How can I save a file to a local directory that has been named after the link to the file ? It does not seem to work if a redirection ...
0
votes
1answer
101 views

Using Wget to download files with a specific name off a site

What would be the best way to look down through a (semi) top level domain and all it's subfolders for occurrences of certain number letter combinations. I have several folders with large lists of ...
1
vote
1answer
124 views

Wget having trouble just saving the files I want - exclude directories doesn't seem to work

I want to download all government spending over £500 by the Department of Energy and Climate change. These are .xls and .xlsx files, generated once per month. They are stored at locations like this: ...
0
votes
0answers
218 views

Why do wget and curl fail to download page source?

Via a shell script, I would like to download the source for a web page with air quality data on aqicn.org. Here is the URL for the page with Beijing air quality data: http://aqicn.org/city/beijing/m/ ...
1
vote
1answer
134 views

How to read Telegraph's rss feed?

According to Telegraph.co.uk, the world news rss feeds should be this one: http://www.telegraph.co.uk/news/worldnews/rss However, with Chrome I only get a blank page. Even wget http://www.telegraph....
0
votes
1answer
106 views

Downloading images via wget

When downloading an image with wget using wget link I get the following errors: HTTP request sent, awaiting response... 302 Moved Temporarily Location: [following] --2015-05-24 14:47:46-- http://...
-1
votes
2answers
124 views

How to install exe versions of linux utilities on windows? [duplicate]

I am aware that we can get .exe versions of many linux utilities. I happen to have wget.exe, curl.exe and busybox.exe. But how can I install them so that instead of typing C:\Users\Rahul>cd ...
0
votes
0answers
67 views

wget converting http but not https links for offline use

I am trying to download a website for offline use using the following wget command on an Ubuntu machine: wget --recursive --background --adjust-extension --continue --page-requisites --convert-links -...
0
votes
1answer
177 views

Wget - repeat download of the same image from URL (with backup and timestamping)

I am trying to download images from a webcam that refreshes the image about every 3 seconds. I want to use the timestamping option in case their webcam stops updating the files (happens from time to ...
1
vote
1answer
337 views

fix wget “invalid or incomplete multibyte or wide character”

Wget show me Can't write to ... (invalid or incomplete multibyte or wide character) when i try to dump a website with Persian or Arabic characters in urls. Have wget any flag to fix this characters ...
0
votes
1answer
35 views

Craft HTTP GET on Mac to get file

So I'm trying to download a file on my mac but the requirements to get the file are very strict: Username: testx Password: testing123 Authorization You can craft the ...
0
votes
0answers
134 views

WGET convert links not working with URLs that start with slash (relative URL)

Knowing SuperUser family, I am attempting to download mirror a site with the following command: wget --recursive --background --adjust-extension --continue --quiet --page-requisites --convert-links -...
0
votes
0answers
30 views

How to make wget resolve HTTP queries? [duplicate]

Consider the download link http://trac-hacks.org/changeset/latest/accountmanagerplugin/tags/acct_mgr-0.4.4?old_path=/&filename=accountmanagerplugin/tags/acct_mgr-0.4.4&format=zip with the ...
0
votes
1answer
825 views

Modifying wget permissions and download directories on Cygwin

I have Cygwin installed on my Windows 7 64 bit PC and I often download large files using wget. wget, however, saves all files with permissions that forbid me from executing the files when they're ...
3
votes
1answer
350 views

wget starting from arbitrary byte

Using the -c flag, I can tell wget to resume a download. wget will then query the size of the pre-existing file to tell the server from where to resume the download. Is there a way to tell wget to ...
0
votes
2answers
435 views

Why wget fails with 'https: Host not found' on Windows?

I'm trying to use wget on Windows 8.1 prompt. I installed wget.exe in the folder System32 and run wget on cmd. But the following error appear: M:\site>wget http://wordpress.org/latest.zip --15:44:...
1
vote
0answers
202 views

How to download files from Website using bash [closed]

I am trying to download a collection of files from a website. Where the collection is drawn from "TAGS". I have tried using many combinations of wget and bash scripts with no luck. Each file is nested ...
0
votes
1answer
147 views

Wget not getting cookies, not pulling back results in check.

I'm having a couple of issues when porting check scripts between a couple of servers. The script. #!/bin/bash hostname=$1 port=$2 retstatus=0 retStr= st=`date "+%s.%N"` if [ $# -ne 2 ]; then ...
2
votes
0answers
46 views

wget - limit following to specific links

I want to do mirror of website but limit following only to specific links. Command: wget --recursive --page-requisites --html-extension --convert-links --domains linuxmagazine.pl http://www....
1
vote
0answers
35 views

wget only new or updated files when the source files are compressed but the destination files are uncompressed

I would like to know how to wget files from a url which are stored with bz2 compression. However, I would like to only wget the new and updated files, relative to my destination, where the files are ...
1
vote
0answers
157 views

FTP: wildcard downloading

I am trying to download a file with name ending with *.csv The problem is complicated by the fact that this *.csv contains in directory of unknown name: eg. Sometimes it would be 2015->Date 29:10:...
0
votes
0answers
46 views

Wget which stops when it has a download limit

I have over 200 wget in screens on my server. I set a download limit in each low wget: --limit-rate=1K. The only problem is that after one hundred seconds, he considers as a finished one, so this ...
1
vote
1answer
39 views

Advanced Wget usage

I have a list of about 200 servers which contain links to individual files, for this question, let's just pretend they're .txt files. I need to download every file, however some servers only have the ...
0
votes
3answers
3k views

Problems with Wget to a CloudFlare hosted site: 503 Service Unavailable

I have seen other instances of 503 errors using Wget, but to no available I cannot solve this. When I try to download a certain website, I get a 503 Service Unavailable error. This does not happen to ...
0
votes
1answer
3k views

bash- how can I use wget to send POST data with files?

I want to make following POST request to my server using wget: email=abc@abc.com&file1=@FILE_HERE&file2=@FILE_HERE In above request, there are three POST parameteres called email , file1 ...
0
votes
1answer
30 views

Keyboard shortcut to kill a script containing multiple wgets?

I have a Python script that wgets a load of files: for year in range(2010, 2015): for month in range(1, 13): filename = url filename += "/T%s%s_BNFT.CSV" % (year, month) ...
8
votes
3answers
27k views

Why do images from some Tumblr pages not load, but using wget on them works?

Helping a friend out with their Internet connection because “some pages won’t load”, I noticed that the problem was that the images of certain blogs' image posts weren’t loading on the browser. I ...
0
votes
1answer
49 views

integrate wget with opera 27

I want to integrate wget into opera, I didn't find anything on the web except this article http://archive09.linux.com/feature/59457 which is old(back to 2007) and I didn't find tools->preferences ...
2
votes
1answer
170 views

Bulk SFTP download of files specified by URL preserving directories

I have a list of several hundred files I need to pull via SFTP. They are located in various directories and I need them to be in the same directory setup on my local machine. It is important the files ...
0
votes
0answers
135 views

wget, how do i download an entire site with login details

having a problem downloading a website called http://connect.cimaglobal.com for accounting students, i have to login a i am a student but every time i try download the site i'm stuck at the log in ...
0
votes
1answer
41 views

How can I wget if a file has changed?

I have a cronjob that downloads a PDF file from a remote server. The file name is always static but I save it into a directory with the date. The file gets saved as the original save time from the ...
2
votes
0answers
143 views

How to curl or wget live tennis scores from a Google search page [closed]

Google has some nice live tennis scores displayed on the search page. If I search for "Halep", for example, I get the tournament she's currently playing, together with all the result and live score. I'...
0
votes
0answers
21 views

How to check what kind of files web page consists

Is there a way to check what files web page consists. For example I want to download .txt files from the web page, but I dont want all .txt files, some of them, and before I want to check are there ...
0
votes
1answer
230 views

Is there a way to download all .txt files from the page via curl or wget command

My question is in a title: Is there a way to download all .txt files from the page via curl or wget command? Thanks
0
votes
0answers
43 views

Listing the Files in a Web Page Directory via Wget

What should I write in the command line to list the files in this webpage: "http://download.xbox.com/content/5454085c/"? Files are supposed to be in ".cab" format.
1
vote
1answer
272 views

How to make wget create appropriate zero-byte files?

I'm trying to use wget to download entire directories with their contents from a webserver. Some of the files there have zero length. Is there a way to make wget create the 0-byte file on the client? ...
0
votes
0answers
803 views

Why is cURL slower than wget

Test page: https://www.beobank.be/nl/Home.aspx This is the result using wget (or a real browser): # time wget https://www.beobank.be/nl/Home.aspx -O /dev/null --2015-01-26 12:05:46-- https://www....
2
votes
1answer
3k views

Can ping but can't connect (curl, wget, apt-get, etc.) (rasberry pi server)

Background: I am trying to set up a rasberry pi server. Everything was going well and my pi could apt-get update etc, until I got the port forwarding to work and ssh to work outside of the LAN. I ...