GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

learn more… | top users | synonyms

0
votes
0answers
79 views

Gmail Automation via Bash (To refresh POP Fetch from other accounts)

Gmail has this option to fetch mails from other accounts. I used this feature to create a central mail box. Works good. Gmail periodically polls other mail box over POP. But the frequency is 1 hour + ...
0
votes
1answer
52 views

How to download streamed files with wget?

Sometimes I stumble in files I cannot download with wget. So maybe someone has a hint for me? For example I want to download via the following link: ...
1
vote
0answers
164 views

Wget slow download speed

I get horribly bad download speeds when I try to download files to Banana Pro (similar to Raspberry Pi) with wget. With wget, I get around 1.5-2MByte/s, while on another computer, I'm getting about ...
0
votes
0answers
77 views

WGET not working but CURL works on the same address/host?

WGET give me this error: HTTP request sent, awaiting response... 403 Forbidden 2015-11-23 12:07:06 ERROR 403: Forbidden. If I try the same URL (it’s an internal network Apache server, firewall ...
0
votes
1answer
143 views

How to download all mp3 files linked in a website with wget?

The links in the page 'http://www.examplepage.com/' are to a different page 'http://www.archive.org/download/examplepage/asdf.mp3'. How would I write the wget command? wget -mH -A mp3 ...
0
votes
0answers
21 views

Recursive wget strip GET parameters

I am downloading a website I have using wget -r http://example.com but in order for the css files to not be cached, the website adds GET parameters to those files such as styles.css?foobar. How can I ...
0
votes
1answer
26 views

How to ignore parent directories with wget?

I am trying to scrape a particular online directory of its .flac files. I am typing the following command wget -r -a flac,FLAC http://website.com/directory Basically this give me all the files I ...
0
votes
0answers
39 views

How to save a copy of this web page?

I'd like to save a copy of this page offline so that I can grep through it: http://forum.devmaster.net/t/advanced-rasterization/6145 But I'm unable to download a copy of this page with wget, saving ...
1
vote
2answers
65 views

Downloading file using WGET when run under CMD file

I'm trying to download a photo from the internet but an error always occur. I tried to run the command under a batch program ".cmd" with the following codes in it: wget ...
1
vote
1answer
115 views

wget --continue does not work with --no-clobber

I am working under Windows 7 and I want to download all new files from a Windows server directory over https using wget. In addidtion, I want to resume the download of large files in case of ...
1
vote
1answer
75 views

wget parsing tag script

I'm trying to backup a website that contains pages with a tag like this one: <a id="12379-video"></a> </div> <script> jwplayer("12379-video").setup({ file: ...
1
vote
0answers
76 views

How to get uncompressed content when using recursive wget?

i am downloading many single pages with all static content (js, css, imgs...) via wget recursive. It showed up, that served content, which was compressed (gzip), is stored by wget in compressed form. ...
0
votes
0answers
20 views

wget detect and alter downloaded resource URL

I'm using wget right now to fetch information image resources from a website. My current script is: wget -e robots=off --wait 0.1 -E -H -k -K -p http://summonerswar.wikia.com/wiki/Fire_Monsters On ...
1
vote
0answers
48 views

Wget -nv doesn't print any error messages

I'm trying to write a script to set up a cron job to automatically download a file, using wget. Here's the important part: $ wget -N --passive-ftp -P /home/myuser ...
1
vote
0answers
75 views

Get filename from a wget --content-disposition

Inside of a bash script I am dynamically downloading a file and so I use wget --content-disposition to ensure the file names are correct, but how would I go about retrieving the name of the file wget ...
1
vote
2answers
142 views

Powershell equivalent for wget switches “-nc” and “-i” [duplicate]

What is the Powershell equivalent of this wget command? wget -nc -i downloadList.txt Where -i downloadList.txt downloads a list of urls' in the specified file. -nc skips the ...
1
vote
3answers
75 views

Renaming files fetched via “wget --mirror” in Bash before uploading to an Amazon S3 statically hosted area

I’m trying to archive and upload an old website to statically hosted Amazon S3 area. I was able to get the contents with wget, using the following command: wget --mirror --no-parent --html-extension ...
1
vote
1answer
111 views

Transfer huge files to new server through FTP

I have a dedicated server in Germany with 120GB data, since I have bought a new dedicated server in U.S, I would like to transfer all the files to new server through FTP connection, to achieve this ...
22
votes
1answer
2k views

Why does wget give an error when executed with sudo, but works fine without?

I tried the following command: $ wget -q --tries=10 --timeout=20 --spider http://google.com (From this SO post. I want to check my internet connection in bash.) I get following output: Spider ...
0
votes
2answers
39 views

Tell wget to mirror the parts of an URI's path to folders

I have a list of URIs in a file uris.txt: http://example.com/file1.pdf http://example.com/file2.pdf http://example.com/Folder/file3.pdf http://example.com/Folder/Subfolder/file4.pdf The ...
0
votes
0answers
11 views

ftp or rsync equivalent on windows 7 to acquire entire directory [duplicate]

I am not a Windows User, I am much more comfortable in Linux. My institution blocks ftp and rsync from my local Apple machine, and forces me to log on to a Windows 7 Virtual Machine. I want to ...
0
votes
0answers
172 views

wget or cURL for dynamic recursive URLs (generated through directories and adding specific same text at the end)

I'm using a wget command with some options in order to refresh the cache module from all URLs of a Prestashop. This is the command: wget -r -D www.domain.com -l3 -P temp -x -R "*cart*","*order*","*My ...
2
votes
1answer
63 views

How to access internet on my firefox through ssh remote?

Alright, I'm completely blank into this weird world. Let's say, I can access a ssh remote server(A server which has no GUI and no monitor but have internet connection) using terminal in ubuntu(Which ...
1
vote
1answer
69 views

How can I run the equivalent “curl” command that performs similarly to this “wget” command?

I have a wget command: wget -r --no-parent -nH -nd -q --show-progress --progress=bar:force ftp://ftp.ncdc.noaa.gov/pub/data/swdi/database-csv/v2/ But it only works in newer versions of wget. ...
1
vote
0answers
97 views

Can ping URL but cannot access via cURL or wget

I am experiencing an issue where I can ping, but I cannot use wget or curl. This is the output from these commands. I have copied a working Vagrant VM from one Win7 PC to another and this is the ...
4
votes
1answer
233 views

Using wget to copy website with proper layout for offline browsing

This is the proper way to download a website with all the images and css files so that it has the same layout as the original but I don't know why the -K --backup-converted and -E --adjust-extension ...
0
votes
1answer
696 views

Cannot write to `-' (Broken pipe) when trying to install a repository signature key

I’m trying to install INetSim on Ubunutu 14.04 desktop. I edited the sources list with the pre compiled Debian Packages. Then when trying to install the signature key using wget I run across this ...
2
votes
1answer
627 views

wget throws a 'missing url' when trying to install a repository signature key

I am installing INetSim on Ubuntu and when running this command to install the signature key for the program: sudo wget -O - http://www.inetsim.org/inetsim.org-archive-signing-key.asc | apt-key add - ...
0
votes
1answer
506 views

How can I download files from shared Amazon Cloud Drive links through the command-line?

I have been using Amazon Cloud Drive (ACD) for a while now and I am very happy someone has developed a command-line-client for it (called acd_cli). It allows me to keep my files in "sync" with my ...
1
vote
2answers
77 views

Download a list of files and use domain name as filename

I got a bunch of files to download which works great with wget -i list_of_urls but my problem is, that wget uses the filename of the downloaded file. Is there a way (or a different tool) which is ...
0
votes
1answer
58 views

How to extract ip address from no-ip account webpage in the shell?

I've set dynamic dns for my home network(which is behind NAT) using www.noip.com. I use ddclient to send updates of my ip, but sometimes it takes a while to update the ip. Anyways it's not the point. ...
0
votes
0answers
167 views

Making wget follow “javascript links”

I'd like to mirror a webpage recursively, but it uses javascript links, such as "javascript:readArticle(x)". I know that such links are simply translated into an URL, say "/articles/read?id=x", can I ...
1
vote
1answer
194 views

Is there a simple way to install a list of packages in Cygwin? [duplicate]

Is there a simple way to install a list of packages in Cygwin? I’ve tried the command line arguments with no success. For example I’ve tried the following with no success in uninstalling wget. ...
2
votes
2answers
346 views

Should I be concerned about hacking attempts via wget on a CentOS/LAMP-based web server?

This is a bit confusing, by the way I’m not a system administrator and only know a little bit about handling a Linux. I’m running a LAMP-based website and hosting it on Digital Ocean. The server is ...
2
votes
0answers
327 views

Windows internet speed test

I am trying to check the internet speed from my windows 8 laptop over a period of say 24 hours. So I would like to repeatedly run a command to test the speed. Various websites (e.g. ...
0
votes
1answer
143 views

What is the curl equivalent of wget -qO (wget -q -O)?

I need to run the following command in Red Hat Linux: wget -q -O http://somepage/somefile.cgi| bash But I don't have wget installed. I could install it, but I want to know how to do this with curl. ...
1
vote
0answers
239 views

Complete site rip of a website using wget

I am attempting to perform a complete site rip of a website, I know where the streamed videos locations are however getting the process to run as efficiently and productively is proving difficult. ...
0
votes
1answer
68 views

wget's downloading progress in reverse order?

i have no idea what happened, but suddenly it started showing progress bar in the reverse order... HTTP request sent, awaiting response... 200 OK Length: 4102725632 (3.8G) ...
1
vote
0answers
29 views

can wget limit number of retrieved pages / links in recursive / spider mode?

i use wget for either: retrieving some sites recursively, spider (retrieve links from websites) Sometimes i'd like to get only a certain arbitrary maximum of pages / links. The only limits I've ...
1
vote
1answer
402 views

Download directory & subdirectories via wget

I have a folder with folder view on the web (http://example.com/folder1/folder2/) /folder2 has multiple folders with pdf files within them. I want download to my server via ssh all the content of ...
2
votes
2answers
301 views

Wget download 403 Forbidden or 503 service unavaible

First of all I need to say that there are a lot of question about this error but I almost tried all but still I have. ...
0
votes
1answer
180 views

How to download list of files and rename them incrementally using wget?

I have a list which contains imgur links, I want to download the links from it and auto rename them with wget. For example, these are the links in the list: http://i.imgur.com/YSLB7eh.jpg ...
2
votes
0answers
262 views

403 when downloading from cloudfront with wget (after 302)

I'm trying to download file from cloudfront. It is a file archive from scala-sbt website. When using browser (Chrome) everything is ok. Basically the flow looks like: ...
0
votes
0answers
98 views

Not able to download all files from nexus to git repository folder with wget

I am downloading files from the nexus repo using wget command. Using the command, I am able to download all the files in the C:/ area, but when I am using the same command inside ...
0
votes
0answers
338 views

wget through proxy in virtual box (cloudera quickstart VM - Cent OS)

I have installed the Cloudera Quickstart VM 5.3.0 running on virtual box for windows 7 and I would like to run a wget command but I have troubles with my company proxy (which needs authentification). ...
0
votes
1answer
31 views

apple store showing different page in wget

istore.apple.com live url is different that url which is downloaded using wget Open this url in browser https://itunes.apple.com/us/genre/ios-games-action/id7001?mt=8&letter=A You will see ...
1
vote
0answers
89 views

Make static site from wordpress site using bash

I have tried a few other scripts that can make a static version of a wordpress site but none of them are designed for bulk sites and I haven't found any that will just serve a static version while ...
2
votes
2answers
130 views

Fast way to download many small webpages?

I need to download lots of small pages as fast as possible. Average size for one page is 3kb. There are millions of pages. First, I tried curl but it was way too slow, probably because it was waiting ...
2
votes
0answers
350 views

How do I download an entire folder with aria2?

Aria2 is a great wget replacement, having a lot of features like multi-threading, split download,download resume, etc But having all this features and option can make it hard to do some specific ...
0
votes
0answers
29 views

wget a whole website, no sound

My friends showed me a site and I want to download this site for offline use, in basic .html format Using the command: wget -r -k -p "www.flappyplane.net" I was able to obtain the site for offline ...