GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

learn more… | top users | synonyms

0
votes
0answers
50 views

Clone a GET request-based website

I'm trying to make a static clone of a website because it's running on some ancient CMS that's going to crash any minute now. I've tried using httrack and wget, but none of them work. The problem ...
0
votes
0answers
71 views

Downloading video using command line

I use right click on a button and select Download Linked File As to download a video file into my computer. I'd like to automatize it with scripts, so I opened the source code and copy the link ...
0
votes
0answers
28 views

Command to fetch a list of urls with connection keep-alive but fail on the first error (404/500)

I am trying to reproduce a potential bug in the Jetty web server with regard to serving static files. I need to repeatedly download a list of urls with "Connection: keep-alive" until one of them ...
2
votes
0answers
50 views

msys2 network performance woes

With a fresh install of msys2 32-bit (needed for git-annex) from https://msys2.github.io/ on Windows 7 SP1 64-bit, I'm seeing incredibly slow network transfer speeds on wget, ssh, rsync, etc. For ...
0
votes
3answers
93 views

How to download a file from URL in Linux

Usually one would download a file with a URL ending in the file extension. To download Ubuntu ISO, one would simple wget http://releases.ubuntu.com/14.04.3/ubuntu-14.04.3-desktop-amd64.iso ...
0
votes
0answers
29 views

OSX terminal : log on on a website

I would like to login/ out on a Web site using the terminal, I currently using a appleScript I wrote but was wondering can you use terminal command line instead (not running my script from terminal) ...
0
votes
0answers
70 views

httrack getting tumblr photoposts but not photosets; why?

I'm trying to save tumblr image posts. For a post with one image, like this http://textsfromthepurikyua.tumblr.com/post/124075150094/textsfromthepurikyua-image-cure-heart httrack -n gets that image. (...
0
votes
1answer
44 views

How to wget with pics on www.x.org/yy/index.php

I have an unusual case. I want to use wget to download all pictures on http://www.indyprojectseed.org/2011/index.php and all the other years. How can I do this, with a loop maybe? I am currently ...
0
votes
0answers
126 views

Download a single web page (including images from CSS)

I want to download a single web page including everything needed to view the web page locally (this includes images linked in the css). Example URL i want to download: http://www.example.com/page1 ...
0
votes
0answers
13 views

Indexing file resources from a website

How can I get a list of all (image) resources from a website? Is there a better wait than using wget and parsing the log stream in order to get the list of all images? Worst is, I'm pretty use I ...
0
votes
0answers
36 views

Improper clone of a site with wget

I was trying to clone the homepage of Google with wget as explained here, wget -E -H -k -K -p https://www.google.com/ But I am not able to clone an exact copy of the live Google page. Please refer ...
1
vote
1answer
14 views

How to handle Proxy with Sudo on VM

I have Ubuntu installed on a VirtualBox VM and I am using NAT. The host network requires NTLM proxy authentication to work. I tried export HTTP_PROXY=proxy.mycomp.com:80 & HTTPS_PROXY=$HTTP_PROXY ...
1
vote
2answers
56 views

Wget Bash Script File Not Found

I have the following script: for i in {1..16}; do $wget http://url/$i.jpg -O C:/Users/User/Desktop/Test/ done Saved the file as test.sh and run it within Git on windows. The files are live on the ...
-4
votes
1answer
21 views

How do I take a url and rename it to a .mp4 using wget? [closed]

Need help for a project and have a bunch of URLs that need to be renamed to .mp4 Here are some of them 1280x720.mp4 - 640x360.mp4 - 320x180.3gp
0
votes
0answers
70 views

Slow download over libraries on certain Windows systems

I am using custom libraries for file download (cURL and wget) on Windows client over HTTP from my server. On most of systems the download speed is as it should be - maximum determined by network ...
0
votes
1answer
71 views

Does wget -O always write a zero byte file on failure or can it be a different size?

When using wget's -O flag, the program writes a bunch of zero byte files when it fails. My question is, will the size always be zero? Or, can a non-zero byte file also be created when a download ...
0
votes
0answers
24 views

wget on Windows - same file keeps getting overwritten

I'm using wget on Windows 7. I'm downloading a lot of files such as this: http://stats.grok.se/json/en/200712/Wikipedia:WikiProject%20Video%20games except that I am changing the date each time. The ...
0
votes
1answer
71 views

wget — retrieval of pdf files from online database

I am using wget to retrieve particular pdf files from http://www.aph.gov.au/ I only want to retrieve Hansard files (transcripts of Chamber proceedings). Two scenarios: There is a page where ...
0
votes
1answer
549 views

Bash script not waiting for wget to finish download

I am using the following bash script to merge various gzip archives together: wget -O bluelvl1.gz http://list.iblocklist.com/?list=ydxerpxkpcfqjaybcssw&fileformat=p2p&archiveformat=gz wget -O ...
0
votes
0answers
97 views

Gmail Automation via Bash (To refresh POP Fetch from other accounts)

Gmail has this option to fetch mails from other accounts. I used this feature to create a central mail box. Works good. Gmail periodically polls other mail box over POP. But the frequency is 1 hour + ...
0
votes
1answer
71 views

How to download streamed files with wget?

Sometimes I stumble in files I cannot download with wget. So maybe someone has a hint for me? For example I want to download via the following link: https://exchange.nagios.org/components/com_mtree/...
1
vote
0answers
189 views

Wget slow download speed

I get horribly bad download speeds when I try to download files to Banana Pro (similar to Raspberry Pi) with wget. With wget, I get around 1.5-2MByte/s, while on another computer, I'm getting about ...
0
votes
0answers
108 views

WGET not working but CURL works on the same address/host?

WGET give me this error: HTTP request sent, awaiting response... 403 Forbidden 2015-11-23 12:07:06 ERROR 403: Forbidden. If I try the same URL (it’s an internal network Apache server, firewall isn’...
0
votes
1answer
167 views

How to download all mp3 files linked in a website with wget?

The links in the page 'http://www.examplepage.com/' are to a different page 'http://www.archive.org/download/examplepage/asdf.mp3'. How would I write the wget command? wget -mH -A mp3 http://www....
0
votes
0answers
22 views

Recursive wget strip GET parameters

I am downloading a website I have using wget -r http://example.com but in order for the css files to not be cached, the website adds GET parameters to those files such as styles.css?foobar. How can I ...
0
votes
1answer
33 views

How to ignore parent directories with wget?

I am trying to scrape a particular online directory of its .flac files. I am typing the following command wget -r -a flac,FLAC http://website.com/directory Basically this give me all the files I ...
0
votes
0answers
39 views

How to save a copy of this web page?

I'd like to save a copy of this page offline so that I can grep through it: http://forum.devmaster.net/t/advanced-rasterization/6145 But I'm unable to download a copy of this page with wget, saving ...
1
vote
2answers
78 views

Downloading file using WGET when run under CMD file

I'm trying to download a photo from the internet but an error always occur. I tried to run the command under a batch program ".cmd" with the following codes in it: wget http://weather.is.kochi-u.ac.jp/...
1
vote
1answer
155 views

wget --continue does not work with --no-clobber

I am working under Windows 7 and I want to download all new files from a Windows server directory over https using wget. In addidtion, I want to resume the download of large files in case of ...
1
vote
1answer
89 views

wget parsing tag script

I'm trying to backup a website that contains pages with a tag like this one: <a id="12379-video"></a> </div> <script> jwplayer("12379-video").setup({ file: "...
1
vote
0answers
97 views

How to get uncompressed content when using recursive wget?

i am downloading many single pages with all static content (js, css, imgs...) via wget recursive. It showed up, that served content, which was compressed (gzip), is stored by wget in compressed form. ...
0
votes
0answers
20 views

wget detect and alter downloaded resource URL

I'm using wget right now to fetch information image resources from a website. My current script is: wget -e robots=off --wait 0.1 -E -H -k -K -p http://summonerswar.wikia.com/wiki/Fire_Monsters On ...
1
vote
0answers
55 views

Wget -nv doesn't print any error messages

I'm trying to write a script to set up a cron job to automatically download a file, using wget. Here's the important part: $ wget -N --passive-ftp -P /home/myuser ftp://ftp.example.com/path/to/file....
1
vote
0answers
102 views

Get filename from a wget --content-disposition

Inside of a bash script I am dynamically downloading a file and so I use wget --content-disposition to ensure the file names are correct, but how would I go about retrieving the name of the file wget ...
1
vote
2answers
170 views

Powershell equivalent for wget switches “-nc” and “-i” [duplicate]

What is the Powershell equivalent of this wget command? wget -nc -i downloadList.txt Where -i downloadList.txt downloads a list of urls' in the specified file. -nc skips the ...
1
vote
3answers
86 views

Renaming files fetched via “wget --mirror” in Bash before uploading to an Amazon S3 statically hosted area

I’m trying to archive and upload an old website to statically hosted Amazon S3 area. I was able to get the contents with wget, using the following command: wget --mirror --no-parent --html-extension ...
1
vote
1answer
142 views

Transfer huge files to new server through FTP

I have a dedicated server in Germany with 120GB data, since I have bought a new dedicated server in U.S, I would like to transfer all the files to new server through FTP connection, to achieve this I'...
22
votes
1answer
3k views

Why does wget give an error when executed with sudo, but works fine without?

I tried the following command: $ wget -q --tries=10 --timeout=20 --spider http://google.com (From this SO post. I want to check my internet connection in bash.) I get following output: Spider ...
0
votes
2answers
48 views

Tell wget to mirror the parts of an URI's path to folders

I have a list of URIs in a file uris.txt: http://example.com/file1.pdf http://example.com/file2.pdf http://example.com/Folder/file3.pdf http://example.com/Folder/Subfolder/file4.pdf The ...
0
votes
0answers
11 views

ftp or rsync equivalent on windows 7 to acquire entire directory [duplicate]

I am not a Windows User, I am much more comfortable in Linux. My institution blocks ftp and rsync from my local Apple machine, and forces me to log on to a Windows 7 Virtual Machine. I want to ...
0
votes
0answers
211 views

wget or cURL for dynamic recursive URLs (generated through directories and adding specific same text at the end)

I'm using a wget command with some options in order to refresh the cache module from all URLs of a Prestashop. This is the command: wget -r -D www.domain.com -l3 -P temp -x -R "*cart*","*order*","*My ...
2
votes
1answer
73 views

How to access internet on my firefox through ssh remote?

Alright, I'm completely blank into this weird world. Let's say, I can access a ssh remote server(A server which has no GUI and no monitor but have internet connection) using terminal in ubuntu(Which ...
1
vote
1answer
74 views

How can I run the equivalent “curl” command that performs similarly to this “wget” command?

I have a wget command: wget -r --no-parent -nH -nd -q --show-progress --progress=bar:force ftp://ftp.ncdc.noaa.gov/pub/data/swdi/database-csv/v2/ But it only works in newer versions of wget. ...
1
vote
0answers
108 views

Can ping URL but cannot access via cURL or wget

I am experiencing an issue where I can ping, but I cannot use wget or curl. This is the output from these commands. I have copied a working Vagrant VM from one Win7 PC to another and this is the ...
4
votes
1answer
291 views

Using wget to copy website with proper layout for offline browsing

This is the proper way to download a website with all the images and css files so that it has the same layout as the original but I don't know why the -K --backup-converted and -E --adjust-extension ...
0
votes
1answer
958 views

Cannot write to `-' (Broken pipe) when trying to install a repository signature key

I’m trying to install INetSim on Ubunutu 14.04 desktop. I edited the sources list with the pre compiled Debian Packages. Then when trying to install the signature key using wget I run across this ...
2
votes
1answer
985 views

wget throws a 'missing url' when trying to install a repository signature key

I am installing INetSim on Ubuntu and when running this command to install the signature key for the program: sudo wget -O - http://www.inetsim.org/inetsim.org-archive-signing-key.asc | apt-key add - ...
0
votes
1answer
666 views

How can I download files from shared Amazon Cloud Drive links through the command-line?

I have been using Amazon Cloud Drive (ACD) for a while now and I am very happy someone has developed a command-line-client for it (called acd_cli). It allows me to keep my files in "sync" with my ...
1
vote
2answers
84 views

Download a list of files and use domain name as filename

I got a bunch of files to download which works great with wget -i list_of_urls but my problem is, that wget uses the filename of the downloaded file. Is there a way (or a different tool) which is ...
0
votes
1answer
70 views

How to extract ip address from no-ip account webpage in the shell?

I've set dynamic dns for my home network(which is behind NAT) using www.noip.com. I use ddclient to send updates of my ip, but sometimes it takes a while to update the ip. Anyways it's not the point. ...