GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

learn more… | top users | synonyms

0
votes
1answer
43 views

Mirror server and ignore already-processed files

Before I start writing my own app for this, maybe there is already a better solution for the problem: I need to check an HTTP Server every day for new files to download and process. Basically these ...
-2
votes
1answer
115 views

Download files in bulk, but confirm each download

I wish to download all pictures from an album. The album is an HTML page hosted on the Internet, which has both links to some other stuff in the site and thumbnails, which link to the image viewing ...
2
votes
1answer
138 views

wget download from mirror download links

I'm trying to download using wget a tar.gz. The file is redirected based on the url parameters, so all downloads are redirected automatically. for example: wget ...
0
votes
1answer
97 views

Give Wget priority for my internet bandwidth

Is there any way for me to configure something so that no matter what other apps and machines are using my internet bandwidth, Wget will always get first priority? (So that even if I'm streaming lots ...
-1
votes
1answer
480 views

Iptables: allow wget

for some reason the following rules block wget. iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT iptables -A INPUT -p tcp --dport 80 -j ACCEPT iptables -A INPUT -p tcp ...
1
vote
2answers
349 views

Get latest exchange rate using command line?

I search for a way to get the last exchange rate online using the command line. I want to use it with another program. Until now all I have is this: wget -qO- ...
1
vote
1answer
162 views

Send HTTP request to website with password and username, then record results

I need to record certain numbers (temperature and others) from a web-based monitoring service (LaCrosse Alerts). However, you must login to use this service. I have an account, and am starting to ...
0
votes
1answer
384 views

Does wget have a download history?

I have used wget to download several compressed packages(in a failed attempt to manually emulate apt-get, if that is of any significance), and I don't remember the packages' locations. Is it possible ...
1
vote
1answer
214 views

How do you get the cookie of a website after an HTTPS NTLM authentification?

I am trying to store a cookie (which will help me authentificate myself elsewhere) given by a website after an authentification. The case is as follow: Get to the HTTPS page Get redirected by httpd ...
1
vote
1answer
179 views

Why doesn't twitter.com load on my home network?

I'm trying to figure out why my twitter clients stopped working. I am unable to load twitter via twitter clients or http://twitter.com/ on all machines connected to my home network. The connection ...
1
vote
2answers
1k views

Wget - Download all images from web server

So I want to download all images from a web server, particularly jpegs. The command I am running looks legit and I know the website has jpegs on it. So for example wget -r -P C:/ -A.jpg ...
0
votes
2answers
399 views

Using Wget / shell to download a file from outside of the web root

I am trying to download the httpd.conf file so that I can edit my Apache configuration. I can get access to the directory it is in via the shell (I have root access), but all of the examples on how ...
1
vote
0answers
54 views

Two files with same path and name - corrupted file system?

My computer is running windows XP 32 bit. I am using the following script to download the contents of a website to my hard drive, which is formatted as NTFS. wget-1.10.2.exe -r -x -p -nc -c -e ...
0
votes
3answers
53 views

Linux debian command

i tried to install isp cp omega on my debian dedicated server os, i created a new dir with name /panel/, and i wrote a command cd /panel, then I run the code wget ...
1
vote
1answer
2k views

`no such file or directory` when mirroring with wget

I'm trying to save a local version of the clojure docs with the wget command: wget --user-agent=firefox --ignore-length -c -km "http://clojuredocs.org/quickref/Clojure Core"` but keep getting a ...
0
votes
1answer
123 views

Generate a single PDF from a websites HTML pages

Here's the problem: There's a website that I often need for reference and I would like an offline version that also works in mobile devices, a PDF comes to mind. I can make an offline copy of the ...
0
votes
1answer
193 views

How to get Wget to retrieve entire url, including search terms

I'm trying to use a wget to retrieve a web page that that displays the results for a searched string. The page retrieved however, displays only the results of an empty string. From the looks of it, ...
0
votes
1answer
102 views

Wget getting responce 403

I am using API that has some limit of requests in an hour. But my script does all at one time so I lose about 1/3 of requests because I get 403. Is there any way to check response of wget and if I ...
1
vote
1answer
430 views

wget crawling search results of news website

I am trying to crawl the search results of a news website using wget. The name of the website is www.voanews.com. After typing in my search keyword and clicking search, it proceeds to the results. ...
0
votes
1answer
36 views

use wget to output everything retrieved into a single txt file

Is it possible to do something like this? wget --recursive --output-document=theEntirePagestuffedIntoATextfile.txt www.example.com And in words what it should do is: "download the entire website ...
5
votes
1answer
4k views

How do I use FireFox cookies with wget?

wget --load-cookies will load cookies as a "textual file in the format originally used by Netscape's cookies.txt file". However, firefox keeps its cookies in an sqlite database. Is there a way to ...
2
votes
1answer
2k views

How would I `wget` files and then save them by date downloaded rather than filename?

My goal: To download 131 JPEGs and save them in a file name format that is relative to the date/time format rather than their file name. I have already tried things that involve changing the files' ...
0
votes
2answers
475 views

How to download all of project euler's problems using wget

I'm trying to download all of Project Euler's problems using the command wget -m http://www.projecteuler.net/ but all it downloads is the first page i.e index.html and stops. I've tried using --user ...
1
vote
1answer
514 views

-bash: pear: command not found

When trying to install phpDoc through the Terminal I get the following error: -bash: pear: command not found Even though I have just installed pear. Prior to trying to install phpDoc I installed ...
1
vote
0answers
97 views

Ncat throws errors while being run from crontab

I have a simple script set up on my OpenWrt box to check if one of the computers in a local network is streaming, and if it does, the script starts wget | madplay to play the stream. Script works ...
0
votes
0answers
46 views

Determining wget arguments used, by file and output analysis

The title says almost all. I sometimes use wget to create seamless backups of certain webpages (for offline viewing and most importantly, to preserve them in case they go away). However, I have some ...
0
votes
1answer
56 views

Multiple versions of wget on the server

I would like to have multiple versions of wget installed on my server. I am currently on Debian 6.0 and have wget 1.12. For testing purposes I need also wget 1.14 installed parallel to 1.12. This ...
1
vote
3answers
14k views

Download ALL Folders, SubFolders, and Files using Wget

I have been using Wget, and I have run across an issue. I have a site,that has several folders and subfolders within the site. I need to download all of the contents within each folder and subfolder. ...
1
vote
1answer
792 views

wget execute script after download

I need every time you complete a transfer program wget this run a script. this is my full command line: wget -i urls-to-downloads.txt My problem is that I want to go subtracting it downloads the ...
0
votes
1answer
2k views

How do I use wget to download a list of files from different URLs on the same site and store the files in the same folder structure as the URL?

I have a list of files I need to download at different URLs on the same site. Using the command: wget --http-user:user@domain.name --http-password:password -i download.txt The files download to ...
0
votes
0answers
329 views

Galaxy Tab 10.1 / Android 4.0.4 wget always returning 404 error

I recently rooted my Android and installed busybox, but wget returns the same error not matter what. I even null routed the IP address. ( This is my personal domain, and I have the blackhole route ...
2
votes
1answer
1k views

How to get around ERROR 405 Method Not Allowed with HTTP post

I'm trying to make an HTTP post on this website: https://www.colorado.gov/revenueonline/_/#2 (click Verify a Sales Tax License in the bottom right) using wget with the following command: wget ...
2
votes
2answers
499 views

Wget for Windows error: “The application was unable to start correctly (0xc000007b).”

I downloaded the Wget for Windows binary from here: http://gnuwin32.sourceforge.net/packages/wget.htm I can't execute it without getting the following bizarre error: "The application was unable to ...
3
votes
2answers
480 views

Dowloading all Urls accessible under a given domain with wget without saving the actual pages?

Hi trying to determine all the valid urls under a given domain without having the mirror the site locally. People generally want to download all the pages but I just want to get a list of the direct ...
1
vote
2answers
3k views

Download entire directory from a website using Powershell

I have been using the following command to pull back files from a web server and store them on my local machine: $client = New-Object System.Net.WebClient ...
1
vote
1answer
197 views

How to resume wget mirroring from a different location?

I am out of space in my home folder , where iam currently mirroring this site for the pupose of a cygwin installation in my windows machine using wget from my linux machine. i am aware that i can ...
-2
votes
1answer
224 views

How to move a file of 250 GB to another server? [closed]

The hard disk of one of my servers is full, so I need to move a file away from it to make room. I have tried tar -cvf but when I send it to the other server using wget -c, I see it is 100% complete. ...
-1
votes
1answer
8k views

Wget Intallation in Cygwin

I tried to install wget version-1.14 (tar.xz package) on cygwin, but it failed. My steps were tar -xf wget-1.14.tar.xz cd wget-1.14 ./configure After some time, it spits a configure error. How to ...
1
vote
0answers
149 views

Using Wget to save pages that require logging in?

I've been trying to archive my forum private messages using Wget (its a custom forum server, so i cant just click "export all messages" like you can in phpbb). As a result, they're stuck behind a ...
1
vote
1answer
306 views

wget ftp with credentials for a list of files

This SuperUser question isn't quite what I need. Is there a way to do something like this wget ftp://myname:longpassword@someipddress/images/ -i file_list.txt I want to use the same credentials and ...
0
votes
2answers
349 views

Script for CRON (FTP get files EVERY MINUTE)

I want to create a script that must run with a CRON every minute (Q: Is it wise to run CRON FTP/wget every minute?) compares local directory with remote directory (Q: do I use FTP or wget and how ...
0
votes
0answers
113 views

Files not downloadable via Browser but with Wget. Why?

Recently I came across an issue on a client machine where the machine was not able to download some files. We tried to download the files via Browser, but it did not generate any response. However, ...
3
votes
1answer
2k views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
0
votes
2answers
4k views

Diagnose internet connection issues using command line

I'm having multiple issues trying to get CentOS to download anything. I can't get yum to work because there is a time out when trying to access a file. Anyways, is there a way to trace why a command ...
0
votes
2answers
69 views

Automating OpenOffice Installation

I am attempting to write a shell script which will download the latest version of OpenOffice using wget, and then use the command-line to install the application. I have seen that installing the ...
0
votes
1answer
109 views

Download all images at distance 2 using Mac OS X

I need to download all the images from a website on my computer. The images are not listed all in one page. Instead I have an index page that links a set of pages (page1.html, page2.html, ...) each ...
2
votes
1answer
2k views

Windows: equivalent of OS X's Terminal “&”

Is there any way, under Windows 7, to emulate the behaviour of the & character in OS X's Terminal? I'm trying to run multiple Wget instances simultaneously: I can do this without any problem of ...
6
votes
1answer
517 views

How can I use a command line tool like Wget to log into an OpenID site?

Specifically, I would like to be able to download certain pages from my user profile on the various Stack Exchange sites. I would, however, like to do this automatically (using a cron job), from ...
0
votes
1answer
497 views

Using Wget to download single images from a website using the the img src URL where img class=“decoded”

I want to get single images off a website. The website has photos. Clicking on the photos leads to another page. I want to get the photos from the img src tag in the html. I already have unix code ...
1
vote
1answer
49 views

Any Way to Mirror the Data Needed Only to Load the index.html Page of a Site?

I'm trying to mirror a site's landing page to do testing on it. Is there any way to mirror only the first level of a site? I.E. all requests should become local files only for the first index page ...