Questions tagged [wget]

GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive command line tool, so it may easily be called from scripts, Cron jobs, terminals without X Window System (X11) support, etc.

Filter by
Sorted by
Tagged with
45 votes
1 answer
45k views

Make wget convert HTML links to relative after download if -k wasn't specified

The -k option (or --convert-link) will convert links in your web pages to relative after the download finishes, such as the man page says: After the download is complete, convert the links in the ...
1 vote
3 answers
2k views

Bash: executing concatenated variables

I have a Linux bash script, to the effect of: COMMAND_TO_EXECUTE="foo "$1" --option1 --option2 "$2 exec $COMMAND_TO_EXECUTE Problem: the script fails Debugging clue, If I echo $...
0 votes
1 answer
378 views

Download file on unix from target url which autodownloads file on page visit

I'm trying to download a file to a unix machine on the command line. Accessing a GUI on this particular machine would be quite difficult. The file is autodownloaded when visiting the url in a web ...
55 votes
2 answers
69k views

How do I properly set wget to download only new files?

Let's say there's an url, let's call it http://www.some-url.com/folder/ This location has directory listing enabled, therefore I can do this: wget -r -np http://www.some-url.com/folder/ To download ...
2 votes
1 answer
1k views

Using wget does not give same output as browser

I am using wget for Windows (gnuwin32 wget-1.11.4-1) in Windows 8 and using it for a helpdesk tool called Kayako, telling it to poll from an email queue. The command line looks like this: wget.exe -O ...
0 votes
0 answers
80 views

Http Starts failing after a while using windows 8.1

I have problem with my connection on windows 8.1, it looks fine when restart my system, for this wget call. wget http://www.google.com/ as following result. --2014-11-26 17:59:37-- http://www....
2 votes
2 answers
1k views

Why doesn't wget -m save the index.html documents

I'm making a copy of a website with wget -m -p -q http://example.com The process works fine, but the copy is missing all index.html files, even so I see them being sucessfully requested and all ...
2 votes
1 answer
2k views

How to force wget to output a summary?

I have a bash script that launches a series of wget commands and log the output. I use the following options: wget --no-verbose --page-requisites --adjust-extension --convert-links --backup-converted ...
0 votes
1 answer
446 views

Are there any programs that supports NRPC? (Lotus Notes RPC) [closed]

Are there any implementations of the Lotus Notes RPC (port 1352)? Example: wget NRPC://<Server-hint-or-host>/bar/foo.xls -O foo.xls So that we can download a file that is only reachable via a ...
0 votes
1 answer
98 views

can't connect to local web server

when i try to connect to a web server installed on a pc in lan i can't get the index page, the computer is well configured because i can ping and ssh it: ping $ ping 172.16.69.196 PING 172.16.69.196 ...
1 vote
1 answer
698 views

What is wrong in the Bash script about parameters and Wget?

Runnning the command gives wget_exam -h Usage: wget_exam2 <fileType> <source> exit // then immediately Terminal shut dows The code # example wget_exam2 java http://www.example.com/...
9 votes
3 answers
24k views

Getting all the filenames (not content) recursively from an http directory

A large biological research project has chosen to make its archive available via https here: https://tcga-data.nci.nih.gov/tcgafiles/ftp_auth/distro_ftpusers/anonymous/tumor/ Unfortunately, it ...
0 votes
0 answers
76 views

use wget for the most updated file on http server

I need to download the most updated files with the name suite. I am using windows server to run the command lines. I wrote the following command but not sure of how to get the most updated one wget ...
0 votes
1 answer
5k views

Where is the default directory of saved files using wget in crontab?

I want to download some files using wget command in crontab. I use a text file to make a list of my desired urls. wget's -O parameter works well for setting a destination folder for single files, but ...
7 votes
2 answers
12k views

Wget save cookies not working

I've been trying to login in the pyload through the web api, but wget is not saving the cookies and I don't understand why. I'm using the following command: wget --delete-after --keep-session-...
3 votes
1 answer
5k views

Why does wget download specific files, but can't download them all at once?

I am using Windows XP and am trying to use wget to download all the pictures (and some other files) from my website which is going to be closed on a host in about two weeks (so I need to hurry). I ...
2 votes
2 answers
4k views

curl can't connect; wget can

trying to connect to my script on a remote server with curl but: col-pos-spd# curl -v http://***:80/monitor/reply.php * About to connect() to *** port 80 (#0) * Trying 83.xxx.xxx.2... Connection ...
1 vote
1 answer
3k views

Download all files with some strings in their names

On http://www.inf.usi.ch/carzaniga/edu/algo08f/schedule.html, I would like to download all pdf files of the name *-2up.pdf, for example, http://www.inf.usi.ch/carzaniga/edu/algo08f/intro-2up.pdf Can ...
2 votes
0 answers
2k views

wget: unable to resolve host address

I've RedHat Enterprise version 5.2. I run this wget --http-user=celee --http-password=1010$ -k -nd -np -r http://sp.kch.de/RandD/DesSup/Mask%20Tooling%20Runsets/ -o wget.log The wget.log file give ...
0 votes
0 answers
532 views

Does wget download files more than once?

Suppose there is a website where Page A links to Page B and pages A and B both link to page C. Somthing like A -> B -> C | v C If I run recursive wget on A to begin downloading the pages ...
11 votes
3 answers
5k views

Wget with URL that contains #

I am trying to download a URL that is like http://www.somesite.com/restaurants.html#photo=22x00085. I put it in between single quotes, but it only downloads http://www.somesite.com/restaurants.html ...
0 votes
0 answers
156 views

How to Download HTML Pages of a WebSite, Based on a Time Stamp?

I know this might be a repeated question, and I'm sorry for that, but I failed to find any answer for my problem . I Just want to download the Pages html files, without any other contents, and (If ...
1 vote
3 answers
294 views

wget not doing what its supposed to do

I have a simple batch file which is supposed to install Flash by going to the website and selecting the download button. However what it does is entirely different: it goes to the website and the site ...
2 votes
2 answers
7k views

Downloading an (apache) online directory locally using cli (wget ...)

Not really a question this one - but alternative answers are certainly welcome! :) Sometimes I need to download files listed in an Apache directory listing, say as in: http://www.gnu-darwin.org/...
3 votes
1 answer
2k views

How to use wget to download images from messages on facebook?

This look a bit harder than usual. I'm a new user wget and i was thinking that maybe it's possible to somehow login into facebook and download only images from a specific conversation. I mean, i'd ...
5 votes
2 answers
13k views

How to set wget to timeout after downloading more than a fixed duration

I want wget to (attempt to) start downloading, then if isn't done after 10 seconds, I want it to give up. wget --timeout 10 http://url Seems like a reasonable try, but it seems like it only sets --...
2 votes
1 answer
1k views

Wget - Problems recursively downloading with authentication

So I'm trying to download the entire domain of a private wiki page. I've saved the cookies in a cookies.txt file and using it with wget for authentication like so: wget --load-cookies=cookies.txt --...
12 votes
3 answers
11k views

Make wget not download files larger than X size

Okay, I give up. How do I size limit which files are downloaded, like say I don't want any files bigger than 2 MB?
2 votes
1 answer
6k views

How do you find a size of a web-site? [closed]

Suppose I want to download some wiki site. It looks like it has 2000 articles only - but with wiki technology - they keep several versions of each article... So it still could be terabytes! So my ...
1 vote
1 answer
629 views

Wget downloaded more data than the actual file

I have downloaded lubuntu image using wget with mentioned file size 694MB , but the downloaded file is of size 789MB. when i do wget -c it gives that the file is already downloaded. But the file size ...
0 votes
1 answer
406 views

Mirror an ftp site, but create empty files instead of full files to save bandwidth

I want a listing of all files and directories at ftp://ftp.ncbi.nlm.nih.gov/snp/. But I don't want to download the contents of all the files. To save bandwidth, I hoped that wget would have an option ...
1 vote
1 answer
87 views

amazon ec2 intance rhel7 ftp not downloading

this is driving me insane. i am trying to download files using FTP. i tried with PHP's FTP, the FTP binary, WGET, none of them work. They all have the same problem. When get command is initiated, the ...
7 votes
1 answer
5k views

Can wget convert links within a local html document? [duplicate]

I know that wget can fetch a remote page and its dependencies and rewrite the html so that image src attributes reference the newly downloaded images. I am trying to convert local html files that ...
0 votes
1 answer
452 views

How I can download PDFs of a website by using only the root domain name?

I am using this command: wget -nd -e robots=off --wait 0.25 -r -A.pdf http://yourWebsite.net/ but I can't get PDFs from the website. For example I have a root domain name: www.example.com and ...
4 votes
1 answer
5k views

wget fails but opening in browser succeeds. How?

I was trying to download a pdf from the intel website at this address. When I open it through the browser(google-chrome) there seems to be no problem in viewing it or storing it on the my hard disk. ...
0 votes
1 answer
368 views

can't open .gzip archives retrieved with wget

I'm trying to download the content of a directory hosted on a server using wget in a cygwin terminal. I have no problem getting the files but they are all .gz archive and once downloaded they are ...
0 votes
1 answer
2k views

wget download a URL with filter

I want to download all jpg files of URL "www.link.com". However the jpg download links start with "dl.link.com". When I use the following code, it does not download any file. I assume because the ...
0 votes
1 answer
2k views

How do I mirror sub directory of a website with wget

I need to mirror the following URL: https://web.archive.org/web/20100515230709/http://markowitzbocadentist.com/ But I need to mirror only files from this date '20100515230709' i have tried many wget ...
0 votes
1 answer
501 views

Try to download mp3 with wget but it redirects to another site?

I am trying to download mp3 files from this adress hanialrefai using wget like this: wget -r -l0 -k -p -A.mp3 http://hanialrefai.com but it redirects me to http://quantsa.ru/?pw and then it say: ...
0 votes
2 answers
4k views

What is the simplest way to download with wget in remote putty terminal using my local windows internet connection

I'd like to initate a download using wget in a remote putty terminal (i.e. an ubuntu server), but using my local windows internet connection. I guess there is two way to achieve this: run an HTTP ...
0 votes
1 answer
218 views

Downloading Images through Wget With Proxy

I've written a ruby tool to download multiple images. However, this time I'm having to configure wget with a proxy. Prefix is the specified directory of where I want the images to go and md5_sum is a ...
2 votes
1 answer
13k views

wget mirroring the site fails - 403 Forbidden, even with user-agent

I need to download suricata's latest rules from this url: http://rules.emergingthreats.net/blockrules/ I'm trying to compose my own rules based on different emerging rules and snort's rules too. ...
2 votes
2 answers
6k views

How to download an image with wget and save it with the md5 hash as name?

How can I make a download an image, md5 hash the image, and save that image with the md5 hash as name to a directory using wget? # An example of the image link... http://31.media.tumblr.com/...
0 votes
2 answers
2k views

Can wget decode uri file names when downloading in batch?

Normally when downloading files using wget, the filenames are encoded. In example: wget -c --content-disposition "https://class.coursera.org/ml-005/lecture/download.mp4?lecture_id=1" and the saved ...
0 votes
1 answer
838 views

How to navigate PHP-based pagination with WGET?

I'm trying to come up with a list of possible names for our next baby boy and have been looking through the site IndiaParenting. Each name on the site has a detail page at a URL like this one http://...
0 votes
1 answer
900 views

wget doesn't match patterns with wildcards

I'm trying to download certain url's which I would like to match with the next pattern using wildcards: http://example.org/subtitles?q=NUMBER_en&format=srt where NUMBER = [0-9]* the output of ...
0 votes
1 answer
386 views

How to automatically save a list of files with different names?

I have a text file of URLs, each in the form of: http://domain.com/a/b/c/d.jpg I want to download all of these, but save each file under the name: c_d.jpg In other words, for each file, I want to ...
2 votes
2 answers
9k views

Using wget to get files from local to server to remote server

We have a local server where we carry out much of our web development and then we push this to a live environment so that clients can access. We often end up with huge tarballs (this one in ...
1 vote
1 answer
781 views

How do I make wget match exact domain names?

I'm using the following command to mirror a site: wget -r -p -E -N -l 10 -k -nH -H -Dfoo.com,www.foo.com foo.com The problem I am having is that somewhere on foo.com there are links to domains ...
0 votes
0 answers
593 views

How to download all images from a gallery

I'm searching for a way to download all images (not the thumbnails) of this gallery http://kinki.de/#/?gallery=list&gid=4804 . As I can see, the images are all located at http://kinki.de/...

1
11 12
13
14 15
19