Questions tagged [wget]
GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive command line tool, so it may easily be called from scripts, Cron jobs, terminals without X Window System (X11) support, etc.
901
questions
45
votes
1
answer
45k
views
Make wget convert HTML links to relative after download if -k wasn't specified
The -k option (or --convert-link) will convert links in your web pages to relative after the download finishes, such as the man page says:
After the download is complete,
convert the links in the ...
1
vote
3
answers
2k
views
Bash: executing concatenated variables
I have a Linux bash script, to the effect of:
COMMAND_TO_EXECUTE="foo "$1" --option1 --option2 "$2
exec $COMMAND_TO_EXECUTE
Problem: the script fails
Debugging clue, If I echo $...
0
votes
1
answer
378
views
Download file on unix from target url which autodownloads file on page visit
I'm trying to download a file to a unix machine on the command line. Accessing a GUI on this particular machine would be quite difficult. The file is autodownloaded when visiting the url in a web ...
55
votes
2
answers
69k
views
How do I properly set wget to download only new files?
Let's say there's an url, let's call it http://www.some-url.com/folder/
This location has directory listing enabled, therefore I can do this:
wget -r -np http://www.some-url.com/folder/
To download ...
2
votes
1
answer
1k
views
Using wget does not give same output as browser
I am using wget for Windows (gnuwin32 wget-1.11.4-1) in Windows 8 and using it for a helpdesk tool called Kayako, telling it to poll from an email queue. The command line looks like this:
wget.exe -O ...
0
votes
0
answers
80
views
Http Starts failing after a while using windows 8.1
I have problem with my connection on windows 8.1, it looks fine when restart my system, for this wget call.
wget http://www.google.com/
as following result.
--2014-11-26 17:59:37-- http://www....
2
votes
2
answers
1k
views
Why doesn't wget -m save the index.html documents
I'm making a copy of a website with
wget -m -p -q http://example.com
The process works fine, but the copy is missing all index.html files, even so I see them being sucessfully requested and all ...
2
votes
1
answer
2k
views
How to force wget to output a summary?
I have a bash script that launches a series of wget commands and log the output. I use the following options:
wget --no-verbose --page-requisites --adjust-extension --convert-links --backup-converted ...
0
votes
1
answer
446
views
Are there any programs that supports NRPC? (Lotus Notes RPC) [closed]
Are there any implementations of the Lotus Notes RPC (port 1352)?
Example:
wget NRPC://<Server-hint-or-host>/bar/foo.xls -O foo.xls
So that we can download a file that is only reachable via a ...
0
votes
1
answer
98
views
can't connect to local web server
when i try to connect to a web server installed on a pc in lan i can't get the index page, the computer is well configured because i can ping and ssh it:
ping
$ ping 172.16.69.196
PING 172.16.69.196 ...
1
vote
1
answer
698
views
What is wrong in the Bash script about parameters and Wget?
Runnning the command gives
wget_exam -h
Usage: wget_exam2 <fileType> <source>
exit // then immediately Terminal shut dows
The code
# example wget_exam2 java http://www.example.com/...
9
votes
3
answers
24k
views
Getting all the filenames (not content) recursively from an http directory
A large biological research project has chosen to make its archive available via https here:
https://tcga-data.nci.nih.gov/tcgafiles/ftp_auth/distro_ftpusers/anonymous/tumor/
Unfortunately, it ...
0
votes
0
answers
76
views
use wget for the most updated file on http server
I need to download the most updated files with the name suite.
I am using windows server to run the command lines.
I wrote the following command but not sure of how to get the most updated one
wget ...
0
votes
1
answer
5k
views
Where is the default directory of saved files using wget in crontab?
I want to download some files using wget command in crontab. I use a text file to make a list of my desired urls. wget's -O parameter works well for setting a destination folder for single files, but ...
7
votes
2
answers
12k
views
Wget save cookies not working
I've been trying to login in the pyload through the web api, but wget is not saving the cookies and I don't understand why.
I'm using the following command:
wget --delete-after --keep-session-...
3
votes
1
answer
5k
views
Why does wget download specific files, but can't download them all at once?
I am using Windows XP and am trying to use wget to download all the pictures (and some other files) from my website which is going to be closed on a host in about two weeks (so I need to hurry).
I ...
2
votes
2
answers
4k
views
curl can't connect; wget can
trying to connect to my script on a remote server with curl but:
col-pos-spd# curl -v http://***:80/monitor/reply.php
* About to connect() to *** port 80 (#0)
* Trying 83.xxx.xxx.2... Connection ...
1
vote
1
answer
3k
views
Download all files with some strings in their names
On http://www.inf.usi.ch/carzaniga/edu/algo08f/schedule.html, I would like to download all pdf files of the name *-2up.pdf, for example,
http://www.inf.usi.ch/carzaniga/edu/algo08f/intro-2up.pdf
Can ...
2
votes
0
answers
2k
views
wget: unable to resolve host address
I've RedHat Enterprise version 5.2.
I run this
wget --http-user=celee --http-password=1010$ -k -nd -np -r http://sp.kch.de/RandD/DesSup/Mask%20Tooling%20Runsets/ -o wget.log
The wget.log file give ...
0
votes
0
answers
532
views
Does wget download files more than once?
Suppose there is a website where Page A links to Page B and pages A and B both link to page C.
Somthing like
A -> B -> C
|
v
C
If I run recursive wget on A to begin downloading the pages ...
11
votes
3
answers
5k
views
Wget with URL that contains #
I am trying to download a URL that is like http://www.somesite.com/restaurants.html#photo=22x00085.
I put it in between single quotes, but it only downloads http://www.somesite.com/restaurants.html ...
0
votes
0
answers
156
views
How to Download HTML Pages of a WebSite, Based on a Time Stamp?
I know this might be a repeated question, and I'm sorry for that,
but I failed to find any answer for my problem .
I Just want to download the Pages html files, without any other contents,
and (If ...
1
vote
3
answers
294
views
wget not doing what its supposed to do
I have a simple batch file which is supposed to install Flash by going to the website and selecting the download button. However what it does is entirely different: it goes to the website and the site ...
2
votes
2
answers
7k
views
Downloading an (apache) online directory locally using cli (wget ...)
Not really a question this one - but alternative answers are certainly welcome! :)
Sometimes I need to download files listed in an Apache directory listing, say as in:
http://www.gnu-darwin.org/...
3
votes
1
answer
2k
views
How to use wget to download images from messages on facebook?
This look a bit harder than usual. I'm a new user wget and i was thinking that maybe it's possible to somehow login into facebook and download only images from a specific conversation.
I mean, i'd ...
5
votes
2
answers
13k
views
How to set wget to timeout after downloading more than a fixed duration
I want wget to (attempt to) start downloading, then if isn't done after 10 seconds, I want it to give up.
wget --timeout 10 http://url
Seems like a reasonable try, but it seems like it only sets --...
2
votes
1
answer
1k
views
Wget - Problems recursively downloading with authentication
So I'm trying to download the entire domain of a private wiki page. I've saved the cookies in a cookies.txt file and using it with wget for authentication like so:
wget --load-cookies=cookies.txt --...
12
votes
3
answers
11k
views
Make wget not download files larger than X size
Okay, I give up. How do I size limit which files are downloaded, like say I don't want any files bigger than 2 MB?
2
votes
1
answer
6k
views
How do you find a size of a web-site? [closed]
Suppose I want to download some wiki site. It looks like it has 2000 articles only - but with wiki technology - they keep several versions of each article... So it still could be terabytes!
So my ...
1
vote
1
answer
629
views
Wget downloaded more data than the actual file
I have downloaded lubuntu image using wget with mentioned file size 694MB , but the downloaded file is of size 789MB. when i do wget -c it gives that the file is already downloaded. But the file size ...
0
votes
1
answer
406
views
Mirror an ftp site, but create empty files instead of full files to save bandwidth
I want a listing of all files and directories at ftp://ftp.ncbi.nlm.nih.gov/snp/. But I don't want to download the contents of all the files.
To save bandwidth, I hoped that wget would have an option ...
1
vote
1
answer
87
views
amazon ec2 intance rhel7 ftp not downloading
this is driving me insane.
i am trying to download files using FTP. i tried with PHP's FTP, the FTP binary, WGET, none of them work. They all have the same problem. When get command is initiated, the ...
7
votes
1
answer
5k
views
Can wget convert links within a local html document? [duplicate]
I know that wget can fetch a remote page and its dependencies and rewrite the html so that image src attributes reference the newly downloaded images.
I am trying to convert local html files that ...
0
votes
1
answer
452
views
How I can download PDFs of a website by using only the root domain name?
I am using this command:
wget -nd -e robots=off --wait 0.25 -r -A.pdf http://yourWebsite.net/
but I can't get PDFs from the website.
For example I have a root domain name:
www.example.com
and ...
4
votes
1
answer
5k
views
wget fails but opening in browser succeeds. How?
I was trying to download a pdf from the intel website at this address. When I open it through the browser(google-chrome) there seems to be no problem in viewing it or storing it on the my hard disk. ...
0
votes
1
answer
368
views
can't open .gzip archives retrieved with wget
I'm trying to download the content of a directory hosted on a server using wget in a cygwin terminal.
I have no problem getting the files but they are all .gz archive and once downloaded they are ...
0
votes
1
answer
2k
views
wget download a URL with filter
I want to download all jpg files of URL "www.link.com". However the jpg download links start with "dl.link.com".
When I use the following code, it does not download any file. I assume because the ...
0
votes
1
answer
2k
views
How do I mirror sub directory of a website with wget
I need to mirror the following URL:
https://web.archive.org/web/20100515230709/http://markowitzbocadentist.com/
But I need to mirror only files from this date '20100515230709' i have tried many wget ...
0
votes
1
answer
501
views
Try to download mp3 with wget but it redirects to another site?
I am trying to download mp3 files from this adress hanialrefai using wget like this:
wget -r -l0 -k -p -A.mp3 http://hanialrefai.com
but it redirects me to http://quantsa.ru/?pw and then it say: ...
0
votes
2
answers
4k
views
What is the simplest way to download with wget in remote putty terminal using my local windows internet connection
I'd like to initate a download using wget in a remote putty terminal (i.e. an ubuntu server), but using my local windows internet connection.
I guess there is two way to achieve this:
run an HTTP ...
0
votes
1
answer
218
views
Downloading Images through Wget With Proxy
I've written a ruby tool to download multiple images. However, this time I'm having to configure wget with a proxy. Prefix is the specified directory of where I want the images to go and md5_sum is a ...
2
votes
1
answer
13k
views
wget mirroring the site fails - 403 Forbidden, even with user-agent
I need to download suricata's latest rules from this url:
http://rules.emergingthreats.net/blockrules/
I'm trying to compose my own rules based on different emerging rules and snort's rules too. ...
2
votes
2
answers
6k
views
How to download an image with wget and save it with the md5 hash as name?
How can I make a download an image, md5 hash the image, and save that image with the md5 hash as name to a directory using wget?
# An example of the image link...
http://31.media.tumblr.com/...
0
votes
2
answers
2k
views
Can wget decode uri file names when downloading in batch?
Normally when downloading files using wget, the filenames are encoded.
In example:
wget -c --content-disposition "https://class.coursera.org/ml-005/lecture/download.mp4?lecture_id=1"
and the saved ...
0
votes
1
answer
838
views
How to navigate PHP-based pagination with WGET?
I'm trying to come up with a list of possible names for our next baby boy and have been looking through the site IndiaParenting. Each name on the site has a detail page at a URL like this one http://...
0
votes
1
answer
900
views
wget doesn't match patterns with wildcards
I'm trying to download certain url's which I would like to match with the next pattern using wildcards:
http://example.org/subtitles?q=NUMBER_en&format=srt
where NUMBER = [0-9]*
the output of ...
0
votes
1
answer
386
views
How to automatically save a list of files with different names?
I have a text file of URLs, each in the form of:
http://domain.com/a/b/c/d.jpg
I want to download all of these, but save each file under the name:
c_d.jpg
In other words, for each file, I want to ...
2
votes
2
answers
9k
views
Using wget to get files from local to server to remote server
We have a local server where we carry out much of our web development and then we push this to a live environment so that clients can access.
We often end up with huge tarballs (this one in ...
1
vote
1
answer
781
views
How do I make wget match exact domain names?
I'm using the following command to mirror a site:
wget -r -p -E -N -l 10 -k -nH -H -Dfoo.com,www.foo.com foo.com
The problem I am having is that somewhere on foo.com there are links to domains ...
0
votes
0
answers
593
views
How to download all images from a gallery
I'm searching for a way to download all images (not the thumbnails) of this gallery
http://kinki.de/#/?gallery=list&gid=4804 .
As I can see, the images are all located at
http://kinki.de/...