GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.
1
vote
1answer
152 views
How to download YouTube videos via HTTPS [duplicate]
I am unable to access YouTube on http. But I can access the same on an https.
When I executed the command wget https://www.youtube.com/watch?v=StJLvbPIvTw it returned the following response:-
...
1
vote
1answer
148 views
wget for ftp using a password containing @
I am trying to get some files from my ftp server from the command line. I am using wget to download the whole folder at once. The command is:
wget -m ...
1
vote
1answer
261 views
wget “giving up” — closed fd 3
I apologize in advance for being dense but I can't figure out why wget is "giving up". Here is the command I'm using.
wget --debug --tries 1 --read-timeout=900 --timeout=0 ...
1
vote
1answer
164 views
How to Download Multiple XML files from different URLs
@slhck provided a useful answer to this similar question.
Downloading multiple files with wget and handling parameters
This code was provided to read urls from a text file filelist, then run a loop ...
1
vote
1answer
229 views
How to use delicious API from command line
I'm trying to add a delicious bookmark from a shell script, but keep getting the mysterious "something went wrong" error. I've checked the official spec and several examples and everything looks ...
1
vote
3answers
1k views
wget - download all files but not preceding folders
I'm using wget to download all files from within a folder using the -r and -np options. However this also downloads the preceding folders, which I don't want.
For example:
wget -r -np ...
1
vote
1answer
474 views
Submitting a form using curl
Forgive me for the simple question... I haven't used curl before. I'm trying to automatically download some data from this form: http://ida.water.usgs.gov/ida/available_records.cfm?sn=12040500
which ...
1
vote
1answer
271 views
Scrape a site with curl/wget appending a parameter to each site
I need to scrape a website for all sites and append an Paramater, e.g.
www.theSite.com/allpages?myPara=us
Now what I got is wget --recursive to get all sites, but that doesn't append the parameter ...
1
vote
1answer
132 views
Failed to continue 6.4GB download after 4.2GB already downloaded
I was trying to download a large iso (6,822,254,592 bytes). After 4,423,760,272 bytes (which is more than 4GB, so this isn't a simple 4GB limit problem) my network connection was interrupted for long ...
1
vote
1answer
491 views
curl can't connect; wget can
trying to connect to my script on a remote server with curl but:
col-pos-spd# curl -v http://***:80/monitor/reply.php
* About to connect() to *** port 80 (#0)
* Trying 83.xxx.xxx.2... Connection ...
1
vote
1answer
138 views
How to run a command inside a script without specifying the exact location?
So I have a PHP script that is using wget to download some images. It looks something like this:
exec('wget -O saveImage.png http://location.of.image');
When I run my script on a different machine, ...
1
vote
1answer
752 views
Wget Pattern Problem
I am having a problem with wget and accept patterns.
What I want to do is only download files which match the pattern
\*/images/src/test\*.jpg.
I am using the command
wget -r -A ...
1
vote
1answer
192 views
Website wordcount using WGET
I would like to know if it's possible to use wget to wordcount an entire website.
Has anyone already tried that?
This is what I have found so far:
wget -q -O- http://www.website.com | grep -c .
1
vote
3answers
800 views
Downloading images from ftp site using wget
I'm trying to set up a script that will synchronise all images in a remote FTP directory with a local directory that I can run on a regular basis to make sure I have all new images added to the remote ...
1
vote
2answers
943 views
wget-ing protected content with exported cookies
I have exported a pair of cookies from Firefox that are valid for the URL in question and tried accessing/downloading the protected content off that address, but the end result is a return to the ...
1
vote
1answer
871 views
rss downloader script
I have a Synology NAS that is powered by linux at my house. I'm looking to set up a cron script to check a group of rss feeds and auto download new video podcasts to a shared folder. I can do most of ...
1
vote
1answer
502 views
Wget a directory with exact filenames?
The following works because I inserted the exact filename:
wget --referer=http://www.*****.com --cookies=on --load-cookies=cookie.txt --keep-session-cookies --save-cookies=cookie.txt ...
1
vote
1answer
1k views
How to download images with wget
I'm doing this wget script called wget-images, which should download images from a website. It looks like this now:
wget -e robots=off -r -l1 --no-parent -A.jpg
The thing is, in the terminal when i ...
1
vote
1answer
2k views
What units does wget use for bandwidth?
When downloading, wget reports speed in "K/s". K...what? kilobits? kilobytes? 1024 or 1000?
Update:
wget -O /dev/null http://newark1.linode.com/100MB-newark.bin
produces "348 K/s". ...
1
vote
2answers
348 views
Downloading a large site with wget
I'm trying to mirror a very large site but wget never seems to finish properly. I am using the command:
wget -r -l inf -nc -w 0.5 {the-site}
I have downloaded a good portion of the site, but not ...
1
vote
2answers
736 views
Download Speed cURL
I am having a bit of problem using cURL to download using FTP, that is to say I think it's being rather slow if working at all. I usually use wget an this one file which I downloaded earlier with wget ...
1
vote
1answer
127 views
Wget overwites files when mirroring a multilingual website
I would like to mirror a website using wget. The problem is that the website has several language mutations which are switched using a query-string param and when wget starts to download another ...
1
vote
1answer
2k views
Why does wget download specific files, but can't download them all at once?
I am using Windows XP and am trying to use wget to download all the pictures (and some other files) from my website which is going to be closed on a host in about two weeks (so I need to hurry).
I ...
1
vote
1answer
78 views
wget download pdf files fail
I am using:
wget -r -A pdf https://labraj.uni-mb.si
but instead of pdf's I get this output in console:
--2013-03-23 15:11:03-- https://labraj.uni-mb.si/
Resolving labraj.uni-mb.si ...
1
vote
1answer
37 views
continue downloading files starting from the url where it stops : WGET
what I want to do is to download files using wget. The maximum depth is up to 3. Now, I know that there is a big possibility that the files in the url will be updated.
Is there a way that these ...
1
vote
1answer
35 views
how to download weblinks using wget which do not end with .html?
I want to download this webpage using wget in Win7 : ...
1
vote
1answer
101 views
How to download multiple urls using wget using a single command?
I am using following command to download a single webpage with all its images and js using wget in win7:
wget -E -H -k -K -p -e robots=off -P /Downloads/ ...
1
vote
0answers
139 views
Create a complete local copy of a website using Wget
OK, here is what I need :
I'm on Mac (Mac OS X 10.6.8)
I want to completely mirror a website on my hard drive (this is what I'm using as a test)
I want all images and prerequisites there, so that ...
1
vote
0answers
127 views
Trouble using wget or httrack to mirror archived website
I am trying to use wget to create a local mirror of a website. But I am finding that I am not getting all the linking pages.
Here is the website
...
1
vote
1answer
142 views
Ask wget to download directory index pages?
I'm downloading all the files from a website below a specific node.
wget \
--recursive \
--no-clobber \
--page-requisites \
--html-extension \
--convert-links \
--domains ...
1
vote
0answers
57 views
Unable to get the information using wget using session ID
I have been using Wget to pull the warranty information from my website using the following command.
wget -q -O getinfo.tmp ...
1
vote
0answers
51 views
the right options to traverse/download the pages/directories of a subdomain
Let's suppose exist a site with the following directories (subdomain)
index.php
|-sub1
|-index.php
|-sub1sub1
|-index.php
|-other.php
|-sub1sub1sub1
|-sub2
|-index.php
|- ...
1
vote
0answers
29 views
Syncing offline material with website using wget
I am using wget on Mac OS to download a website. This website is updated once a while and I don't really want to download the whole website everytime it gets updated. is there anyway I can 'sort of' ...
1
vote
0answers
80 views
wget filename and directory name clashes
We are using WGET 1.12 for windows on a site (with follow_tags = a, reclevel = 8, recursive = on and timestamping = on wgetrc settings) and are seeing two issues:
a. The file is deleted and a ...
1
vote
0answers
54 views
Download web server structure with empty files
I want to make a mirror of a Web server, but downloading the actual files will take too long. So I thought of having just the directory and file structure, and when I need the actual contents of the ...
1
vote
1answer
231 views
Exclude list of specific files in wget
I am trying to download a lot of pages from a website on dial-up and it can be brutally slow. I have almost got the perfect wget command, but because I'm downloading pages from the same site wget ...
1
vote
1answer
577 views
wget not working on one machine: 404 Not Found
I am trying to use wget to debug and issue I am having. I have a remote machine from one server trying to reach query.json and it works fine, connects and downloads fine. I have another remote machine ...
1
vote
1answer
226 views
wget not return proper page
When I visited this website throught my Firefox 13, I got a page with some content. But when I use wget to download it :
wget http://tinhvan.com
I got other content on downloaded HTML page. Tried ...
1
vote
1answer
312 views
Grab source from Google Code using wget or curl?
Is it possible to grab the source-code from a Google Code repository (hg) using wget or curl?
Else using Python?
Thanks for all suggestions
FYI: I'm on a shared WebFaction host, so can't install hg
...
1
vote
0answers
162 views
wget - only getting .listing file in every sub-dir
if I use the command "wget --no-remove-listing -P ...../debugdir/gnu// ftp:/gnu//" I will get the .listing file of that directory. But I have to step through each subsequent sub-directories to get the ...
1
vote
0answers
42 views
Download pages that I have to log in to get with wget
There's a website that I must log in to before I can browse the content, and the content would be very helpful for my work if I could browse it while offline. It's training and repair information for ...
1
vote
2answers
470 views
How to set wget to timeout after downloading more than a fixed duration
I want wget to (attempt to) start downloading, then if isn't done after 10 seconds, I want it to give up.
wget --timeout 10 http://url
Seems like a reasonable try, but it seems like it only sets ...
1
vote
1answer
165 views
wget clone not exact
I am trying to get an exact replica of one page. I use:
wget -E -H -k -K -p http://example.com
Everything works well except:
If in my style there is an import of another CSS file it does not get ...
1
vote
0answers
178 views
Debian on Dockstar : querying ADSL modem admin page with wget
New to Linux, I'm trying to query the status page of my DG834G modem with wget from my Dockstar running debian. Not sure it makes any difference but I'm connected to the Dockstar thru SSH from a ...
1
vote
0answers
1k views
Wget: Name or service not known (Arch Linux x64)
I recently returned to my college campus for classes and brought my desktop computer running 64 bit Arch Linux out of storage. After updating my packages with sudo pacman -Syu, I rebooted and began to ...
1
vote
0answers
107 views
wget -nd (--no-directories) option not working as expected
I am using wget 1.12 (in msys if it makes a difference) and am trying to mirror a website with the -nd option, since the file and folder names on this site are very long. The docs state that
‘-nd’
...
1
vote
1answer
156 views
wget -nd (--no-directories) option not working as expected
I am using wget 1.12 (in msys if it makes a difference) and am trying to mirror a website with the -nd option, since the file and folder names on this site are very long. The docs state that
‘-nd’
...
1
vote
1answer
582 views
Can't “wget” from my home Debian Lenny server! But the ports are opened!
I have a very strange problem for me.
In this example dns and IP is hiden for security reason.
When I try from putty manager to connect to my Debian and write next command:
debian:~# wget ...
1
vote
1answer
390 views
Crawling a large directory with wget with two links pointing at the same thing
I'm trying to crawl a directory on a website and basically download everything in it. The structure is simple enough(but there are also multiple folders), but there is one thing that makes wget choke ...
1
vote
2answers
281 views
WGet a Page that Requires Logging in
I’m trying to figure out a way to use WGET or a similar tool so that I can schedule a web page to be downloaded regularly as a sort of updating log. The problem is that the page requires that I be ...
