GNU Wget is a free software package for retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

learn more… | top users | synonyms

0
votes
0answers
8 views

gnuwin32 wget Unable to locally verify the issuer's authority

I have just installed gnuwin32 wget and have had some issues when running it. I appear to have issues with https sites... I get a "Unable to locally verify the issuer's authority." error. I have very ...
3
votes
2answers
25 views

How can I download a file the moment it becomes available on linux?

How can I set up something on a Linux machine so that the moment a file on a web server returns something other than a 404 error it will download it. I would like it to stop checking the moment the ...
1
vote
1answer
17 views

wget for FTP end in “connection refused”

I want to copy a FTP directory to my server (Debian) using wget command. I tried : wget -m --user=user --password=passftp://ftp.domain.com I've got this error : connection... failed : Connection ...
0
votes
0answers
23 views

How can I plot cwnd vs. time and queue occupancy vs. time graph of the buffer from wget output?

I had used a smaller buffer for 2 host network h1 and h2 using sudo ./run-minq.sh command. Now at first, I observed the wget from h1 to h2. Next I start iperf and wget simultaneously and observed the ...
0
votes
1answer
32 views

Download the directory structure of a website without its files?

I googled for this but couldn't find an example of how to use eg. wget to download the directory structure of a website without downloading any of its files. At this point, I just need to understand ...
0
votes
1answer
23 views

Downloading with wget using referer + cookie

I'm trying to download a file from a website that requires login. This command retrieves the cookie.txt and the downloaded html shows I'm logged in (ok): (I broke the command in lines for ...
0
votes
0answers
7 views

mirror site behind login

I would simply like to mirror all the pages (millions) on a site behind a login. This is for personal use. I have a login for the site in question but I want to scrape it into a database. I ...
1
vote
2answers
25 views

How to wget a webpage's actual code for a <div> element

I use wget to save a webpage to a file 'annie.html'. % wget -qO- http://www.swagbucks.com/watch/video/523803/annie > annie.html Inside this file is this line: <div ...
-3
votes
0answers
36 views

Script to download specific files from URL into specific folders & rename some of them according to folder name

I would like to do the following : Download specific files inside some specific directories The created download folder should be named after a certain folder from the URL Files should then be ...
0
votes
0answers
17 views

How to make wget IGNORE certain files?

How do I make wget IGNORE certain files? I ask, since it downloads them and deletes them afterwards, since they're not required (they're excluded). But is there an option to IGNORE them BEFORE even ...
0
votes
1answer
33 views

Cannot write to '-' (Broken pipe) OVH server?

I am trying to run this command to install Windows on my OVH VPS and I get Cannot write to '-' (Broken pipe); root@rescue-pro:~# sudo -i wget -O- 'http://mirror.joodle.nl/Windows_Server2012R2.gz' | ...
0
votes
1answer
22 views

Downloading multiple files in parallel with changed names

I'm writing bash script and don’t know how to solve my problem. I want to to download multiple files at the same time , but ... I want to save the files under modified names. I have a variable ...
0
votes
1answer
10 views

Neither wget nor curl downloads this wikipedia api request

I can perfectly download this link and save it as a text file in a web browser environment. But I can't download and save that wikipedia api request using neither wget nor curl. I use wget -O file.txt ...
0
votes
1answer
24 views

Wget download speed fluctuating

So I'm downloading an ISO from the Uberstudent website using the wget command displayed on their website. But the download speed is constantly fluctuating and I am wondering what could be causing ...
2
votes
0answers
138 views

Retrieving from ftp dirs with wildcards

What is the most efficient way to retrieve all relevant data from ftp if the address of said data can be specified with wildcards as below: ...
1
vote
2answers
32 views

DNS resolution through a proxy

I am behind a very capricious proxy (squid) and it seems only http requests made with a particular user-agent are allowed. When fetching google.com from Google Chrome, I can see a HTTP GET on ...
1
vote
1answer
25 views

Bash: return all the characters between the nth occurence of two different strings within a string

In a bash script (on Ubuntu 14.04) I'm running the command: WP055="$(wget -qO - http://alerts.weather.gov/cap/wwaatmget.php?x=CAZ055&y=1)" Within the WP055 variable string there will be an ...
0
votes
0answers
81 views

How to download source video on vimeo

I would like to download several of my Vimeo videos with wget. From my dashboard, I can generate links for each video. I can choose between Standard Def, Mobile and Source. While Standard and Mobile ...
0
votes
0answers
44 views

Fastest possible 1-liner request generator on Linux webserver

Long story short, I want to make the following 1-line script as fast as it can get: tail -f access_log | grep -v 'some_requests' --line-buffered | sed -u ...
0
votes
0answers
24 views

How can I get wget to always download the file, e.g. I don't want “The file is already fully retrieved; nothing to do” [duplicate]

I'm giving a very simple command, such as wget http://www.example.com/updates/Avail.asp -O avail Which downloads a very small file containing just a few integers indicating which updates to my ...
0
votes
0answers
24 views

Run wget in background but still see download status

I am running wget in Windows (Vista) Command Prompt as follows: wget -b -o test.MP4 -r -A .MP4 -nH --cut-dirs=3 http://website.com If I remove -b I really like that the output showing the download ...
0
votes
2answers
55 views

How to download a file from command line?

I tried wget, powershell and betsadmin, but I can't do it. Can someone help me ? The url (which is a live download link if you click on it in a browser) is: ...
1
vote
1answer
33 views

Fetch source file from GitHub using command line tools?

This question is related to a question offsite at FATAL: MemorySanitizer can not mmap the shadow memory. The person who responded said: Do simple tests (e.g. this one) work for you? The link ...
0
votes
0answers
28 views

Finding a text string on a website and all its links

Can you help me understand process of finding a certain text string on a website and all its immediate links? I started delving into wget and curl with help of grep but was unable to come up with ...
0
votes
0answers
21 views

Why might this website not be yielding the same response in wget as it does in Firefox?

I want to fetch instances of this web page via a script which injects variations of the parameters into HTTP POST. Using the Firefox Tamper Data extension, I captured a sample request. Here are the ...
0
votes
0answers
19 views

Make downloads in wget utilize full bandwith

I am using an app called WSUS to download updates for Windows and Office. It works fine, but the problem is that I am on a time-based connection and wget (what WSUS utilizes to download files) does ...
0
votes
0answers
35 views

Clone a GET request-based website

I'm trying to make a static clone of a website because it's running on some ancient CMS that's going to crash any minute now. I've tried using httrack and wget, but none of them work. The problem ...
0
votes
0answers
49 views

Downloading video using command line

I use right click on a button and select Download Linked File As to download a video file into my computer. I'd like to automatize it with scripts, so I opened the source code and copy the link ...
0
votes
0answers
16 views

Command to fetch a list of urls with connection keep-alive but fail on the first error (404/500)

I am trying to reproduce a potential bug in the Jetty web server with regard to serving static files. I need to repeatedly download a list of urls with "Connection: keep-alive" until one of them ...
2
votes
0answers
37 views

msys2 network performance woes

With a fresh install of msys2 32-bit (needed for git-annex) from https://msys2.github.io/ on Windows 7 SP1 64-bit, I'm seeing incredibly slow network transfer speeds on wget, ssh, rsync, etc. For ...
0
votes
3answers
80 views

How to download a file from URL in Linux

Usually one would download a file with a URL ending in the file extension. To download Ubuntu ISO, one would simple wget http://releases.ubuntu.com/14.04.3/ubuntu-14.04.3-desktop-amd64.iso ...
0
votes
0answers
27 views

OSX terminal : log on on a website

I would like to login/ out on a Web site using the terminal, I currently using a appleScript I wrote but was wondering can you use terminal command line instead (not running my script from terminal) ...
0
votes
0answers
41 views

httrack getting tumblr photoposts but not photosets; why?

I'm trying to save tumblr image posts. For a post with one image, like this http://textsfromthepurikyua.tumblr.com/post/124075150094/textsfromthepurikyua-image-cure-heart httrack -n gets that image. ...
0
votes
1answer
32 views

How to wget with pics on www.x.org/yy/index.php

I have an unusual case. I want to use wget to download all pictures on http://www.indyprojectseed.org/2011/index.php and all the other years. How can I do this, with a loop maybe? I am currently ...
0
votes
0answers
70 views

Download a single web page (including images from CSS)

I want to download a single web page including everything needed to view the web page locally (this includes images linked in the css). Example URL i want to download: http://www.example.com/page1 ...
0
votes
0answers
9 views

Indexing file resources from a website

How can I get a list of all (image) resources from a website? Is there a better wait than using wget and parsing the log stream in order to get the list of all images? Worst is, I'm pretty use I ...
0
votes
0answers
30 views

Improper clone of a site with wget

I was trying to clone the homepage of Google with wget as explained here, wget -E -H -k -K -p https://www.google.com/ But I am not able to clone an exact copy of the live Google page. Please refer ...
1
vote
1answer
14 views

How to handle Proxy with Sudo on VM

I have Ubuntu installed on a VirtualBox VM and I am using NAT. The host network requires NTLM proxy authentication to work. I tried export HTTP_PROXY=proxy.mycomp.com:80 & HTTPS_PROXY=$HTTP_PROXY ...
1
vote
2answers
45 views

Wget Bash Script File Not Found

I have the following script: for i in {1..16}; do $wget http://url/$i.jpg -O C:/Users/User/Desktop/Test/ done Saved the file as test.sh and run it within Git on windows. The files are live on the ...
-4
votes
1answer
20 views

How do I take a url and rename it to a .mp4 using wget? [closed]

Need help for a project and have a bunch of URLs that need to be renamed to .mp4 Here are some of them 1280x720.mp4 - 640x360.mp4 - 320x180.3gp
0
votes
0answers
64 views

Slow download over libraries on certain Windows systems

I am using custom libraries for file download (cURL and wget) on Windows client over HTTP from my server. On most of systems the download speed is as it should be - maximum determined by network ...
0
votes
1answer
35 views

Does wget -O always write a zero byte file on failure or can it be a different size?

When using wget's -O flag, the program writes a bunch of zero byte files when it fails. My question is, will the size always be zero? Or, can a non-zero byte file also be created when a download ...
0
votes
0answers
21 views

wget on Windows - same file keeps getting overwritten

I'm using wget on Windows 7. I'm downloading a lot of files such as this: http://stats.grok.se/json/en/200712/Wikipedia:WikiProject%20Video%20games except that I am changing the date each time. The ...
0
votes
1answer
51 views

wget — retrieval of pdf files from online database

I am using wget to retrieve particular pdf files from http://www.aph.gov.au/ I only want to retrieve Hansard files (transcripts of Chamber proceedings). Two scenarios: There is a page where ...
0
votes
1answer
253 views

Bash script not waiting for wget to finish download

I am using the following bash script to merge various gzip archives together: wget -O bluelvl1.gz http://list.iblocklist.com/?list=ydxerpxkpcfqjaybcssw&fileformat=p2p&archiveformat=gz wget -O ...
0
votes
0answers
63 views

Gmail Automation via Bash (To refresh POP Fetch from other accounts)

Gmail has this option to fetch mails from other accounts. I used this feature to create a central mail box. Works good. Gmail periodically polls other mail box over POP. But the frequency is 1 hour + ...
0
votes
1answer
42 views

How to download streamed files with wget?

Sometimes I stumble in files I cannot download with wget. So maybe someone has a hint for me? For example I want to download via the following link: ...
1
vote
0answers
157 views

Wget slow download speed

I get horribly bad download speeds when I try to download files to Banana Pro (similar to Raspberry Pi) with wget. With wget, I get around 1.5-2MByte/s, while on another computer, I'm getting about ...
0
votes
0answers
71 views

WGET not working but CURL works on the same address/host?

WGET give me this error: HTTP request sent, awaiting response... 403 Forbidden 2015-11-23 12:07:06 ERROR 403: Forbidden. If I try the same URL (it’s an internal network Apache server, firewall ...
0
votes
1answer
123 views

How to download all mp3 files linked in a website with wget?

The links in the page 'http://www.examplepage.com/' are to a different page 'http://www.archive.org/download/examplepage/asdf.mp3'. How would I write the wget command? wget -mH -A mp3 ...