0
votes
1answer
18 views

How to automatically save a list of files with different names?

I have a text file of URLs, each in the form of: http://domain.com/a/b/c/d.jpg I want to download all of these, but save each file under the name: c_d.jpg In other words, for each file, I want to ...
0
votes
0answers
16 views

How can I use Wget / Curl / Plowshare to download from 1fichier.com as a premium member?

How can I download files from 1fichier.com (I am a premium member) with the help of Wget / Curl / Plowshare (Plowdown) or other CLI / command line tool / script?
0
votes
1answer
21 views

Trace web redirection in bash script

I want to make a script to download the image at best resolution (not the preview image) from a deviantART link, as if I clicked the "Download" button. However, it seems that deviantART redirect the ...
0
votes
1answer
40 views

WGET/cron downloading webcamera file

our new town doctor put a web camera to his waiting room for people to see how much crowded it is. As my mom and grandparents go to this doctor, I wanted to download the files periodically and if they ...
1
vote
1answer
35 views

What is the curl equivalent to wget -x?

I want to use curl to save multiple files with the same local paths as would be produced by wget -x (ie, cleaned up versions of the full urls). But I can't find the right option in the curl docs. Is ...
1
vote
0answers
21 views

Does aria2 support write small files in batch?

I'm using aria2 to download 8 million jpg from flickr. Each image is about 100KB. I got a list of urls of these images in a txt file, the format is: ...
0
votes
1answer
60 views

Downloading entire web pages within domain [duplicate]

My computer in on LAN. I'm connected to the local domain at MyDomain. Within that domain exists something like WIKI for a training (http://training-wiki/). I would like to download entire WIKI into my ...
1
vote
1answer
53 views

Mirroring a web site having pages that uses simple JavaScript [duplicate]

There are simple and great web site download tools that allow to create a local mirror of simple sites having no JavaScript infrastructure behind. However, sometimes there cases when a site looks ...
0
votes
0answers
204 views

How can I download private repo from github?

What I want to do is to download private repository archive from GitHub, extract it, remove archive file and copy some directories that are inside downloaded project. I tried to use wget but I cannot ...
0
votes
0answers
59 views

save content and header into different files using wget

I want to get url content and response header data into two file using wget. I see man of wget but i cant find useful method for this. somthing like this: wget -q --output-document=content.html ...
0
votes
1answer
40 views

Wget: Getting Only the WordPress Blog Posts (Not Tag Files etc.)

I've been struggling with Wget to download a readable offline copy of a WordPress.com blog. The blog consists of about 90 posts. In my usage so far, Wget downloads about 2,500 files. The largest ...
0
votes
0answers
15 views

Can One Extract a Hyphenated Filename File From a Server Using Curl?

The images I'm dealing with have file names like part-1.jpg, object-a.jpg, Aspect-0.jpg. What I'm wondering is what I put in place of ???: curl -O http://server/[???].jpg
-2
votes
1answer
60 views

Download files in bulk, but confirm each download

I wish to download all pictures from an album. The album is an HTML page hosted on the Internet, which has both links to some other stuff in the site and thumbnails, which link to the image viewing ...
2
votes
1answer
65 views

wget download from mirror download links

I'm trying to download using wget a tar.gz. The file is redirected based on the url parameters, so all downloads are redirected automatically. for example: wget ...
0
votes
1answer
131 views

Does wget have a download history?

I have used wget to download several compressed packages(in a failed attempt to manually emulate apt-get, if that is of any significance), and I don't remember the packages' locations. Is it possible ...
0
votes
1answer
215 views

wget crawling search results of news website

I am trying to crawl the search results of a news website using wget. The name of the website is www.voanews.com. After typing in my search keyword and clicking search, it proceeds to the results. ...
0
votes
1answer
33 views

use wget to output everything retrieved into a single txt file

Is it possible to do something like this? wget --recursive --output-document=theEntirePagestuffedIntoATextfile.txt www.example.com And in words what it should do is: "download the entire website ...
-1
votes
1answer
4k views

Download ALL Folders, SubFolders, and Files using Wget

I have been using Wget, and I have run across an issue. I have a site,that has several folders and subfolders within the site. I need to download all of the contents within each folder and subfolder. ...
1
vote
1answer
552 views

wget execute script after download

I need every time you complete a transfer program wget this run a script. this is my full command line: wget -i urls-to-downloads.txt My problem is that I want to go subtracting it downloads the ...
1
vote
2answers
1k views

Download entire directory from a website using Powershell

I have been using the following command to pull back files from a web server and store them on my local machine: $client = New-Object System.Net.WebClient ...
1
vote
1answer
72 views

How to resume wget mirroring from a different location?

I am out of space in my home folder , where iam currently mirroring this site for the pupose of a cygwin installation in my windows machine using wget from my linux machine. i am aware that i can ...
0
votes
0answers
121 views

httrack, wget et al: Follow all urls, but only download specific ones

I'd like to download all pages of a website where the URL matches a pattern X. For those pages, I want to download all requisites (CSS, Images etc), regardless of where they are stored (even if that ...
0
votes
0answers
64 views

Files not downloadable via Browser but with Wget. Why?

Recently I came across an issue on a client machine where the machine was not able to download some files. We tried to download the files via Browser, but it did not generate any response. However, ...
0
votes
1answer
80 views

Download all images at distance 2 using Mac OS X

I need to download all the images from a website on my computer. The images are not listed all in one page. Instead I have an index page that links a set of pages (page1.html, page2.html, ...) each ...
0
votes
1answer
298 views

Using Wget to download single images from a website using the the img src URL where img class=“decoded”

I want to get single images off a website. The website has photos. Clicking on the photos leads to another page. I want to get the photos from the img src tag in the html. I already have unix code ...
1
vote
2answers
94 views

Is it possible to download only a certain percentage of a file wit Kget (or wget)?

I need to download some large files (ISOs of some Linux distros) and I'm connected to the internet with a mobile broadband connection, which has a limited amount of daily total data transfer size. ...
0
votes
1answer
78 views

Downloading a website using wget, is download speed more important?

I'm trying to download a favourite website with lots of pictures, its around 10 gigabytes in size. I'm using sitesucker right to download it and I've noticed that although I have a 10Mbps download ...
2
votes
1answer
159 views

wget downloads empty files even with proper referer and cookies

I'm trying to scrape Brenda Enzymes for a bunch of molecule files with a script. If I just download by file URL (it's a PHP redirect), I get a 0Kb file, so I think they're checking for cookies and/or ...
0
votes
2answers
141 views

Cannot write to “HAIB_T47D_FoxA1.tar.gz” (Input/output error)

I am trying to download a dataset (bigger than 1 GB) using wget: wget http://cistrome.dfci.harvard.edu/MACSNatureProtocol/HAIB_T47D_FoxA1.tar.gz However, I always get the following error message: ...
1
vote
1answer
378 views

How to download YouTube videos via HTTPS [duplicate]

I am unable to access YouTube on http. But I can access the same on an https. When I executed the command wget https://www.youtube.com/watch?v=StJLvbPIvTw it returned the following response:- ...
0
votes
0answers
132 views

wget truncated my files

wget -c -b --content-disposition 'http://example.com/somepath.mp4' I had some connectivity problems so I had to rerun this command for a few remote files. According to wget's man page, it will ...
1
vote
3answers
2k views

wget - Many URL's in .txt file - download and save as

I have 2000 URLs in excel file. The URLs are in the first column and in the second column there are names for the files downloaded from URL in the first column. I can copy that and paste to .txt file ...
1
vote
1answer
158 views

How to download multiple urls using wget using a single command?

I am using following command to download a single webpage with all its images and js using wget in win7: wget -E -H -k -K -p -e robots=off -P /Downloads/ ...
1
vote
0answers
317 views

Create a complete local copy of a website using Wget

OK, here is what I need : I'm on Mac (Mac OS X 10.6.8) I want to completely mirror a website on my hard drive (this is what I'm using as a test) I want all images and prerequisites there, so that ...
0
votes
0answers
26 views

Where does wget put downloaded files on Windows? [duplicate]

Possible Duplicate: Where on my computer did Wget download this image? I am trying to download a file using wget in Windows 7. I used the following command to download the file: C:\Program ...
1
vote
0answers
114 views

the right options to traverse/download the pages/directories of a subdomain

Let's suppose exist a site with the following directories (subdomain) index.php |-sub1 |-index.php |-sub1sub1 |-index.php |-other.php |-sub1sub1sub1 |-sub2 |-index.php |- ...
3
votes
1answer
3k views

How to get past the login page with wget?

I am trying to use wget to download my private github pages, but I cannot get past the login screen. How do I send the login/password using post data on the login page and then download the actual ...
0
votes
1answer
60 views

Transfer a file from Server A to B, using your PC as terminal then turning off your PC?

I need to copy server-a.com/file.zip to server-b.com. Let's assume the file is really big. Using my home PC, I SSH to server-b.com and type wget server-a.com/file.zip How do you do it so that the ...
0
votes
1answer
91 views

What happens if wget runs out of disk space?

I'm about to run wget to mirror a large website for offline viewing. I'm worried the site might be too large for my hard drive and my download getting screwed up. Is there any way for me to figure out ...
2
votes
3answers
611 views

download a series of files automatically using command line/wget

I have a case that I would like to trigger an automatic download for a list of 114 file (recitation) for each reader, for example if i want to download the recitations for a reader called abkr, the ...
2
votes
1answer
209 views

wgt downloads all files except for the images i want

i have trouble using wget to download images from a gallery. as start i use the overwiew page. it has thumbs that link to the individual pages with the large images. here is the script i use: wget ...
1
vote
1answer
1k views

How do I remotely fetch files from redirected URLs from a terminal?

I want to fetch a tarball of this python library from terminal. https://github.com/simplegeo/python-oauth2/downloads However, I cannot simply call wget ...
4
votes
4answers
594 views

How to automate downloading files?

I got a book which had a pass to access digital versions of hi-res scans of much of the artwork in the book. Amazing! Unfortunately the presentation of all the these are 177 pages of 8 images each ...
3
votes
2answers
1k views

Wget - if / else download condition?

I want wget to prefer a certain filetype over another, if the files have the same basename. For example: if foo.ogg available, don't download foo.mp3 the way i use wget so far to ...
1
vote
1answer
5k views

Downloading only images using curl or wget?

UPDATED: I've found using this Bash script fixes the problem of having GIF files with a .jpg extension. I'm attempting to download images from a forum who's url uses the following format: ...
2
votes
4answers
4k views

Copy entire website including PHP and SQL?

Is it possible when copying an entire website and template using wget (httrack or other program) to also get the PHP code and the SQL database? What software would I need to copy/mirror a site like ...
2
votes
1answer
1k views

wget fails but opening in browser succeeds. How?

I was trying to download a pdf from the intel website at this address. When I open it through the browser(google-chrome) there seems to be no problem in viewing it or storing it on the my hard disk. ...
2
votes
1answer
476 views

Resuming Cancelled Download with WGET

Half-way through a 4GB download I accidentally rebooted the system. Upon loading back up, I had the sheer horror of realising I cancelled the bloody download because of it. I've booted my Ubuntu VM ...
0
votes
1answer
721 views

How to configure wget to mass file download keeping the original filename

I have some problem configuring wget right for my needs, i have a file, named images.txt, containing a list of urls to required images, the command for wget is simple: wget -i images.txt however ...
2
votes
1answer
920 views

Downloads stall and won't resume

I have a shaped 384kbps ADSL connection (entry-level "broadband") with an ISP, which runs over the compulsory analog voice line provided by the only telephone provider in the country. "Shaped" means ...