Tag Info

New answers tagged

0

My best guess is the host isn't routable meaning it either doesn't have a public IP or there is some VPC configuration affecting the communication since you state wget doesn't work either.


1

A pre-built binary 32-bit version of wget for Windows (currently version 1.11.4-1) can be found on SourceForge at: http://gnuwin32.sourceforge.net/packages/wget.htm You have the choice of downloading the wget package as a setup program or a zip file. As described on the SourceForge download page, here is the difference: If you download the Setup ...


0

Applying the provided command to an exemplary website worked out on my Ubuntu 12.04 machine using wget version 1.13.4: $ wget -nd -e robots=off --wait 0.25 -r -A.pdf http://www.cvpapers.com However, it takes some time until the first pdf is downloaded, since the site provides many html-files, which need to be skipped. Leaving out the "--wait 0.25" option ...


0

The solution with wget would be something like: wget --header="Authorization: token <OAUTH-TOKEN>" -O - \ https://api.github.com/repos/<owner>/<repo>/tarball/<version> | \ tar xz --strip-components=1 && \ cp -r <dir1> <dir2> ... <dirn> <destination-dir>/ Notes: --strip-components=1 will ...


0

I've found this one: If you request gzip'ed content (using the accept-encoding: gzip header, which is correct), then it's my understanding that wget can't then read the content. So you will end up with a single, gzipped file on disk, for the first page you hit, but no other content. i.e. you can't use wget to request gzipped content and to recurse the ...


1

You can try with a workaround. The milestons are: Download the full list of files and store it in file Change the final part of the filenames Download only the files with modified name. You need a file filter.awk similar to this /^--.*-- http:\/\/.*[^\/]$/ { u=$3; } /^Length: [[:digit:]]+/ { print u; } Step by step: wget -r -np --spider ...


0

From the wget manual (1.15): Actually, to download a single page and all its requisites (even if they exist on separate websites), and make sure the lot displays properly locally, this author likes to use a few options in addition to ā€˜-pā€™: wget -E -H -k -K -p url Also, robots.txt may be blocking some content, so add -e robots=off So you get: ...


1

wget by default have something like "Wget/1.11.4" user agent. Site probably forbidden this user agent. So you should change user agent to browser like with "--user-agent" command: --user-agent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0"



Top 50 recent answers are included