2

There are simple and great web site download tools that allow to create a local mirror of simple sites having no JavaScript infrastructure behind. However, sometimes there cases when a site looks simple (and even is old enough to be a real "web 1.0" site), but none of such download tools are able to make a mirror of it. I've tried to make a local mirror for this fan site (for example, it has some transcriptions that cannot be found elsewhere, and some more rare stuff) using HTTrack Site Copier, Free Download Manager, and wget. All of them made simple mirrors lacking a lot of content. I saw some scripts at those pages, and probably that prevents from mirroring.

How can I mirror a web site with simple JS enabled? Perhaps, there's a web browser-driven solution that could support JS out of box (any browser extension, etc)?

Thanks in advance.

7
  • Apologies, but what do you mean by mirror a website?
    – Dave
    Apr 4, 2014 at 10:20
  • 1
    @c0dev not sure if the possible duplicate is able to help, because that solution relies on wget. I can't make wget work to satisfy my needs. However, I've seen wget.addictivecode.org/FeatureSpecifications/JavaScript in that question, and I'll check it later. Apr 4, 2014 at 10:25
  • @DaveRook I mean "to make an exact (more or less) copy of a web site". Please sorry if my English is not fine. Apr 4, 2014 at 10:26
  • Ah I see - no, your English is great, I just wasn't sure. Why not just try saving the site to your computer? In IE you can save entire site
    – Dave
    Apr 4, 2014 at 10:30
  • @DaveRook thank you. :) As far as I know, all major browsers can save single webpages only, but not entire web sites. The problem with that site is that it uses, as far as I see, some JavaScript to load content dynamically. I just want to make a recursive copy of the site (like web crawlers do), but unfortunately none of those tools can do that (at least for what I tried) in this case. Apr 4, 2014 at 10:36

1 Answer 1

0

In this particular case I've ended up with the following bash script:

#!/bin/bash

DOWNLOAD="wget -m -p -E -np -k"
SITE="http://homepage.tinet.ie/~themma"

$DOWNLOAD $SITE/
$DOWNLOAD $SITE/songs/
$DOWNLOAD $SITE/songs/songs.html
$DOWNLOAD $SITE/songs/disco.html
$DOWNLOAD $SITE/links/
$DOWNLOAD $SITE/other/
$DOWNLOAD $SITE/tour/

for i in `seq 1 8`;
do
    $DOWNLOAD $SITE/images/bar_0$i.gif
    $DOWNLOAD $SITE/images/bar_0$i-over.gif
    $DOWNLOAD $SITE/images/bar_0$i-bar_03_over.gif
done

for i in `seq 1989 2003`
do
    $DOWNLOAD $SITE/images/$i.gif
done

I couldn't simulate JavaScript behavior, sure, but since the site is extremely simple, that's not much to analyze it to write a wget-based shell script. It's a little tricky, but it works. Thanks everyone for suggestions.

I don't mark this answer as the best one, because my answer implements a particular case only. So any ideas regarding the more general case is really welcome (any "intelligent" cmd-line tools, browser extensions, etc).

Not the answer you're looking for? Browse other questions tagged .