0

I'm trying to backup a website that contains pages with a tag like this one:

<a id="12379-video"></a>
</div>

<script>
    jwplayer("12379-video").setup({
          file: "http://xyz.cloudfront.net/abc/moviename.mp4",
          flashplayer: "/sites/all/jwplayer/player.swf",
              stretching: "exactfit",
          height: 480,
          width: 640    });
</script></span>
  </div>
  1. Does wget extract the url and follows it? (and what are the right parameters to give to the command)

  2. What is exactly the mechanism wget uses to parse the file is retrieving to extract the urls?

  3. Is there any way to tell the parser to consider that tag too? (using regular expressions or similar)

  4. In case it's not possible, what other strategies do you suggest? (scripting with grep or others using bash)

1
  • 1
    Why are you making backups with wget? If it's your webserver, log in to it to do your backups. If it's not your webserver, you're not doing a "backup". Oct 18, 2015 at 12:16

1 Answer 1

0

There is -p/--page-requisites parameter (with -r together) which would download the most external URLs necessary to properly display a given HTML page (unless they're excluded in robots file).

More about this can be read in the manual (man wget):

It's worth knowing that Wget's idea of an external document link is any URL specified in an <A> tag, an <AREA> tag, or a <LINK> tag other than <LINK REL="stylesheet">.

Since Wget does not ordinarily distinguish between external and inlined documents, one is generally left with "leaf documents" that are missing their requisites.

Please note that Wget has an internal table of HTML tags/attribute pairs that it considers when looking for linked documents during a recursive retrieval. To add one, try using --follow-tags=list (comma-separated list), opposite of --ignore-tags=list.

This list of tags is probably defined in html-url.c and it goes as follow:

/* For tags handled by tag_find_urls: attributes that contain URLs to
   download. */
static struct {
  int tagid;
  const char *attr_name;
  int flags;
} tag_url_attributes[] = {
  { TAG_A,              "href",         ATTR_HTML },
  { TAG_APPLET,         "code",         ATTR_INLINE },
  { TAG_AREA,           "href",         ATTR_HTML },
  { TAG_BGSOUND,        "src",          ATTR_INLINE },
  { TAG_BODY,           "background",   ATTR_INLINE },
  { TAG_EMBED,          "href",         ATTR_HTML },
  { TAG_EMBED,          "src",          ATTR_INLINE | ATTR_HTML },
  { TAG_FIG,            "src",          ATTR_INLINE },
  { TAG_FRAME,          "src",          ATTR_INLINE | ATTR_HTML },
  { TAG_IFRAME,         "src",          ATTR_INLINE | ATTR_HTML },
  { TAG_IMG,            "href",         ATTR_INLINE },
  { TAG_IMG,            "lowsrc",       ATTR_INLINE },
  { TAG_IMG,            "src",          ATTR_INLINE },
  { TAG_INPUT,          "src",          ATTR_INLINE },
  { TAG_LAYER,          "src",          ATTR_INLINE | ATTR_HTML },
  { TAG_OBJECT,         "data",         ATTR_INLINE },
  { TAG_OVERLAY,        "src",          ATTR_INLINE | ATTR_HTML },
  { TAG_SCRIPT,         "src",          ATTR_INLINE },
  { TAG_TABLE,          "background",   ATTR_INLINE },
  { TAG_TD,             "background",   ATTR_INLINE },
  { TAG_TH,             "background",   ATTR_INLINE }
};
1
  • this answers point 2 and 3. AFAI understand, the answer to point 1 is no. for point 4, i will try with grep to extract the links and i will pass them to wget again. I'm wondering if this is related to parsing javascript in wget for dynamically generated content. Oct 18, 2015 at 17:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .