Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Whilst creating a site mirror using wget 1.12 on Ubuntu links with a rel attribute set are not downloaded:

 <a href="link" rel="tag">text</a>

Rel="tag" is a microformat (By adding rel="tag" to a hyperlink, a page indicates that the destination of that hyperlink is an author-designated "tag" (or keyword/subject) for the current page).

My WordPress theme uses this for link to tags, so 99% of the site is ignored.

Edit: it turns out all my permalinks use rel="bookmark" and are skipped as well.

I'm using the following wget command (this ignores robots.txt and also follows nofollow links):

wget -mkp -e robots=off http://site

How do I make wget follow links with rel set?

share|improve this question
    
did you try it with --follow-tags=rel already? – JohannesM Mar 23 '12 at 10:16
    
@JohannesM Manual says: "If a user wants only a subset of those tags to be considered, however, he or she should be specify such tags in a comma-separated list with this option. " your answer would only follow rel tags, which don't exist on the page. --follow-tags does not add to the internal list of tags/attributes to follow but replaces it. And no --ignore-tags= doesn't work either.. – svandragt Mar 23 '12 at 10:20
up vote 3 down vote accepted

I compiled wget 1.13 from source and that fixes the issue (I think it's this line even though I'm not talking about CSS links): Parsing links from CSS files, and from CSS content found in HTML style tags and attributes):

cd /tmp
wget ftp://ftp.gnu.org/gnu/wget/wget-1.13.tar.gz
gunzip < wget-1.13.tar.gz | tar -xv
cd wget-1.13
./configure --with-ssl=openssl
make
sudo make install
mkdir ~/bin
sudo echo "export PATH=$PATH:~/bin" >> ~/.bashrc
cp /usr/local/bin/wget ~/bin
share|improve this answer
    
you should accept your own answer :) – JohannesM Mar 23 '12 at 11:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .