Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to mirror a website that will be taken down soon, the problem im facing at the moment is simple -> mirror whole website1 and all links (will be files/images/similiar) to website2 too, so i got a nice "merged" mirror.

-Therefore the Question would be:

How to do this with wget? Are the other ways to solve this problem (if not possible with wget)?

-Logic Example:

The Website is http://example.org and it will get mirrored (wget -mk). wget should also mirror every content being hosted on http://foo.bar too but nothing else.

share|improve this question

migrated from stackoverflow.com Dec 20 '12 at 17:33

This question came from our site for professional and enthusiast programmers.

up vote 1 down vote accepted

Something like this is what you are looking for:

wget -mk -w 20 http://www.example.com/ --exclude-domains sunsite.foo.edu --domains yahoo.com,google.com
  • m Turn on options suitable for mirroring; time-stamping & infinite recursion depth keeping directory listings.
  • k After the download is complete, convert the links in the document to make them suitable for local viewing.
  • w SECONDS Introduce a delay between accesses to the server.
  • --exclude-domains DOMAIN-LIST Domains that are not to be followed.
  • --domains DOMAIN-LIST A set domains to be followed, a comma-separated list of domains.
share|improve this answer
    
im looking for a way to do --include-domain <LIST> not exclude specific sites. – K1773R Dec 20 '12 at 23:00
    
@K1773R Checkout my updated answer, you may include a list of domains also – X.Jacobs Dec 20 '12 at 23:11
    
--domains was what i was looking for, ty! – K1773R Dec 20 '12 at 23:22

rysnc will mirror the files

rysnc -auvz source destination

the -u flag will skip files that are newer on the destination, so this is probably what you want.

share|improve this answer
    
the second website has no links nor directory listings. – K1773R Dec 20 '12 at 11:35
    
Why does that matter? :) wget -m download the first site, and merge together with rysnc? – Sam Doidge Dec 20 '12 at 11:49
    
first site got tons of links to other sites, but i want only specific ones to be followed (to foo.bar as example) – K1773R Dec 20 '12 at 12:38
wget -p -k http://example.org

The -p will get you all the required elements to view the site correctly (css, images, etc). The -k will change all links (to include those for CSS & images) to allow you to view the page offline as it appeared online.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .