2

I want to mirror a website that will be taken down soon, the problem im facing at the moment is simple -> mirror whole website1 and all links (will be files/images/similiar) to website2 too, so i got a nice "merged" mirror.

-Therefore the Question would be:

How to do this with wget? Are the other ways to solve this problem (if not possible with wget)?

-Logic Example:

The Website is http://example.org and it will get mirrored (wget -mk). wget should also mirror every content being hosted on http://foo.bar too but nothing else.

|improve this question

migrated from stackoverflow.com Dec 20 '12 at 17:33

This question came from our site for professional and enthusiast programmers.

1

Something like this is what you are looking for:

wget -mk -w 20 http://www.example.com/ --exclude-domains sunsite.foo.edu --domains yahoo.com,google.com
  • m Turn on options suitable for mirroring; time-stamping & infinite recursion depth keeping directory listings.
  • k After the download is complete, convert the links in the document to make them suitable for local viewing.
  • w SECONDS Introduce a delay between accesses to the server.
  • --exclude-domains DOMAIN-LIST Domains that are not to be followed.
  • --domains DOMAIN-LIST A set domains to be followed, a comma-separated list of domains.
|improve this answer
  • im looking for a way to do --include-domain <LIST> not exclude specific sites. – K1773R Dec 20 '12 at 23:00
  • @K1773R Checkout my updated answer, you may include a list of domains also – X.Jacobs Dec 20 '12 at 23:11
  • --domains was what i was looking for, ty! – K1773R Dec 20 '12 at 23:22
0

rysnc will mirror the files

rysnc -auvz source destination

the -u flag will skip files that are newer on the destination, so this is probably what you want.

|improve this answer
  • the second website has no links nor directory listings. – K1773R Dec 20 '12 at 11:35
  • Why does that matter? :) wget -m download the first site, and merge together with rysnc? – Sam Doidge Dec 20 '12 at 11:49
  • first site got tons of links to other sites, but i want only specific ones to be followed (to foo.bar as example) – K1773R Dec 20 '12 at 12:38
0
wget -p -k http://example.org

The -p will get you all the required elements to view the site correctly (css, images, etc). The -k will change all links (to include those for CSS & images) to allow you to view the page offline as it appeared online.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.