My website is being scraped by archive.today. This site is similar to the reputable archive.org, but archive.today does not obey robots.txt, spoofs a generic user-agent and tried to hide its IPs. The site then proceeds to regurgitate your content and index it is search engines.
-
On Webmasters: Deny access to Archive.is– unorDec 26 '15 at 19:12
Add a comment
|
You can block the crawler by blocking the following IP ranges:
78.46.174.128/27 & 108.61.0.0/16 & 5.101.96.0/21
Once you block these, trying to index a page with archive.today should result in an endless loop on their site.
You can do it in iptables like:
sudo iptables -A INPUT -s 78.46.174.128/27 -j DROP
sudo iptables -A INPUT -s 108.61.0.0/16 -j DROP
sudo iptables -A INPUT -s 5.101.96.0/21 -j DROP
And I imagine Apache and ngix have a way to do something similar.
archive.today defeated:
If you do not see this, they have probably got some new IPs.