Questions tagged [robots.txt]
The robots.txt tag has no usage guidance.
8
questions
0
votes
0
answers
33
views
Index a whole website that has blocked Google?
I tried to do a site:site.com [search terms] on Google but site.com has blocked Google from indexing it via its robots.txt. How can I get around this? Can I download and index the whole site myself ...
0
votes
1
answer
170
views
Can ROS be played in Windows 10, 64bit?
I try to install ROS(robot operating system) in my computer with windows 10, 64bit.
Is it possible?
And is there any process to do it?
1
vote
1
answer
152
views
How can we know which URLs can be crawled as robots.txt tells if we don't know to which folder a URL belong to?
I'm going to code a web crawler but before I want to know what is going to be possible to crawl.
Tell me if I'm wrong, but in robots.txt websites indicate folders not URLs that can and can't be ...
-1
votes
1
answer
450
views
Googlebot blocked by robots.txt [closed]
So recently I've tested my site with Google mobile-friendly test and the main loading issue was "Googlebot blocked by robots.txt"
My robots.txtdoes allow Google bot I think?
What do you think guys? ...
-1
votes
1
answer
134
views
How to to prevent Google from indexing
We have setup one of web site on server and site build in php and symfony framework, As my requirement is to prevent Google from indexing and below of my robot.txt and Is prevent using .htaccess?
...
2
votes
2
answers
179
views
What software is needed for membership websites and how can they still be indexed by Google [closed]
I notice that in some cases paywalled news articles seem to have been indexed by Google because excerpts from the story appears in the search hit.
However, when I go to these web sites using a ...
1
vote
1
answer
680
views
Apache wont start, port 80 in use by a system process, found baiduspider
Ok so I have uninstalled IIS on my windows server and decided to try Xampp to host my domains. Port 80 is in use and I have tried all of the fixes that I have came across for the past 2 days. I was in ...
4
votes
0
answers
879
views
Is automated web site access legal? [closed]
Many web sites include in their terms of service things about automated access being prohibited. One example is in ebay's robots.txt file:
The use of robots or other automated means to access the ...