Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [robots.txt]

The tag has no usage guidance.

-1
votes
1answer
150 views

Googlebot blocked by robots.txt [closed]

So recently I've tested my site with Google mobile-friendly test and the main loading issue was "Googlebot blocked by robots.txt" My robots.txtdoes allow Google bot I think? What do you think guys? ...
-1
votes
1answer
100 views

How to to prevent Google from indexing

We have setup one of web site on server and site build in php and symfony framework, As my requirement is to prevent Google from indexing and below of my robot.txt and Is prevent using .htaccess? ...
2
votes
2answers
139 views

What software is needed for membership websites and how can they still be indexed by Google [closed]

I notice that in some cases paywalled news articles seem to have been indexed by Google because excerpts from the story appears in the search hit. However, when I go to these web sites using a ...
1
vote
1answer
351 views

Apache wont start, port 80 in use by a system process, found baiduspider

Ok so I have uninstalled IIS on my windows server and decided to try Xampp to host my domains. Port 80 is in use and I have tried all of the fixes that I have came across for the past 2 days. I was in ...
3
votes
0answers
412 views

Is automated web site access legal? [closed]

Many web sites include in their terms of service things about automated access being prohibited. One example is in ebay's robots.txt file: The use of robots or other automated means to access the ...