Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've got a personal website I wish to lock down to everyone else except me. I've been using an .htaccess file for a bit and quite like it (they don't even see a webpage, just the login box), but will this protect me from everything? And can search engines still scan me?

I'm thinking of adding password lists here and I want them to be locked down.

share|improve this question
Can't help you if you don't mention what exactly you are doing with .htaccess, or post it. You can put a lot of directives in there, and you haven't even told us which version of Apache (or other) you are using. Can you fill in those details? Please see the FAQ here: Also, you will likely get a better response over at – Brad May 30 '11 at 15:12
support to move to – bubu May 30 '11 at 15:14

In terms of blocking search engines via user-agent string, that is fairly trivial. In terms of blocking users by their user-agent string, forget about it, it's a waste of time.

If you are setting up a basic authentication login, be aware that these are susceptible to brute force attacks and passwords are sent in plain text. I would recommend using SSL if you are going to set this up for some added security. You may be interested in ModSecurity as well if you are concerned about brute forcing.

I would recommend posting your .htaccess file as well before putting it in place so it can be evaluated for any problems.

share|improve this answer
Are you getting confused between robots.txt and .htaccess? – PriceChild May 30 '11 at 15:46
@Price, no:… . Crawlers can also send different user agents which can be annoying. – John T May 30 '11 at 16:06
Right... but nohat says "they don't even see a webpage, just the login box". Suppose we do need to know more about the intended purpose and his current .htaccess. – PriceChild May 30 '11 at 16:17
Bah sorry but my main point was that "most popular search engines will respect the .htaccess file" - it isn't about the search engines 'respecting' it. That file is read by the web server and acted upon. If a search engine bot is masquerading as a standard firefox browser then well who's to say it isn't. I think you should have less emphasis that its the search engine's responsibility. .htaccess is a set of rules. – PriceChild May 30 '11 at 16:19
@Price poor choice of words, I agree. Modified. – John T May 30 '11 at 17:11

An .htaccess file is a set of instructions to your web server. If you tell it to require authentication before displaying pages, this will apply to everyone, search engines included (and only you have the password).

Make sure that your password file is not in a publicly-accessible directory. You can use this directive to completely block access to files starting with ".ht", although if your entire website is password-protected, this shouldn't be an issue.

<FilesMatch "^\.ht(.*)$">
  Deny from all

Note that if your connection is not SSL-encrypted, this setup will not be entirely secure. Anyone between you and the web server can intercept your connection and sniff your password - for example, if you are using public wi-fi at a coffee shop. However, consider how much security you need, because SSL can be complicated and expensive to set up. For practical purposes, an .htaccess file should be sufficient to keep unwanted visitors away from your website.

P.S. Use HTTP Digest authentication instead of Basic - it's a little more secure.

share|improve this answer
SSL is "complicated and expensive"? O.o – Bacon Bits May 30 '11 at 16:04
@Bacon I have to pay my hosting provider for a dedicated IP, and although StartSSL offers free certificates, it took me forever (plus a few support calls) to set up. SSL might not be worthwhile; it depends on how much security you need. – user775598 May 30 '11 at 16:06
I guess I'm just used to self-hosted systems with self-signed SSL, or hosted and purchasing an SSL cert for ~$10/yr (or getting one from our own root CA for work purposes). I'm not sure why you required a static IP for that, however. SSL doesn't require that. – Bacon Bits May 30 '11 at 16:18
@Bacon Basically, shared hosting providers (likely what nohat uses) use the Host: header to determine which website a client is requesting. However, this header isn't send until the connection is established, so a certificate would have to be presented before the server knows which one to select. The SNI extension, which fixes this, isn't widely deployed. []. BTW, StartSSL [] offers free Class 1 certificates. – user775598 May 30 '11 at 16:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .