Browse > Home / Archive: 27. November 2006

Are you blocking IP addresses of Search Engine Spiders in your .htaccess File ….inadvertently?

November 27, 2006

How often do you check your Server Error Logs?

A prudent webmaster needs to keep track of the various IP addresses belonging to — at least — the major search engines: Google, Yahoo and MSN.

You might be blocking vital search engine spiders from indexing your site and hence denying you the traffic that you deserve.

The best place to look for such anomalies is in your server logs, notably the Error Logs.

What makes this task even more difficult is that like all other IP addresses, quite a chunk are dynamically generated, sometimes reassigned or abandoned totally.

Nevertheless, I can’t stress more, the importance of matching your error logs with search engine IP addresses every now and then, to ensure that you are not blocking any — remember, if you block out Google or Yahoo Bot IPs, they won’t send any traffic to your website — something that can be very costly.

Most web hosting companies provide a readable error log within your websites control panel. Alternatively you can setup a Cron Job to email your error log in plain text format, which you can then read in any test editor.

Using your server logs you can also pin-point script errors, internal 500 errors etc., and also determine whether your .htaccess file is operating as it should.

Written by James Opiko · Filed Under Marketing, Search Engines, Web-Dev-Design | Leave a Comment