Browse > Home / Search Engines, Web-Dev-Design / How Google indexes (spiders) websites

How Google indexes (spiders) websites

July 19, 2006

Google currently is reported to have indexed 8 billion pages and counting. Google utilizes an array of bots A.K.A spiders or crawlers.

Among the specialized bots Google uses include: The web spider Googlebot, the Adsense spider MediaBot, the image spider ImageBot, the AdWords spider AdsBot, the RSS feed spider FeedFetcher-Google, and Googlebot-Mobile spider for mobile devices.

Why is it important for an Internet Marketer to know how Google spiders (crawls) your website?

A search engine crawler is your best visitor. Giving a crawler easy and uninhibited movement in your website is necessary for good search engine rankings.

Your website must be spider (search engine) friendly if you want any traffic from the search engines. A search engine spider does not read your website the way we humans do. The spider reads web-page source code(HTML) that renders your page, therefore ‘bad code’ can be an impediment to the spider, sometimes causing it to give up crawling your website.

Spiders love content(text), therefore a website that is packed with images with no ALT tags to assist the spiders will not be indexed successfully. So when designing your website you must incorporate structural website design principals that elicit search engine friendliness.

According to various surveys, more than 86+% of all people find out about other sites through the search engines. Unless your site appears in the top 30 of the major search engines & directories, you will hardly get any worthwhile traffic. Therefore, Search Engine Positioning is vital to your success on the web! Research shows that 66+% of online marketers find that search engine positioning drives traffic.

A top ranking in the search engines can bring you highly targeted traffic. If someone visits your site after searching for a product or service that you are selling, it means that he/she is interested in what you are selling and hence is a potential customer for you. Hence, search engines send pre-qualified customers to you.

Remember, search engine bots are your most important visitors, you must seduce, ‘open your doors’ and accommodate them in order to gain any measurable success in your Internet Marketing endeavor.

Kim Roach in her article titled: Today’s Google Bots and What They Do - attempts to reveal some of the most important Google spiders, their function, and how they affect you as a web master.

References:

1. SEO (Search Engine Optimization Articles)
2. Articles on Search Engines

Written by James Opiko · Filed Under Search Engines, Web-Dev-Design 

Comments

Got something to say?

You must be logged in to post a comment.