Robot Friendly Web Sites
Why make my Web site Robot friendly?
Why do some web sites appear higher up the list of results from search engines than others? Why are some pages listed when they seem to be almost unrelated to the query? Search engines apply a number of rules to determine which pages are most relevant to a particular query. Some of these you have to work hard to achieve, such as sites with large numbers of links to them are favoured by some engines. As are links that other people have chosen most when presented with results.
Similarly, whilst you can pay money to people like "Submit It" to make sure your web site is known to as many search engines as possible, most people use the large search engines (Lycos, Yahoo, Altavista), and most people never get past the first page or two of results. So your ranking is everything for the serendipitous discovery of your web site by potential clients.
The one thing you can do, that is quick, easy, and effective is to ensure that your site is robot friendly.
The programs which index the Internet are called robots. Other programs which automatically research a topic are called Spider's, crawlers, agents, web-bot's and other exotic names. Your web site should try to be as friendly to these programs as it is to human surfers.
How Do I make my site Robot friendly?
-
You could pay one of the many web ranking firms to do it for you
-
Well you could go learn all about every major search engine, and automatic search tool
-
You could check out what the highest ranking sites do
-
Or you could read the 'source' for http://www.eighth-layer.com/, we are still working on improving the ranking.
As a minimum your 'home page' should indicate in hypertext meta tags:
-
The language, or languages your site is in (English)
-
The area or category of activity you are in (Computers)
-
The keywords that you would like your site indexed by, including if relevant geographical names - the Internet is big - (Exeter, Devon, UK, Hewlett, Packard, Oracle, UNIX, DNS, security, firewalls)
The icing on the cake is to indicate which area's of your site should be indexed. Some web information is transitory, and allowing it to be indexed will just result in lots of '404 Page not found' errors, irritating potential clients.
The keywords meta-tag is chiefly recognised only by Altavista. The ranking of your site will be determined by a number of factors, such as relevant links to your site, and occurence of keywords. Some search engines will also index words that occur in comments, which can be useful to supplement pages which whilst relevant do not have much visible text.