How to Get Your Website Listed in the Search Engines

Search engines are the librarians of the internet world. Every day they send out their interns, called Robots, into cyberspace to forage for information. Each piece of information that is returned to the library is filed in a digital catalog. To access any piece of information in this catalog, all one needs to do is go to their library of choice (Google, Yahoo, MSN or ASK), type in a few search words and wait while the librarian (the search engine) serves up sites matching the query.

As a webmaster, you may be wondering how to get your brand new, still in the bubble wrap, website listed in these giant cyber catalogs. Well, there are two ways robots find new websites. Either by following links from other sites or from the queue of direct submissions.

It has been my experience that sites found via links from other, trusted, sites are indexed faster and avoid the sandbox effect than when submitted directly to the search engine. I call this the referral effect and the keyword here is “trusted”. A trusted site is a site that’s been around for a while, uses no or very little reprinted content and has accumulated a good quantity of quality links itself. The trick here is finding a site that fits all of these qualities, that is active and does not use the no-follow link attribute.

A link from these sites can gained in several ways. You could get added to a blogroll or link page. You could purchase advertising on the site or you could post comments in the area designated for them. To view whether or not a site uses the no-follow link attribute you can either view the source or install an extension such as SearchStatus for Firefox by Quirk.

The second way to get indexed is by direct submission. Though you can submit to every Search Engine you come across most search engines get their results from the big three; Google, Yahoo and MSN. So directly submitting to them is your best bet.

Google: http://www.google.com/addurl/?continue=/addurl

Yahoo: http://search.yahoo.com/info/submit.html

MSN: http://search.msn.com/docs/submit.aspx

Enter your web address and then wait. It usually takes a few days before the robots show up at your site to index it.

Additionally, all of the search engines support sitemap submission. A sitemap is a document that lists all of the content to be found on your website and makes it easier for the robot to fully catalogue your site.

Go to:

Google Webmaster Tools

MSN Webmaster Tools

Yahoo Site Explorer

As a bonus, you can also ping the search engines to notify them when you have new content and therefore entice the robot to come visit your site again.

Yahoo Ping

http://search.yahooapis.com/SiteExplorerService/V1/ping?sitemap=
http://www.yoursitemaplocation.com

Google Ping

http://blogsearch.google.com/ping?name=You+Site+Name&
url=http://www.yoursiteaddress.com&changesURL=
http://www.yoursitemapaddress.com

MSN Ping (Note: Moreover is MSN’s ping service)

http://api.moreover.com/ping?u=http://www.yoursitemapaddress.com

If you use WordPress, you can just set it up to automatically ping the appropriate servers by going into your dashboard, click on Options and then Writing. At the very end of the page will be a box that says Update Services. Just add the above to the list. For an entire list of places you can ping visit Update Services.

You can also use your Robots.txt page to help robots find your sitemap when they visit. Robots will usually hit this file first when visiting your site for instructions on what they can and cannot index on your site. To direct them to the sitemap, add the following code to your robots.txt file:

Sitemap: http://www.yoursite.com/yoursitemapaddress.xml

Getting your website indexed is only the first step towards doing well with the search engines. In an upcoming article, we will talk about how to optimize your website so that it ranks well with the search engines so that they can actually bring you enough traffic to make you the internet rockstar you were meant to be.