Posted: Wed Jan 23, 2008 6:22 pm Post subject: Disable Seach Engine Listings
Search engines use web crawler (spider/robot) which browses WWW in an automated manner to provide up-to-date data. But what if one doesn’t want to get listed? Yes, “robots.txt” comes into play when you have to set your own trend whether to allow bots or prevent your site from being crawled!
Place robots.txt in your root directory with the following parameters and it will keep all robots out:
Quote:
User-agent: *
Disallow: /
An alternate way to prevent robots from indexing your page is by placing <META> tag in the <HEAD> section of your page. To block all robots, use:
Code:
<META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”>
To allow other robots to index except Googlebot, use the following tag:
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
