Wednesday, November 4, 2009

Denied By Robots.Txt and How to Fix It

This tutorial shows you how I fix the denied by robots.txt problem that I found in Google webmaster tool. One of my blog shows that it has denied Googlebot from accessing it. It get very frustrated as the impact is so huge, my blog has dropped in Google ranking for primary keywords. Since my blog is a Blogspot, I cannot edit the robots.txt. So here is what I did that has helped me to fix the problem:

1. Log on to your Google Webmaster.
2. Click on Site Configuration.
3. Click on crawler access.
4. Scroll down and you will see Test button. Click on the button.
5. At this point my ths status has been changed to success (200) without the denied by robots.txt again.
6. However if the problem still unsolved, click on Generate robots.txt.
7. Check Allow All and try to download the generated text file.
8. Even for a blogspot blog this is not important however it is among the steps I did to overcome the problem.
9. Click the Test robots.txt and try to test again.
10. Click on Labs on the left menu and choose Fetch as Googlebot. Click the Fetch button. At this point the error of denied by robots.txt should already fixed.

0 comments:

Original design by andrastudio Blogger port by Blogger Templates

Powered by Blogger