Robots.txt If you are using blogger for your hosting you have a robots.txt file automatically, and can NOT change it. To find the robots.txt, open up your web browser, and type in your Blogger blog’s URL, at the end of the URL add robots.txt. For example, if the URL of your blog is http://myblog.blogspot.com, then enter http://myblog.blogspot.com/robots.txt You may find the following entries: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Sitemap: http://myblog.blogspot.com/feeds/posts/default?orderby=updated Mediapartners-Google is an Adsense crawler, which crawls pages to determine AdSense content. Google only use this bot to crawl your site if AdSense ads are displayed on your site. So the first two lines means your blog allow Mediapartners-Google to crawl blog contents, nothing are disallowed, so it's empty after "Disallow:". "User-agent:* " means all search engine, a star sign '*' means all. The robots.t...
Lost in the Digital Land, Leaves behind Footprints on the Sands of Time.