How To Set Custom Robots.txt and Custom Robots Header Tags In Blogger

The file Robots.txt is a file which tells search engine crawlers how they should visit your site. If there are files and directories you do not want indexed by search engines, you can use a robots.txt file to define where the robots should not go. These files are very simple text files that are placed on your web server.

Not only Robots.txt help search engines to crawl through website but also Custom Robots header tags plays a great role in SEO i.e. Search Engine Optimization I have provided here with both settings which you can use to get in search results.

Even if you want to be shown up in search engine results then you must not ignore this files settings.

Here is the content which you should paste in your custom robots.txt column:

User-agent: Mediapartners-Google
User-agent: *
Disallow: /search
Disallow: /p/*
Disallow: /view/*
Disallow: /?m=1
Disallow: /?m=0
Disallow: /*?m=1
Disallow: /*?m=0
Allow: /
Sitemap: Your site link/feeds/posts/default?orderby=UPDATED
Sitemap: Your site link/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap:Your site link/atom.xml?redirect=false&start-index=501&max-results=500
Sitemap:Your site link/atom.xml?redirect=false&start-index=1001&max-results=500

Setup Custom Robots Header Tags

First enable this option from your blogger settings and then I’ll tell you how to use it.

  1. Log in to your blogger account.
  2. Now navigate to Setting >> Search Preferences

  3. Click on edit link under the Custom Robots Header Tags section as shown in below screenshot.
  4. Once you click on the edit link you will see many options. Simply tick on the options as shown in the below image.
  5. Now click on “”Save changes” button.
  6. You are done!

But let’s have a look that what these tags means:

Allow search engine to index all pages and blog content.
Not allow search engine to index this page on result page, also not to display ” cached ” link on SERP.
This tag is not allow search engine to follow links on this page.
This tag has a same value as noindex, nofollow.
Not allow search engine to display “Cached” link on SERP.
Search engine is not allow to show short description on SERP for this page.
This tag is not allow search engine to use metadata from Open Directory Project on tittle for this page.
Search engine is not allow to offer a translation for this page on SERP
Image on this page is not allow to index by search engine.
Unavailable after:
This page will not show on SERP on specific date/hour that is setting in RFC 850 format.
You can use these setting as it is a default recommendation for a better SEO. but think that Any wrong check can cause your blog disappear or get punish by google because violent their TOS. Like the warning said, so use this tool careful. 

Article written by:

Satyendra is a Biotechnologist from the city of Nawabs, Lucknow. Apart from blogging, his other interests are Indian rap knowledge, affiliate marketing, poetry, and basically anything nerdy.

Join the discussion

  1. Anonymous

    Really good info about BlogSpot. I went through your blog and I have completed this section as explained. Such a good and informative things you had shared for us.

    One point I want to noted that; not just because of SEO but there are various plugins in CMS platforms, that make it more professional and user-friendly. However, I find blogger more user-friendly for the non-technical person like me because of these QUICK SEO settings. Thanks for sharing.

    We expecting much for the new and advanced information and thoughts from you. This post is an awesome and great source of information for a newbie.

  2. Edeh Samuel

    Good work sir. I have done that to my blog.

  3. Unknown

    Thanks for this informative blog post. Yes, custom then header tags settings is a great part for blogger blog.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

back to top