X-Robots-Tag HTTP header and Google Webmaster Tools URL Removal

I recommend

I read about being able to use the HTTP header X-Robots-Tag to specify Google search index control options. For my purposes, I wanted to specify that certain pages on my blog would not go in the index. So I specified the header X-Robots-Tag: noindex using Microsoft’s URL Rewrite module 2 which can use out bound rules to rewrite page content and page headers.

The header was correctly being added to the pages, but when I used the remove URL tool in Google webmaster tools to remove the pages the result was the request for page removal was denied.

image

The information about the X-Robots-Tag discusses using it for page content like PDF files and such. It does not specifically state that it will or will not work for regular content type text/html. I asked on the Google webmaster tools help forum, but I did not get any response.

I ended up modifying my blog’s code to add a meta ROBOTS tag to the HTML head element of my pages which works as expected.

I can’t be 100% sure that the X-Robots-Tag will not do the job, but I could not get it to work. If someone has found out otherwise please let me know.

Technical January 6, 2010


Comments