How to Stop Your Tags and Categories Being Indexed by Search Engines

How to Stop Your Tags and Categories Being Indexed by Search Engines

Since using Hugo, I’ve been trying to work out what to do for the best way to sort out my search engine optimisation now that I am happy with the platform that I am using. When I realised that my tags and categories are being indexed by Google and causing in some cases stopping the actual articles from being indexed because the algorithm is thinking the pages are duplicates.

Not wanting to delve too deeply into my theme I decided to use a robots.txt file to achieve blocking of tags and categories in the short term until I could find the right code to add to my theme to stop Google making the mistake of thinking my content is duplicated. This was relatively easily done I created the robots.txt file with these lines:

User-agent: *

Disallow: /images/*
Disallow: /tags/*
Disallow: /categories/*
Sitemap: /sitemap.xml

Here basically the code is telling the crawlers are told they can index everything except for what has a URL that has started with /tags/ and /categories/ along with /images/. While doing this a found that you could add your sitemap to tell the crawler exactly where to find it which was something new I didn’t know. Which should help with all the search engines, not just Google. Once this text file was created I put it in my static folder so every time I generate the site it will put the robots.txt in the right place.

While this was the easiest solution I didn’t want to leave it there I wanted to edit the right parts of the template as well as Hugo is supposed to be able to add no index to the pages themselves. I found this bit of code that was supposed to be able to do this but wasn’t sure where to put it.

{{ if .Data.Singular }}
    <meta name="robots" content="noindex">
{{ end }}

However, I quickly worked out it was in your theme folder layout _default and then baseof.html and you just add it to your head tag.

So those are the two methods to block categories and tags from being indexed in Hugo but will I keep these measures? In all honesty in the time it has taken for me to achieve this I probably won’t because of the way I tag this blog I am not losing anything by having the tag indexed in some cases instead of the post. This is because I write quite a lot of book reviews and someone interested in one book may be interested in more books by the same author. However, depending on how you use your tags you might decide that doesn’t add too much value for your readers.