Google’s New Robots Meta Tag

Google’s massive indexing efforts are legendary. They drive our whole industry and form the backbone of our clamor over rankings. Traditionally our control over how Google indexes our site has been somewhat limited but has been growing lately. Another update came through quietly in January this year. So, almost half a year out now, are you enjoying the effects of this new addition? But then – What update are we talking about? Today, we will walk you through Google’s new “indexifembedded” robots tag.

Using the new tag, you can tell Google to index content embedded in iframes and HTML tags, even if their page has a noindex tag. Google provided a use case example, saying that the new tag “addressed a common issue that affects media publishers.” These businesses, and others, may want their content indexed when it’s on third-party pages. They may not, however, want their media pages indexed alone. On such pages, it’s standard to use a “noindex” tag on such pages, but this prevents the content from appearing on other sites when indexed.

To give an example of this use case, say, a podcast hosting service features media. The podcast service page uses both the “noindex” and “indexifembedded” tags. This means the content will not be indexed on its own but will index if it is embedded on other pages. The correct order of operations is to place the “noindex” tag first.

The use case might be a little niche for some businesses, but there are ways to leverage this and other robots tags. Right now, only Google supports the tag, but if its use expands to other search engines, you may benefit by adopting early.

What Are Robots Tags?

The reality is that while everything on your site is probably important, not all of it is important to index. To prevent these pages from being indexed, you can use robots tags, also called meta tags or x-robots-tags. You may have seen a file called robots.txt in the data of some websites. This file tells search crawlers what pages they can go to on your site. Pages must be here to be indexed. Once on the page, the robot’s tag will determine whether it can be indexed. Think of the robots.txt as a roadmap and the robots tag as the attractions.

These tags go in the <head></head> portion of a web page’s HTML code. They are called meta tags because of how they are written. This is an example use case of a “noindex” tag:
<meta name=”robots” content=”noindex” />
Because these tags are determined on a page-by-page basis, you can control how search engines interact with your website on a more granular level.

What Are Robots Tags Good For?

Using meta tags is important for ensuring search engines show the side of your website important to you. There may be images that you don’t want to be indexed, such as page layout items or content you’d rather show after a click. With robots.txt, you can apply a variety of attributes to your pages. If, for example, you’d rather direct people to your storefront page, you might choose to index it, or you might choose only to have it displayed on other websites while indexing your homepage instead. It all depends on your needs and goals.

Controlling Snippets with Robots Meta Tags

Interacting with the indexing is how most meta tags are used; you can also use them to change how snippets appear on SERPs. This has been a feature since 2019, and with the changes, we’re seeing there’s a chance this feature could expand. For example, you can use tags like “nosnippet” to hide the meta description for a page or “max-video-preview:[X]” to show a video for a maximum duration of X seconds.

Are X-Robots-Tags Different?

Yes and no. These are alternate ways to write out meta tags, which allows for more control at the cost of more complexity. You write these in your site’s configuration file, usually a .php, .htaccess, or other files. You use these tags specifically when you want to control how crawlers handle things other than HTML pages. These also allow you to assign behaviors globally (On your whole site) rather than page by page.

Mistakes to Avoid

Because this technical, demanding work can be a bit opaque, making mistakes early on is common. For example, pages may behave incorrectly if incorrectly tagged. If you intend to have crawlers use a tag on a page, that page has to be accessible to the crawler. It is also important not to remove “noindex” pages from sitemaps if you use a sitemap.

The various robot tags are robust tools you can use to direct traffic to your website in a more focused way. More importantly, you can avoid common SEO missteps, like your indexed results competing with one another. You put a lot of effort into how your website is branded. It will help if you put some effort into how your site is represented on search engines. Robots.txt and meta tags are a great way to do this. If you want more information like this, check our website.