Are Sitemaps Important?
Google’s Webmaster Guidelines are indispensable for many businesses, new and old. They help inform users about some of the finer inner workings of Google’s services that may not be clear. They teach users how to make websites readable for Google’s indexers. Users can follow the guidelines to help make their pages easier to find, index, and rank. This is because it’s such a great resource.
So, it may seem odd that Google’s John Mueller is telling some users they can ignore the part about HTML sitemaps. Why is that? What do sitemaps do for your website? This article will cover some of the insights Google has given us about these features.
What is a Sitemap?
Sitemaps are files that show information about the pages, files, images, and so on that build up your website. They show the way these pages link together and reference one another. Sitemaps are important because they allow web crawlers to make connections between pages on your site easily. For example, a search engine result may show a page and then the page that it is linked from. If you have a web store and somebody searches for products, a sitemap will allow a search engine to see that the product page is connected to a general store page. Sitemaps can also contain important and useful information. They can display video running time and categories, article titles or publication dates, and more.
Should I Use a Sitemap?
Websites that have properly linked pages can usually be crawled without help. Google’s web crawlers can usually discover which pages are related through a site’s navigation. A collection of links in a menu or links on existing pages can typically be crawled. Even so, sitemaps can benefit sites that are larger or have specialized file types. It is important to note that sitemaps don’t mean every item on the map will be crawled and indexed. Google’s processes use a complicated series of algorithms to determine what to crawl and skip. However, your site will usually benefit from a sitemap, and it will never be a negative.
Sitemaps are most useful for websites that meet the following conditions:
- Very large websites. Sites like catalogs, encyclopedias, or stores tend to have so much ground to cover that crawlers might skip new pages.
- Sites that have isolated pages without reference to many other pages. This is common with archival services or individual content pages.
- New sites with very few references on the internet. Web crawlers tend to find websites by following external links. A new website may not have many sites linking to it, meaning Google’s web crawlers may not find it
- Your site features rich media. For example, videos and images may be skipped without references in a sitemap.
By contrast, small and simple sites may not need a sitemap. To Google’s web crawlers, a “small” website is one with less than 500 pages. This count doesn’t even necessarily include every page on your site. It only includes the pages you think should appear in search results. Sites with thorough connectivity between pages also may not need a sitemap. Google can usually find everything it needs to by following robust links.
Do Sitemaps Need to be Human-Readable?
Mueller tells people with smaller websites that they may not need a human-readable sitemap. “If we can crawl your site normally, especially if the site is smaller, that’s not necessary,” said Mueller on Twitter. This is contrary to the Webmaster Guidelines page, which says you should include “a page with a human-readable list of links to these pages (sometimes called a site index or site map page).” Because sitemaps are for algorithmic web crawlers first and foremost, human readability is secondary.
How Do Web Crawlers Work?
As we mentioned, web crawlers operate based on an algorithm. A web crawl begins with a list of websites and sitemaps that have already been crawled. The web crawlers that Google employs begin by crawling over these sites again. This is why new websites need to be referenced on other websites. They go from link to link and send what they find back to Google to be indexed. Google saves keyword associations in the index to make new sites easier to find.
Google’s internet index is over 100,000,000 gigabytes large and will only get bigger as the company crawls more sites. It is an enormous amount of information, and knowing how to efficiently hand searchability information to Google is essential. If your website could benefit from a sitemap, Google’s documentation has an entire section on sitemaps. You can learn how to create a Google-friendly sitemap straight from the company itself. Alongside Google, our article databank at Emu Web Marketing might be an essential reservoir of notable points for you to draw from.