A sitemap is a file that contains a list of the web addresses (URLs) for all the important pages on your website. Its main function is to assist search engines in comprehending your site and easily finding specific pages.
Types of sitemaps
There are two types:
- XML sitemaps: These are specially formatted sitemaps intended for search engine crawlers.
- HTML sitemaps: These sitemaps resemble regular web pages and aid users in navigating the website.
In relation to SEO, we typically refer to XML sitemaps. Therefore, in this guide, we will focus on XML sitemaps.
XML Sitemaps
An XML sitemap is a file that outlines the crucial pages of a website, ensuring that Google can locate and index them all. It also assists search engines in comprehending the structure of your website. It is important to have Google crawl every significant page on your site.
However, there are instances where certain pages lack internal links, making them difficult to discover. An XML sitemap provides a standardised method of listing posts and pages, making them easily and quickly discoverable by search engines.
Here’s a basic example containing just one URL:
The sitemap comprises a few components:
- XML version declaration: This helps search engine crawlers identify the file type they are reading.
- URL set: It informs search engines about the protocol being used.
- URL: This section contains the page’s URL.
- Lastmod: It denotes the date of the page’s last modification in a specific format.
To be considered valid, every sitemap must adhere to this standard. While there are additional properties such as <priority> and <changefreq>, they do not impact its functionality or performance.
An example of an XML sitemap index
Below, you’ll see a screenshot of the base XML sitemap of webintelligenz.com.
This index includes various sitemaps for different sections of webintelligenz.com. Each line is accompanied by a date, indicating the most recent update of each post. This is beneficial for SEO because you want Google to crawl and index your updated content promptly. When the date changes within the sitemap, Google recognizes the presence of new content to be crawled and indexed.
The XML sitemap for webintelligenz.com reveals multiple ‘index’ sitemaps: post-sitemap.xml, page-sitemap.xml, services-sitemap.xml, and so on. This categorisation simplifies the structure of the site.
If you click on one, such as post-sitemap.xml, you will find a comprehensive list of all the post URLs on webintelligenz.com.
How do search engines use sitemaps
Sitemaps play a crucial role in SEO by aiding web crawlers in understanding a website’s structure, thereby facilitating easier evaluation and ranking.
To grasp their significance in SEO, it’s important to comprehend how search engines operate, specifically the concepts of “crawl” and “index.”
Google utilizes bots or spiders that continuously scour the internet, cataloging web pages through the process of crawling your website. These bots then classify and store each discovered page within Google’s vast index, known as indexing.
As a result, when you perform a search on Google, it doesn’t actually search the entire web in real time. Instead, it searches its well-organized index, enabling it to deliver search results within a fraction of a second.
The implication of this process is that if your page is difficult to crawl, it may not be included in Google’s index. If it’s not in Google’s index, it won’t appear in search results and this is where sitemaps come into play.
Do you need a sitemap?
In general, Google is quite proficient at discovering web pages on the internet by itself. However, as we discussed earlier, having a sitemap can enhance your SEO, although its impact may vary depending on the website.
According to Google, having a sitemap is beneficial under the following circumstances:
- If you have a large website with 500 or more pages. With thousands of pages, there is a higher likelihood that Google’s crawlers might overlook newly added or updated pages.
- If your internal linking is insufficient, resulting in numerous orphan pages that lack proper connections.
- If your website is new or has limited backlinks. Web crawlers typically find website pages by following links from one site to another.
- If you have a significant amount of rich media, such as images, videos, or news pages, that you wish to display in search results.
Benefits of sitemaps
The better Google understands and crawls your site, the more effectively you can rank for your target keywords and attract more traffic. Building upon the information provided above, let’s delve deeper into the advantages of having a sitemap:
Expedite crawling and indexing of your pages:
Google cannot crawl the entire internet on a daily basis. It follows different crawl schedules for various websites and content types. Consequently, it may take days, weeks, or even months for Google to discover new pages on your site. Sitemaps come to the rescue by helping Google find and index new pages more swiftly.
Maintain optimal performance of high-value pages:
Have you ever made updates to a page on your site, such as refreshing your evergreen content, only to find that the changes are not reflected in the search engine results pages (SERPs)? This is likely because Google hasn’t crawled the page since your update. By enabling more efficient crawling and indexing, you can ensure that users are presented with the most recent version of your most important or frequently updated pages.
Assist search bots in locating orphan pages:
Google’s bots discover pages on your site in a similar way to how visitors do—by following the links found on the pages they crawl (which emphasizes the importance of internal linking). Orphan pages, however, lack other links pointing to them, making them challenging for Google to access. By including these pages in your sitemap, Google can more easily locate and index them.
Aid Google in identifying duplicate content:
Business websites often have scenarios where duplicate or nearly identical pages exist, such as multiple product pages with different color variations on an e-commerce site. In such cases, Google may struggle to determine which version of the page is the primary one that should be ranked. With a sitemap, you can utilize canonical tags to inform Google about the main version of the page and identify the duplicates.
How to Check a Website Sitemap
Manual Check
The easiest way to find a website’s sitemap is to look for it manually. Most commonly, it will be located at this URL address:
https://domain.com/sitemap.xml
Quite often—especially for WordPress sites that use the Yoast SEO plugin—you’ll be redirected to a sitemap index (/sitemap_index.xml).
Search Operators
Search operators are special phrases or symbols that you can include in a search query to obtain more precise results.
Here are a few search operators you can employ to locate a sitemap on a website:
- “site:domain.com filetype:xml”
- “site:domain.com inurl:sitemap”
- “site:domain.com intitle:sitemap”
These operators can be used on popular search engines like Google, Bing, or Yahoo. Simply enter the desired operator into the search bar, replacing “domain.com” with the actual website address.
If the sitemap exists and has been indexed by the search engine you’re using, the search results should provide the location.
Google Search Console
If you have access to a website’s Google Search Console, it is possible that the sitemap has already been submitted there.
To check, navigate to the “Sitemaps” report in the “Indexing” section found in the left menu.
Within this section, you will come across a category labeled “Submitted sitemaps.” If someone has previously submitted one, you will find its URL listed there.
We will discuss the process of submitting a sitemap to Google Search Console later in this guide.
Robots.txt
Robots.txt is a file on your website that communicates with search engine crawlers, instructing them on which sections of the site to crawl and which ones to avoid.
To ensure it is accessible, place the robots.txt file in the root folder of your site, typically found at https://domain.com/robots.txt.
If the robots.txt file adheres to best practices, it will include a link to the website’s sitemap.
The section that directs to the sitemap will have a similar structure as shown below:
If you have attempted all the methods mentioned above and still cannot locate your XML sitemap, it is likely that your website does not have one.
How to create a sitemap
Most CMSs automatically generate and update your XML sitemap without requiring manual intervention. Some examples of CMSs that offer this functionality are:
When using a CMS, there is usually limited or no opportunity for manual edits of your sitemap.
XML Sitemap Plugins in WordPress
If you’re using WordPress and desire more control, you can opt to replace the default sitemap by using plugins.
In general, there are two types of WordPress plugins you can utilise to generate a sitemap:
- General SEO plugins (like Yoast SEO, All in One SEO, or Rank Math)
- Dedicated sitemap plugins (like XML Sitemaps or XML Sitemap & Google News)
Which pages to include in your sitemap?
When it comes to deciding what pages to put in your sitemap, keep it simple. Just think about whether a URL is relevant and valuable to someone who visits it. If it doesn’t provide any value or you don’t want people to land on it, then there’s no need to include it.
How to submit sitemap to Google
Submitting your XML sitemap to Google is an SEO best practice. That’s mainly for two reasons:
- It can speed up the process of Google discovering your sitemap
- It can help you detect issues with your sitemap
To submit your sitemap login to your Google Search Console account.
Then, go to “Index” → “Sitemaps” in the sidebar.
If you already submitted one, you’ll see a list of “Submitted Sitemaps” on this page:
Either way, to add a new sitemap, enter your sitemap’s URL into this field and hit “Submit”.:
And if everything is all setup, you’ll start to see information on your sitemap on this page under the “Submitted Sitemaps” section.
Need help creating and submitting a sitemap?
Are you looking to enhance your website’s SEO and ensure maximum visibility in search engines?
With our expertise, we will guide you through the process of setting up a comprehensive sitemap tailored to your website’s specific needs.
Not only will we assist you in the initial setup, but we’ll also provide you with ongoing support and training on how to maintain it effectively. Our friendly and knowledgeable team will ensure that you have the necessary tools and insights to optimize your website’s performance continuously.
Don’t miss out on valuable opportunities to reach your target audience. Take advantage of our SEO services at Web Intelligenz and let us help you unlock the full potential of your website
