XML Sitemap Best Practices for SaaS Sites at Scale

An XML sitemap lists the canonical URLs you want search engines to discover. Generate it from the published content source, keep modification dates accurate, and monitor fetch and indexing results.

What Google Uses from Your Sitemap

Before optimizing a sitemap, it helps to know what Google pays attention to.

What Google uses:

• The URL list, to discover pages it has not yet crawled.
• The lastmod field, but only if your site has a track record of updating it accurately. Use lastmod for significant page changes; it does not guarantee a recrawl schedule. Sites that set lastmod to the current date on every build teach Google to ignore it.

What Google ignores entirely:

• priority — Google's own documentation confirms this field has no effect.
• changefreq — Also confirmed as ignored. Do not spend time on these.

Understanding this shapes a cleaner sitemap strategy: focus on URL accuracy and honest lastmod timestamps.

XML Sitemap Best Practices: What to Include and Exclude

Include the URLs you want considered for indexing.

Include:

• Pages you want indexed and that currently return a 200 status.
• Pages that are canonical — the definitive version, not a filtered or sorted variant.
• Core product pages, feature pages, integration pages, blog posts, and landing pages.

Exclude:

• Pages that are noindex — including them signals a contradiction to Google and wastes resources.
• Redirected URLs — always use the final destination URL.
• Canonicalized duplicates — include only the canonical URL, not the alternatives that point to it.
• Admin pages, staging paths, or anything behind authentication.
• Thin pages that you would not want ranked even if Google did find them.

A sitemap that includes only what you want indexed is a cleaner, more trustworthy signal than one that includes everything.

Handling Large Page Sets with Sitemap Index Files

Google limits a single sitemap file to 50,000 URLs and 50 MB uncompressed. SaaS products that generate programmatic pages — integration hubs, comparison pages, template libraries — can exceed this quickly.

The solution is a sitemap index file. A sitemap index is an XML file that lists the locations of multiple individual sitemap files. Structure the index by section:

• sitemap-blog.xml — editorial content
• sitemap-integrations.xml — programmatic integration pages
• sitemap-comparisons.xml — comparison and alternative pages
• sitemap-features.xml — core product and feature pages

Segmenting by section makes it easier to monitor which parts of the site are being indexed and at what speed. If your integration pages are slow to index, you can investigate without filtering through a single large file.

Submit the sitemap index URL in Google Search Console, not the individual sitemap files. Search Console shows crawl statistics per sitemap file, which is a useful diagnostic when specific page types are underperforming.

Generating Sitemaps Programmatically

For any SaaS product generating pages from a database or CMS, sitemaps should be generated automatically from the same data source that drives the pages. Static sitemap files updated manually fall out of sync as content changes and new pages are added.

A well-designed sitemap pipeline:

1. Reads the URL set from the source of truth — database, CMS API, or build output.
2. Filters out excluded pages based on index status, canonical target, and HTTP status.
3. Writes the lastmod field based on the actual last-modified timestamp of the content record, not the current build date.
4. Splits the URL set across segment files when a segment exceeds the 50,000 URL limit.
5. Rebuilds and republishes automatically whenever content changes.

Most modern frameworks — Next.js, Astro, Nuxt — support programmatic sitemap generation either natively or through well-maintained plugins. Use them rather than manually maintaining a static file.

Validating and Monitoring Your Sitemap

After publishing, validate the sitemap structure using Google Search Console's Sitemaps report. It will flag format errors, show how many URLs were submitted versus how many were indexed, and surface any fetch errors.

A gap between submitted URLs and indexed URLs is normal in the short term. A large persistent gap — after four to six weeks — usually indicates one of three problems:

• Pages are being submitted but are noindex or have conflicting canonical tags.
• Pages are too thin or too similar to be worth indexing.
• The site lacks the authority to support rapid indexing of a large new page set.

Monitor the gap regularly. For programmatic pages specifically, tracking the ratio of submitted-to-indexed across each sitemap segment gives a clear signal of which page types Google finds valuable enough to index.

A Note on Robots.txt

Your sitemap URL should be declared in robots.txt:

Sitemap: https://yoursite.com/sitemap-index.xml

This lets supporting crawlers discover the sitemap without relying on a Search Console submission. It is a minor addition but costs nothing.