XML Sitemap Best Practices: Limits, Index Files and Dead URLs
The hard limits on an XML sitemap, when you need a sitemap index instead, and how to find the dead and redirecting URLs that accumulate inside one while nobody is looking.
Like it ? share it
A small site has one file, sitemap.xml, and that is the end of the subject. At ten or a hundred thousand pages it stops working, because the format has hard limits:
- 50,000 URLs per sitemap.
- 50MB uncompressed per file.
Past either of those you need a sitemap index.
What a sitemap index is
A standard sitemap lists pages. A sitemap index lists sitemaps. It contains no page URLs at all, only pointers to other files that do. A typical structure separates content by type:
sitemap_index.xml
├── post-sitemap.xml every blog post
├── page-sitemap.xml static pages
└── category-sitemap.xml category archives
Three things come out of splitting it up this way.
Debugging gets narrower. If new products are not being indexed, you look at product-sitemap.xml rather than scanning fifty thousand blog URLs to find the ones that matter.
Files stay small. Smaller files are quicker for a crawler to fetch and re-fetch, and re-fetching is the normal case.
The ceiling moves a long way up. An index can hold 50,000 sitemaps, each holding 50,000 URLs. You will not reach 2.5 billion pages, which is the point.
Auditing a nested tree
The awkward part is reading one. Open a WordPress or Yoast sitemap_index.xml expecting a URL list and you get five links to other files. Everything you actually wanted is one level down, and there is no obvious way to flatten it by hand.
The Sitemap URL Extractor follows the tree. Paste the index URL, it recognises that it is an index rather than a sitemap, fetches each child, extracts the URLs from those, and compiles one list you can export. It handles gzipped .xml.gz files too.
The URLs inside it are the real problem
A sitemap is a list of pages you are asking a search engine to index. If a third of that list answers with a 404 or bounces through a redirect, you are asking it to spend crawl budget on pages that are not there.
Nobody sets out to do this. It happens because a sitemap is generated automatically and then nobody looks at it again, while products get deleted and URLs get restructured underneath it.
The traditional way to check is four steps: download the .xml, convert it to CSV, load it into a crawler, wait for the crawl. It works, and it is enough friction that in practice it happens after a problem rather than before one.
Read the metadata first
Every URL comes with its lastmod, changefreq and priority if the sitemap declares them, and these are worth a look on their own. A lastmod of two years ago on a page you rewrote last month means your generator is not picking up changes. A whole sitemap with identical priority values is telling a search engine nothing.
Then check the status codes
This is the part that matters, because a URL that looks right in a list is not the same as a URL that answers.
| Status | What it means | Fix |
|---|---|---|
| 200 | Healthy | Nothing |
| 404 | The page is gone | Remove the URL from the sitemap. Do not add a redirect just to keep it in the list |
| 301 or 302 | The sitemap has the old URL | Replace it with the destination. A sitemap should list final URLs |
| 500 | The page is erroring, not missing | Look at your server logs. This one is a bug, not a sitemap problem |
The 404 case is where people go wrong most often. The instinct is to redirect the dead URL to something. But a sitemap is a list of pages that exist, and a redirect entry in it is a page that does not. Take it out.
Practical notes
Serve it gzipped. .xml.gz is supported and saves real bandwidth on a large file that gets fetched repeatedly.
Submit only the index. In Search Console you submit sitemap_index.xml and Google discovers the children itself. Submitting all of them individually gives you more rows to read and no more coverage.
Keep noindex pages out. A page in a sitemap is a page you are asking to have indexed. If it carries a noindex tag you are sending two contradictory instructions, and the report you get back will be confusing rather than wrong.
Re-check on a schedule, not just after a migration. The slow version of this problem is the one that matters: a store that deletes twenty products a month has a sitemap that quietly rots, and nobody notices until indexed page counts have been drifting down for two quarters.
Whether it is one file or fifty, the thing worth knowing is which URLs you are actually submitting. That list is rarely the one you think it is.