Consider not storing full sitemap XML

[mjangda]

Right now, sitemap XML is generated async and stored in the database to allow them to be served super quickly. The downside is that any code changes that modify the XML output means all sitemaps need to be re-generated which can be a very slow, time-consuming process on really large sites with thousands of sitemaps.

We should explore alternate ways to handle this (while maintaining backwards compat with existing actions/filters) and evaluate whether those approaches make sense.

[systemseven]

Sitemaps have to be regenerated when the template changes, no way around that. The trick here is to figure out how to do that efficently.

Right now the flow is as follows (simplistic version)

• Cron job runs and creates a post for a particular day (sitemaps are technically custom post types)
• As part of that cron job, using simpleXML to create the necessary XML nodes
• Run several queries to get all the data we need, and save all that to a string as a postmeta option
• Display that saved data string (from postmeta) when a user visits the page

Here's what I'm proposing

• Cron job stays 99% the same, still creates a custom post type (although, we could look at moving away from it and to straight queries in the template I'm proposing below, but that may be past the scope of this individual issue)
• BUT we don't create the simpleXML nodes, instead we create a list of posts (basically just their urls) that need to be included in that sitemap, we could cheat if we wanted to and make this list of posts the post content of the sitemap post, which removes the data from postmeta, which would mean 1 less query. (vs making a postmeta option containing the list of posts and querying to get that)
• On display, we can grab that list, loop over, and display.
• For the display, I say we move to a custom post template, we can still support the hooks and we can support changes. And it could be wired up in a way to allow people to create their own template for display if necessary.
• For the XML rendering. Alot of the tags never change, so we could hard code them in the template vs using SimpleXML (ie: loc, lastmod, etc..). In fact, MOST of the template could be hard coded and we could use hooks for the stuff that could change (and/or users can always fall back to the custom template idea)
• Performance wise I think we're gonna net out close to the same, but this method I'm proposing may be a bit more expensive
• That being said, we could add in a check, were the sitemap caches itself for XXX amount of time, the cache is served if it hasn't expired, if it has expired it serves and creates a new cached version.

[mjangda]

we create a list of posts (basically just their urls) that need to be included in that sitemap

What happens if the URL structure changes? How do we get other related data like the post modified date from just the URL?

Alot of the tags never change, so we could hard code them in the template vs using SimpleXML (ie: loc, lastmod, etc..)

Do we get any major benefits from switching to a hard-coded template? How will we maintain backwards compatibility (e.g. some filters pass in the simplexml object that sites use to add things like images)?

Performance wise I think we're gonna net out close to the same, but this method I'm proposing may be a bit more expensive

This is probably the biggest thing we'll need to watch. Some of the sites using this plugin have millions of posts dating back 5/10/20 years. If the newer method is significantly slower, it may not be worth it so it would be good to gather and compare some data as we work on this.

[systemseven]

Hey Mo,

A few others asked the some of the same questions you did on the internal site.

For the urls, that's just an example, we'll just need to make sure we store the right data needed, and in the post_meta, the the post_content

Good catch about the backwards compatibility on the SimpleXML objects, I'm going to take a look at that.

Thanks

[GaryJones]

Analysis of the 2.x architecture and proposal

The 2.x branch has been refactored to use Domain-Driven Design with value objects (UrlEntry, UrlSet, SitemapDate, etc.) and clean separation between data and presentation. This positions us well to address this issue.

Proposed approach: Store structured data, render XML on request

Instead of storing pre-rendered XML, store the sitemap content as serialised UrlSet data (JSON). The XML rendering becomes a presentation concern handled at request time.

Current flow:
  Generate → Render XML → Store XML → Serve XML

Proposed flow:
  Generate → Store UrlSet data → [Request] → Render XML → Serve
                                     ↓
                              (Object cache layer)

Implementation outline:

1. Storage format change

   • Replace msm_sitemap_xml post meta with msm_sitemap_data containing JSON-serialised UrlSet
   • The UrlSet already contains all necessary data: URLs, lastmod, changefreq, priority, images
   • Migration path: check for legacy XML format and serve directly during transition

2. Request-time rendering with caching

   • On sitemap request, deserialise UrlSet and render via existing SitemapXmlFormatter
   • Cache rendered XML using wp_cache_get/set with a composite key:

     msm_sitemap_{post_id}_{format_version}
     

   • format_version is a hash of relevant filter callbacks + plugin version
   • Cache TTL: indefinite (invalidated by version change or explicit flush)

3. Cache invalidation triggers

   • Plugin update (version change)
   • Filter registration changes (detected via has_filter snapshot)
   • Manual "clear format cache" admin action
   • Individual sitemap regeneration (existing flow)

What this solves:

• XSL stylesheet changes → immediate effect
• XML structure changes → immediate effect
• Adding/modifying filters that affect output → immediate effect
• No need to regenerate thousands of sitemaps for presentation-only changes

What still requires regeneration (unavoidable):

• Permalink structure changes (URLs are different)
• Post content changes affecting sitemap data
• Adding/removing posts from date ranges
• Changes to which post types are included

Content provider changes require regeneration

When content providers are added, enabled, or disabled, the underlying data changes - not just the presentation. For example:

• Enabling the Image provider: Existing UrlSet data doesn't contain image entries; they need to be fetched and added
• Disabling a provider: Stored data contains entries that should no longer appear
• Adding a custom provider: New URL entries need to be generated

This means content provider changes fall into the "requires regeneration" category. However, we can make this less painful:

1. Detect configuration changes

   • Store a hash of the active provider configuration alongside each sitemap
   • On request, compare current config hash with stored hash
   • If mismatched, either serve stale + queue background regeneration, or regenerate on-demand for that single sitemap

2. Incremental regeneration option

   • Rather than regenerating all sitemaps at once, regenerate on first request (lazy)
   • Spread the load over time as sitemaps are naturally accessed
   • Background cron catches any that aren't accessed organically

3. Provider-aware partial updates

   • For additive changes (enabling images), enhance existing UrlSet data without full regeneration
   • Fetch just the new provider's data and merge with stored entries
   • More complex but avoids re-querying all post data

The key distinction:

Change type	Affects	Action needed
XSL styling	Presentation only	Clear render cache
XML structure/filters	Presentation only	Clear render cache
Content provider enabled	Stored data	Regenerate sitemaps
Content provider disabled	Stored data	Regenerate sitemaps
Post type settings	Stored data	Regenerate sitemaps

Performance considerations:

Scenario	Current	Proposed
Cache hit (object cache)	Read meta + echo	Read meta + cache get + echo
Cache miss (object cache)	Read meta + echo	Read meta + deserialise + render + cache set + echo
No object cache	Read meta + echo	Read meta + deserialise + render + echo

The overhead is minimal for sites with persistent object cache (the VIP/enterprise use case). Sites without object cache see slightly more work per request, but this is offset by never needing bulk regeneration for format changes.

Backwards compatibility:

The msm_sitemap_entry_xml filter currently passes a SimpleXMLElement. Options:

1. Maintain filter with rendered element (slight overhead to parse/re-render)
2. Deprecate in favour of new msm_sitemap_entry_data filter passing UrlEntry
3. Support both during a transition period

Storage efficiency:

JSON UrlSet data is typically smaller than rendered XML (no repeated namespace declarations, closing tags, etc.), so database storage may actually decrease.

[pkevan]

The use of caching in the XML has the potential to have very little benefit, unless there is enough capacity to store these sets of data for extended periods of time, which memcache isn't really know for, and I'm assuming most other cache systems will work on a LRU basis as well, so thereby reducing the potential benefit (if i'm reading it correctly, it's only going to be for saving processing, which is usually a relatively efficient part).

From some initial research it seems like msm_sitemap_entry is where most customers choose to inject their variants to sitemap output, and skipping inclusion of items in sitemaps - i'd concentrate on making sure this has good back-compat.