Cached Includes not tracked during generation

[mijewe]

Bug Report

On a page with cached includes (ESI enabled), when using the GenerateCacheJob, only the page will become tracked - any cached includes on the page will not be tracked.

Steps:

1. purge, clear, flush Blitz cache so diagnostics are clear
2. trigger a GenerateCacheJob for a page with cached includes
3. notice that the page itself appears in Tracked Pages, but nothing in Tracked Includes.
   Expected behaviour: all cached includes on the page are now tracked.

I believe the issue is that the GenerateCacheJob adds a ?token so that it will bypass cached pages, but this token param also forces ESI includes to be inlined.

I hackily solved this issue for my project with a custom generator that uses getUrlsToGenerate($siteUris, false); to disable the token, but that doesn't work 100% well since it now doesn't regenerate already-cached pages - but it does now track includes.

tldr:

• opening a page via the browser does track includes 👍
• opening a page via cURL does track includes 👍
• opening a page via GenerateCacheJob does not track includes ❌

related (and maybe even the same) issue: #780

Diagnostics Report

Application Info

• PHP version: 8.3.21
• Craft edition & version: Pro 5.7.10
• Database driver & version: MySQL 28.1

Installed Plugins

• Amazon S3: 2.2.3
• Announce: 1.0.0
• Bespoken: 5.0.7
• Blitz: 5.12.6
• Cache Ramp: dev-develop
• CKEditor: 4.9.0
• Cloudflare Turnstile: 2.0.1
• Content Importer: dev-develop
• Embedder: 4.0.0
• Feed Me: 6.8.0
• Imgix Asset Transformer: dev-main
• Lantern: 1.0.1
• Navigation: 3.0.15
• Postmark: 3.1.0
• Retour: 5.0.13
• Sentry Logger: 5.0.2
• SEOmatic: 5.1.20
• Servd Assets and Helpers: 4.2.3
• Sprig: 3.7.1
• Webperf: 5.0.0

Loaded Modules

• tacklebox: modules\tacklebox\Tacklebox
• golocalprov: modules\golocalprov\GoLocalProv
• cachebuster: modules\cachebuster\CacheBuster
• cpassets: modules\cpassets\CpAssets
• imgmapper: modules\imgmapper\ImgMapper
• codeeditor: nystudio107\codeeditor\CodeEditor
• sprig-core: putyourlightson\sprig\Sprig
• verbb-base: verbb\base\Base

Blitz Plugin Settings

{
    "debug": true,
    "hintsEnabled": false,
    "cachingEnabled": true,
    "refreshCacheEnabled": true,
    "refreshMode": 0,
    "includedUriPatterns": [
        {
            "siteId": "",
            "uriPattern": ".*",
            "enabled": true
        }
    ],
    "excludedUriPatterns": [
        {
            "enabled": true,
            "siteId": 1,
            "uriPattern": "articles\/|ad-zone|site.webmanifest"
        }
    ],
    "cacheStorageType": "putyourlightson\\blitz\\drivers\\storage\\YiiCacheStorage",
    "cacheStorageSettings": {
        "compressCachedValues": true
    },
    "cacheStorageTypes": [],
    "cacheGeneratorType": "honcho\\craftcacheramp\\generators\\EsiHttpGenerator",
    "cacheGeneratorSettings": {
        "useBasicAuth": true,
        "username": "$BASIC_AUTH_USERNAME",
        "password": "********************"
    },
    "cacheGeneratorTypes": [
        "honcho\\craftcacheramp\\generators\\EsiHttpGenerator"
    ],
    "customSiteUris": [],
    "cachePurgerType": "servd\\AssetStorage\\Blitz\\CachePurger",
    "cachePurgerSettings": [],
    "cachePurgerTypes": [],
    "deployerType": "putyourlightson\\blitz\\drivers\\deployers\\DummyDeployer",
    "deployerSettings": [],
    "deployerTypes": [],
    "ssiEnabled": false,
    "ssiTagFormat": "<!--#include virtual=\"{uri}\" -->",
    "detectSsiEnabled": true,
    "esiEnabled": true,
    "cachedIncludePathParam": "p",
    "onlyCacheLowercaseUris": false,
    "cacheActionRequests": false,
    "queryStringCaching": 1,
    "includedQueryStringParams": [
        {
            "enabled": true,
            "siteId": "",
            "queryStringParam": "^(relatedTo|excludeId)$"
        }
    ],
    "excludedQueryStringParams": [
        {
            "enabled": "1",
            "queryStringParam": "gclid"
        },
        {
            "enabled": "1",
            "queryStringParam": "fbclid"
        }
    ],
    "apiKey": "**************",
    "generatePagesWithQueryStringParams": true,
    "purgeAssetImagesWhenChanged": true,
    "refreshCacheAutomaticallyForGlobals": true,
    "refreshCacheWhenElementMovedInStructure": true,
    "refreshCacheWhenElementSavedUnchanged": false,
    "refreshCacheWhenElementSavedNotLive": false,
    "refreshExpiredCacheAfterVisit": true,
    "cacheNonHtmlResponses": false,
    "trackElements": true,
    "trackElementQueries": true,
    "excludedTrackedElementQueryParams": [],
    "cacheDuration": null,
    "nonCacheableElementTypes": [],
    "sourceIdAttributes": [],
    "liveStatuses": [],
    "integrations": [
        "putyourlightson\\blitz\\drivers\\integrations\\CommerceIntegration",
        "putyourlightson\\blitz\\drivers\\integrations\\DatastarIntegration",
        "putyourlightson\\blitz\\drivers\\integrations\\SeomaticIntegration"
    ],
    "defaultCacheControlHeader": "no-store",
    "cacheControlHeader": "public, s-maxage=31536000, max-age=0",
    "cacheControlHeaderExpired": "public, s-maxage=5, max-age=0",
    "sendPoweredByHeader": true,
    "outputComments": true,
    "refreshCacheJobPriority": 10,
    "driverJobBatchSize": 100,
    "driverJobPriority": 100,
    "queueJobTtr": 300,
    "maxRetryAttempts": 10,
    "maxUriLength": 2048,
    "maxUriIndexLength": 767,
    "mutexTimeout": 1,
    "commands": [],
    "injectScriptEvent": "DOMContentLoaded",
    "injectScriptPosition": 3
}

Recommendations

• ❌ One or more globals exist and refreshCacheAutomaticallyForGlobals is enabled.
• ✅ Blitz is configured not to refresh cached pages when an element is saved but unchanged.
• ✅ Blitz is configured not to refresh cached pages when an element is saved but not live.
• ✅ Image transforms are configured to be generated before page load.
• ✅ The @web alias is explicitly defined.
• ✅ Queue jobs are configured not to run automatically via web requests.
• ✅ The Async Queue plugin is not installed or enabled.
• ✅ The blitz/cache/refresh-expired console command has been executed within the past 24 hours.

Site Tracking

• Tracked Pages: 4,788
• Tracked Includes: 5,175
• Tracked Actions: 0
• Tracked Query String Params: 0
• Tracked Elements: 46,001
  • craft\elements\Entry: 22,602
  • craft\elements\Asset: 13,276
  • craft\elements\Entry (nested): 10,083
  • verbb\navigation\elements\Node: 40
• Tracked Element Queries: 100
  • craft\elements\Entry: 86
  • verbb\navigation\elements\Node: 14
• Tracked Tags: 39

[bencroker]

I believe the issue is that the GenerateCacheJob adds a ?token so that it will bypass cached pages, but this token param also forces ESI includes to be inlined.

It does, and the reason the cached include is not generated is that, as a unique page, it is not visited by the generator. So this is expected behaviour at present, rather than a bug. I don’t see a strong argument to regenerate the cached include when a page is generated. Is there something I’m missing?

[mijewe]

eg:

a news website with an index page displaying a card for each news article, where each card is a cached include.

The reason each card is a cached include is because the cards are displayed in multiple places throughout the site, and doing it this way reduces the number of pages to recache when an entry is saved.

If I warm the cache of an index page via Blitz, I would expect my first human visit to be fast - and crucially to not really do any PHP/DB work. But instead, each cached include card has to now be generated for the first time, which kind of defeats the point of warming the cache.

[bencroker]

There is no explicit concept of “warming” the cache in Blitz. Cache generators do generate cached pages, of course, but cache warming is the responsibility of plugins like Cache Igniter.