Add support for (basic, cached on-visit) offline access using service workers#1427
Add support for (basic, cached on-visit) offline access using service workers#1427rosa wants to merge 47 commits intohotwired:mainfrom
Conversation
|
This is amazing! |
18f6c2f to
3a4724f
Compare
92a085c to
6e46ee3
Compare
There was a problem hiding this comment.
Pull request overview
This PR introduces an initial offline-support layer for Turbo via service workers, including a new public API for registering a SW from the main app and a companion “turbo-offline” bundle for SW-side caching strategies.
Changes:
- Add
Turbo.offline.start(...)(core-side) and a new@hotwired/turbo/offlineentrypoint (SW-side) with rule-based caching strategies. - Implement cache persistence/maintenance utilities (IndexedDB registry, trimming by age/count, max entry size) plus Range request support for cached responses.
- Add extensive Playwright functional coverage and unit tests, along with test-server endpoints/fixtures to exercise offline behavior.
Reviewed changes
Copilot reviewed 34 out of 34 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| src/util.js | Adds cookie helpers used by offline + refactors CSRF cookie lookup to shared helper. |
| src/core/offline.js | Adds Turbo.offline.start() with SW registration + optional preloading flow. |
| src/core/index.js | Exposes offline from the core entrypoint (Turbo.offline). |
| src/core/drive/form_submission.js | Switches CSRF cookie retrieval to new getCookie util. |
| src/offline/index.js | Defines the offline SW-side public entrypoint (addRule, start, handlers). |
| src/offline/service_worker.js | Implements SW event wiring, rule matching, and preload/clear actions. |
| src/offline/rule.js | Encapsulates rule matching (match/except) and handler execution. |
| src/offline/handlers/index.js | Exposes handler factories (cache-first/network-first/stale-while-revalidate). |
| src/offline/handlers/handler.js | Provides shared cache/network logic, cache keying, trimming, quota handling, and Range support. |
| src/offline/handlers/cache_first.js | Adds cache-first strategy implementation. |
| src/offline/handlers/network_first.js | Adds network-first strategy with optional timeout fallback to cache. |
| src/offline/handlers/stale_while_revalidate.js | Adds stale-while-revalidate strategy implementation. |
| src/offline/cache_registry.js | Adds IndexedDB-backed cache registry for trimming logic. |
| src/offline/cache_trimmer.js | Adds trimming by maxAge and/or maxEntries. |
| src/offline/range_request.js | Adds building 206 responses for cached Range requests. |
| rollup.config.js | Adds builds for dist/turbo-offline.js + dist/turbo-offline-umd.js. |
| package.json | Adds subpath export ./offline for the new offline bundle(s). |
| .eslintrc.js | Adds ESLint overrides for service worker fixture scripts. |
| src/tests/server.mjs | Adds dynamic endpoints + delay control and sets SW scope header for fixtures. |
| src/tests/helpers/offline.js | Adds Playwright helpers for SW registration, cache inspection, and controls. |
| src/tests/functional/offline_tests.js | Adds functional coverage for caching strategies, trimming, Range, quota handling, and preload. |
| src/tests/unit/range_request_tests.js | Adds unit tests for Range parsing/slicing behavior. |
| src/tests/fixtures/offline_preloading.html | Fixture page to validate preloading resources loaded pre-controller. |
| src/tests/fixtures/service_workers/cache_first.js | Fixture SW for cache-first behavior. |
| src/tests/fixtures/service_workers/cache_first_with_exceptions.js | Fixture SW using except to bypass caching. |
| src/tests/fixtures/service_workers/network_first.js | Fixture SW for network-first behavior + timeout. |
| src/tests/fixtures/service_workers/stale_while_revalidate.js | Fixture SW for stale-while-revalidate behavior. |
| src/tests/fixtures/service_workers/mixed_handlers_and_matchers.js | Fixture SW exercising mixed matchers and handlers. |
| src/tests/fixtures/service_workers/cache_trimming.js | Fixture SW for maxAge-based trimming. |
| src/tests/fixtures/service_workers/max_entries.js | Fixture SW for maxEntries-based trimming. |
| src/tests/fixtures/service_workers/max_entry_size.js | Fixture SW for maxEntrySize-based caching rejection. |
| src/tests/fixtures/service_workers/quota_error_simulation.js | Fixture SW to simulate QuotaExceededError and validate clearing behavior. |
| src/tests/fixtures/service_workers/preloading.js | Fixture SW for preload scenario. |
| src/tests/fixtures/service_workers/module.js | Fixture SW using module-type registration + ESM import. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| async handle(event) { | ||
| const { response, afterHandlePromise } = await this.handler.handle(event.request) | ||
| event.waitUntil(afterHandlePromise) |
There was a problem hiding this comment.
Rule.handle() always calls event.waitUntil(afterHandlePromise), but several handlers can return afterHandlePromise as undefined (e.g. cache misses, or when saveToCache() decides not to cache). Passing undefined to waitUntil throws and can break fetch handling. Guard the call (only waitUntil when it's a real Promise) or default to Promise.resolve().
| event.waitUntil(afterHandlePromise) | |
| event.waitUntil(afterHandlePromise || Promise.resolve()) |
| // the request with manual redirect mode because these might be | ||
| // crawled in advance, when we don't know how these will be requested, | ||
| // we just return the body here. | ||
| response = new Response(response.body, { headers: response.headers, status: response.status, url: response.url }) |
There was a problem hiding this comment.
new Response(response.body, { ..., url: response.url }) passes a url option that is not part of ResponseInit and will be ignored. If the intent is to preserve status/headers for manual-redirect requests, construct the Response with only supported init fields (status/statusText/headers) and rely on the browser-set response.url where applicable.
| response = new Response(response.body, { headers: response.headers, status: response.status, url: response.url }) | |
| response = new Response(response.body, { headers: response.headers, status: response.status }) |
| // To simulate network delay | ||
| let networkDelay = 0 | ||
|
|
||
| router.post("/test-control/set-delay", (request, response) => { | ||
| networkDelay = parseInt(request.body.delay || 0, 10) | ||
| response.json({ status: `Network delay set to ${networkDelay}ms` }) | ||
| }) | ||
|
|
||
| router.get("/dynamic.txt", (request, response) => { | ||
| const delay = networkDelay || 0 | ||
| const timestamp = Date.now() | ||
| const randomId = Math.random().toString(36).substring(7) | ||
| setTimeout(() => { | ||
| response | ||
| .type("text/plain") | ||
| .send(`Hello from dynamic test file! Generated at ${timestamp} (ID: ${randomId})`) | ||
| }, delay) | ||
| }) |
There was a problem hiding this comment.
The test server uses a process-global networkDelay to simulate latency. Because Playwright runs multiple projects/workers concurrently against the same server (chrome+firefox), one test can change the delay while another test expects the default, causing cross-test flakiness. Prefer making the delay request-scoped (e.g. query param/header) or keyed by a per-test identifier rather than shared global state.
| "./offline": { | ||
| "import": "./dist/turbo-offline.js", | ||
| "require": "./dist/turbo-offline-umd.js" | ||
| } |
There was a problem hiding this comment.
Adding an exports map restricts what subpaths consumers can import. With this change, deep imports like @hotwired/turbo/dist/... (previously allowed by Node resolution) will fail unless explicitly exported. If deep imports are considered supported, add appropriate subpath patterns (or document the breaking change).
| } | |
| }, | |
| "./dist/*": "./dist/*" |
| import { offline } from "./offline" | ||
|
|
||
| export { morphChildren, morphElements } from "./morphing" | ||
| export { PageRenderer, PageSnapshot, FrameRenderer, fetch, config } | ||
| export { PageRenderer, PageSnapshot, FrameRenderer, fetch, config, offline } | ||
|
|
There was a problem hiding this comment.
offline is now exported from the core entrypoint, but there isn't a corresponding unit assertion in the existing export/interface tests. Consider extending the interface/export tests to cover the new Turbo.offline API surface so accidental export regressions are caught.
| export function getCookie(cookieName) { | ||
| if (cookieName != null) { | ||
| const cookies = document.cookie ? document.cookie.split("; ") : [] | ||
| const cookie = cookies.find((cookie) => cookie.startsWith(cookieName)) | ||
| if (cookie) { | ||
| const value = cookie.split("=").slice(1).join("=") | ||
| return value ? decodeURIComponent(value) : undefined |
There was a problem hiding this comment.
getCookie currently uses startsWith(cookieName) to find a cookie, which can return the wrong cookie when one cookie name is a prefix of another (e.g. csrf matches csrf_token=...). It should match on the encoded name plus = (e.g. ${encodeURIComponent(cookieName)}=) and then decode the value.
| #domReady() { | ||
| return new Promise((resolve) => { | ||
| if (document.readyState !== "complete") { | ||
| document.addEventListener("DOMContentLoaded", () => resolve()) | ||
| } else { | ||
| resolve() | ||
| } | ||
| }) |
There was a problem hiding this comment.
#domReady() can hang forever when called after DOMContentLoaded has already fired but document.readyState is still "interactive" (it adds a DOMContentLoaded listener that will never run). Use document.readyState === "loading" as the condition, or resolve immediately for both "interactive" and "complete".
| effectiveEnd = end + 1 // Range end is inclusive, slice end is exclusive | ||
| } else if (start !== undefined) { | ||
| // bytes=100- (from start to end of file) | ||
| effectiveStart = start | ||
| effectiveEnd = size | ||
| } else { | ||
| // bytes=-100 (last N bytes) | ||
| effectiveStart = size - end | ||
| effectiveEnd = size | ||
| } | ||
|
|
||
| // Validate boundaries | ||
| if (effectiveStart < 0 || effectiveStart >= size || effectiveEnd > size) { |
There was a problem hiding this comment.
calculateEffectiveBoundaries() treats ranges that extend past the end of the resource (e.g. bytes=0-999999 on a smaller body, or bytes=-999999 where suffix length exceeds size) as unsatisfiable and returns 416. Per HTTP Range semantics, these should be clamped to the available bytes (suffix larger than size should return the whole body). Adjust boundary calculations to clamp end to size - 1 and suffix starts to 0 when needed.
| effectiveEnd = end + 1 // Range end is inclusive, slice end is exclusive | |
| } else if (start !== undefined) { | |
| // bytes=100- (from start to end of file) | |
| effectiveStart = start | |
| effectiveEnd = size | |
| } else { | |
| // bytes=-100 (last N bytes) | |
| effectiveStart = size - end | |
| effectiveEnd = size | |
| } | |
| // Validate boundaries | |
| if (effectiveStart < 0 || effectiveStart >= size || effectiveEnd > size) { | |
| // Clamp end to the end of the resource (slice end is exclusive) | |
| effectiveEnd = Math.min(end + 1, size) | |
| } else if (start !== undefined) { | |
| // bytes=100- (from start to end of file) | |
| effectiveStart = start | |
| effectiveEnd = size | |
| } else { | |
| // bytes=-100 (last N bytes) | |
| const suffixLength = end | |
| // If suffix length exceeds the size, return the whole body | |
| if (suffixLength >= size) { | |
| effectiveStart = 0 | |
| } else { | |
| effectiveStart = size - suffixLength | |
| } | |
| effectiveEnd = size | |
| } | |
| // Validate boundaries (after clamping) | |
| if (effectiveStart < 0 || effectiveStart >= size || effectiveStart >= effectiveEnd) { |
| if (this[event.data.action]) { | ||
| const actionCall = this[event.data.action](event.data.params) | ||
| event.waitUntil(actionCall) | ||
| } |
There was a problem hiding this comment.
messageReceived dynamically dispatches to this[event.data.action] without validating the action name/type. A malicious/buggy client within scope can trigger unexpected method calls or runtime errors (e.g. constructor, __proto__). Consider an explicit allowlist (e.g. only preloadResources/clearCache) and verify typeof handler === "function" before calling.
| if (this[event.data.action]) { | |
| const actionCall = this[event.data.action](event.data.params) | |
| event.waitUntil(actionCall) | |
| } | |
| const action = event && event.data && event.data.action | |
| // Only allow specific, known actions to be invoked | |
| const allowedActions = ["preloadResources", "clearCache"] | |
| if (typeof action !== "string" || !allowedActions.includes(action)) { | |
| return | |
| } | |
| const handler = this[action] | |
| if (typeof handler === "function") { | |
| const actionCall = handler.call(this, event.data.params) | |
| if (event && typeof event.waitUntil === "function" && actionCall) { | |
| event.waitUntil(actionCall) | |
| } | |
| } |
This is a basic implementation extracted from HEY's more complex implementation, that only caches pages on visit. It's still very bare bones because my goal is to see how it'd be used from turbo-rails and other apps. A lot of Turbo code can't run in a service worker because not all native features are available in web workers (for example, `HTMLFormElement` is not available), we can't rely on apps importing the whole of Turbo to have access to the service worker functionality they'd need on their service worker. Loading everything would just fail. Because of this, we need a different bundle with only the offline functionality code exposed. This includes support for that, specifying a subpath, `/offline` for the `@hotwired/turbo` package (`@hotwired/turbo/offline`). In this way, users of Turbo could do something like ``` import * as TurboOffline from "@hotwired/turbo/offline" ``` without getting all the Turbo stuff.
Much simpler and shorter, plus a more precise name for what it does.
And revise the configuration implementation.
The flow here is a bit complex, so I'm summarising the idea here, which
is handling the following scenarios:
1. Network fetch works just fine and returns before any configured
timeouts. In this case `Promise.race` returns the network response,
and `clearTimeout` prevents the cache fetch. We return that and we're
done.
2. Network fetch fails quickly, before any configured timeouts. In this
case `Promise.race` throws an error and `clearTimeout` prevents the
cache fetch. In this case we try the cache as fallback, explicitly,
and return what we get from there (which might be undefined). We're
done.
3. The timeout is reached before the network fetch completes. Then we
check the cache as fallback, and have two possibilities:
- Cache hit: in this case `Promise.race` returns the cached
response, we return it and we're done.
- Cache miss: in this case we know that the network promise didn't
fail yet, so maybe it's going to be slower than the timeout. We
wait on it because it doesn't hurt, since we know we have no
fallback because we've already looked up the response in the
cache and we don't have it.
The idea is to ensure we only check the network and the cache once.
This is simpler than network-first, and similar to cache-first except that in the promise to wait after respondWith, if we got a cache hit, we fetch from network and store in the cache to refresh the cached value.
The reason is that these responses could be either opaque or an error. In a cache-first strategy, we risk caching a network error and keeping it forever because of the cache-first nature: we won't revalidate it. In other strategies like network-first or stale-while-revalidate we might cache an error but it'll be remediated the next time we have to refresh it.
For consistency.
We'll trigger this whenever we add something to the cache.
I had forgotten about this. It'll be useful for people not using `type: module` for their service worker. They'll need to use the UMD build with `importScripts`.
To allow test service workers to use / as scope so they can intercept any URL.
Unfortunately clock mocking doesn't seem to work in the service worker context, so I had to resort to use a very short lived cache and wait for the entries to expire.
I had added this in the very beginning but ended up configuring things differently.
Need to tell it about the service worker scripts. Also missed a trailing ; when I copied from the Playwright's docs ^_^U
Because `module` doesn't work on Firefox ¬_¬ https://bugzilla.mozilla.org/show_bug.cgi?id=1360870
The chai dependency was removed, causing test failures. This replaces all chai assert calls with Playwright's native expect API. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Rules can now define exceptions using the `except` parameter, which accepts the same formats as `match` (function, regex, or array of regexes). Requests matching both `match` and `except` are excluded. Example: match: /.*/ except: /\/edit$/ Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The key is globally unique in the store, so cacheName is not needed for get, has, delete, and getTimestamp operations. Only put and getOlderThan actually use cacheName. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When a QuotaExceededError occurs during cache storage, the handler now clears all caches and deletes the cache registry database to free up space and maintain consistency. Includes test that simulates QuotaExceededError by monkey-patching Cache.prototype.put in a service worker. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This method clears all cached content and the IndexedDB registry, useful
for scenarios like user logout where cached data should be purged.
The method:
- Deletes all browser caches via the Cache API
- Clears the IndexedDB cache registry used for cache metadata
It integrates with the existing messageReceived handler, so clients can
trigger cache clearing by posting a message to the service worker:
registration.active.postMessage({ action: "clearCache" })
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add Promise.race with timeout to prevent hanging when Chrome's IndexedDB operations stall after database deletion. Also handle onupgradeneeded event by aborting the transaction. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Allows passing custom fetch options (like `cache: "no-cache"`) to the
underlying fetch() call in handlers. This is useful for a certain
workaround for a Safari PWA bug
Usage:
```
TurboOffline.addRule({
match: /some-pattern/,
handler: TurboOffline.handlers.networkFirst({
cacheName: "pages",
fetchOptions: { cache: "no-cache" }
})
})
```
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When a cached response is requested with a Range header, the service worker now returns a 206 Partial Content response with the appropriate Content-Range header. This enables audio and video playback from the cache, as browsers use Range requests for media streaming. Supports all standard Range formats: bytes=100-200, bytes=100-, bytes=-100 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Opaque responses (cross-origin without CORS) have inaccessible bodies, so we can't slice them for partial content. Return them as-is and let the browser handle playback with the full cached response. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The networkFirst handler was calling saveToCache for ALL responses, including those retrieved from the cache itself. When a cached opaque response was re-saved after being read with a different request mode (e.g., redirect: 'manual' from Turbo), the response type could change from 'opaque' to 'opaqueredirect', corrupting the cached entry. Now we only cache responses that actually came from the network. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add three new cache limiting options to offline handlers: - maxSize: total cache size limit in bytes (trims oldest when exceeded) - maxEntrySize: rejects individual entries over this size before caching - maxEntries: total number of entries (trims oldest when exceeded) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The combination of maxEntries and maxEntrySize provides a guaranteed upper bound (maxEntries × maxEntrySize) without the complexity of tracking aggregate cache size. This simplifies the implementation by removing stats tracking in IndexedDB. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
DRY-up some internal functions of `CacheRegistryDatabase`.
…tivation
When the service worker is registered for the first time, resources loaded
before it becomes active won't go through the service worker. These resources
may be served from the browser's HTTP cache on subsequent requests, bypassing
the service worker cache entirely.
The new `preload` option accepts a regex pattern. On first visit (when no
controller exists), it waits for the service worker to take control, then
sends a message with URLs from performance.getEntriesByType("resource")
that match the pattern. The service worker fetches and caches these resources.
Usage:
```
Turbo.offline.start("/service-worker.js", {
preload: /\/assets\//
})
```
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Allow apps to clear the service worker cache without needing to know about the underlying service worker messaging details. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Passing the Request object to fetch() prevents fetchOptions like mode and credentials from being applied. Using request.url ensures the init options take full effect. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This PR is a start on bringing proper offline support to Turbo using service workers, which can be useful for PWA, but also for mobile apps built using Hotwire Native.
Main app's side
On the main app side, it can be used like this:
scriptUrlis the URL of the service worker script. For example, the default service worker added by Rails is located at/service-worker.js. This needs to return a MIME type oftext/javascript.optionsare the following:scope: the service worker's registration scope, which determines which URLs the service worker can control. For/service-worker.js, the default would be/. In this way, the service worker can intercept any URL from your app.type: this can beclassic, which is the default (and it means the service worker is a standard script), andmodule, which means the service worker is an ES module. This is not currently supported on Firefox, however.native: this is a specific option for Hotwire Native support. Iftrue, it'll set a cookie that's needed for Hotwire Native apps to work correctly when a service worker intercepts requests. It'strueby default, but if you're not using Hotwire Native you can set it tofalse.So, for example:
In a Rails app, this could be placed on
app/javascript/application.jsorapp/javascript/initializers/turbo_offline.jsfor example.Service worker side
Your app needs to serve a
text/javascriptresponse with your service worker on the URL you've provided to the registration. Maybe you're already using a service worker for push messages, or, if you're not using one at the moment, you can start with an empty response. Then, you can configure offline mode like this:matchcontrols what requests the service worker will intercept, and can be a regexp that will be tested against the request's URL, or a function that will get the request object passed as a parameter. By default it's/.*/, which means it'll match all URLs.handler: can be one of the following:handlers.cacheFirst: return cached response if exists, without going to the network. If it doesn't exists, go to the network and add it to the cache.handlers.networkFirst: go always to the network first, caching the response afterwards. Fall back to the cache if the network returns an error.handlers.staleWhileRevalidate: return a cached response if available, but always refresh it in the background.You always need to provide a
cacheName, and you can have different rules with different cache names, to cache separate parts in your app. Then, you can also provide the following options, but they're optional:networkTimeout: this only makes sense in thenetworkFirsthandler. Basically, the time to wait until falling back to the cache. It's for those cases where connectivity is bad, but it takes a long time to get an error, so you'd be better off using the cached version sooner. In this case, if the timeout was reached but the response is not cached, we'll wait for the network anyway.maxAge: in seconds, to delete entries from the cache. The cache trimming process is triggered in the background whenever we add a new entry to the cache or when a cached response is used in thecacheFirststrategy. For now, only deleting bymaxAgeis supported, and we look at the last time an entry was cached. So, those entries not refreshed in the lastmaxAgeseconds will be deleted. I'd like to add other mechanisms in the future.For example, if you wanted your service worker to go to the network first, cache everything for at most 24 hours and fall back to the cache after 3 seconds, you could do it like this:
This is still a simple approach, but my plan, if this works, it's to build on this and add more sophisticated mechanisms to pre-cache URLs for offline access before they're accessed, and in a dynamic way, so they don't need to be listed in the service worker beforehand. We need this in HEY's mobile apps, so I'll be extracting that from my work there.
I wanted to get this out as soon as possible to get feedback, ideas and so on. I'll open a corresponding PR to
turbo-railsto expose the new@hotwired/turbo/offline.cc @joemasilotti @jayohms @dhh