Set Redis TTL on tab storage keys

[quantumdark]

Motivation

When using Redis as a storage backend (NICEGUI_REDIS_URL), tab storage keys accumulate forever in Redis. The existing prune_tab_storage() function only clears keys from the in-memory _tabs dictionary. If the application restarts, _tabs is empty and the app has no knowledge of orphaned Redis keys, causing them to persist until Redis runs out of memory (OOM).

Related to the max_tab_storage_age configuration which currently only works for in-memory cleanup but has no effect on Redis key expiration.

Implementation

Added TTL (Time To Live) support to RedisPersistentDict for tab storage:

• Added optional ttl: float | None parameter to RedisPersistentDict.init (follows type of max_tab_storage_age)
• Modified publish() to use Redis SETEX command (atomic set-with-expiry) when TTL is configured, otherwise falls back to regular SET
• Tab storage now passes ttl=Storage.max_tab_storage_age when creating RedisPersistentDict instances
• TTL is refreshed on every write (every data change), so active tabs never expire while inactive/orphaned tabs are automatically cleaned up by Redis

Key behavior:

• Only tab storage uses TTL (user and general storage remain permanent)
• TTL defaults to max_tab_storage_age (30 days) for consistency with existing pruning behavior
• Redis handles cleanup automatically, even after app restarts

Progress

• I chose a meaningful title that completes the sentence: "If applied, this PR will..."
• The implementation is complete.
• Pytests have been added (or are not necessary).
• Documentation has been added (or is not necessary).

[evnchn]

Surface-level-speaking, let's use set for expiry if we really want to handle expiry in Redis.

And, generally we do not prefer to pass a parameter around for single-purpose. RedisPersistentDict can read core.app.storage.max_tab_storage_age (not Storage.max_tab_storage_age, let's align with the codebase) directly.

---

But I want to ask: Is that really a good idea?

Tracing back max_tab_storage_age, it is originally exclusively used in:

nicegui/nicegui/nicegui.py

Lines 246 to 254 in dc49827

	async def prune_tab_storage(*, force: bool = False) -> None:
	"""Prune tab storage that is older than the configured ``max_tab_storage_age``."""
	tab_storages = core.app.storage._tabs # pylint: disable=protected-access
	for tab_id, tab in list(tab_storages.items()):
	if force or time.time() > tab.last_modified + core.app.storage.max_tab_storage_age:
	tab.clear()
	if isinstance(tab, PersistentDict):
	await tab.close()
	del tab_storages[tab_id]

and notably: it is invoked every 10 seconds as seen in:

nicegui/nicegui/nicegui.py

Line 146 in dc49827

app.timer(10, prune_tab_storage)

Then, I have the following questions:

• What happens to the tab storage, if Redis tossed the key because it is sharp-pass the deadline, but NiceGUI did not prune it due to the polling interval?
  • Should, in the next polling interval, the tab storage is expired, does the pruning still occur normally?
  • Should the tab storage is further mutated before the next NiceGUI poll occurs, then isn't the state torn between Redis and NiceGUI? Are there side-effects?
• Owing to the above concerns, is it better to handle the purging of Redis tab storage at prune_tab_storage instead?

Thank you for reading my longwinded comment!

[evnchn]

Sorry, did I scare you away? 🥺

[quantumdark]

Apologies, I was off for the holidays.

I've chosen the parameter approach because to me the alternative was too implicit. To address your feedback I added a flag based on the id prefix to avoid an explicit parameter, but I'm open for other ideas.

To answer your question is this a good idea, to me this approach seems very reasonable. We have a Redis server that we have to reboot from time to time due to is running out of memory because of the orphaned tab storage keys. When using Redis storage, if a NiceGUI server crashes or restarts:

• The in-memory _tabs dict is lost
• Redis keys persist forever
• On restart, prune_tab_storage() only knows about tabs in the current _tabs dict
• Old Redis keys from previous server instances become orphaned forever

The drift between the expiration time counted in Python vs in Redis should be pretty close because they both update on writes. If you think this is still a concern we could add a small delta (max_tab_storage_age + delta) that is greater than polling interval when saving to Redis, this way if it's not pruned by that time, it is safe to assume it is orphaned.

[evnchn]

Thanks. It seems that using ex is non-negotiable because the server does not know about tab IDs in the past-life.

I simplified the code a bit, and added a delta. Rationale for the delta value chosen is in the commit message of f643226

Rationale: 1 cycle is definitely not enough due to potentially last-modified being 70 seconds:

• NiceGUI's not pruning it
• Redis, due to network latency, gets the request at 70.1 seconds, expired

20 seconds is a really conservative setting considering that attempts to connect to a Redis server with 10 seconds of network latency will likely fail horribly, but better safe than sorry!

@falkoschindler Should be ready for review!

[falkoschindler]

Thanks for the pull request, @quantumdark!

I just reviewed it thoroughly and came up with one small improvement: redist_persistent_dict.py and storage.py were coupled in a very fragile way based on the suffix "tab-". Instead I made ttl a parameter to be is set by storage.py. Furthermore I declared modul-level constants for various strings to reduce the risk for regressions.

Ready to merge! 👍🏻

[evnchn]

Seems like my guessing of preference was wrong in:

And, generally we do not prefer to pass a parameter around for single-purpose. RedisPersistentDict can read core.app.storage.max_tab_storage_age (not Storage.max_tab_storage_age, let's align with the codebase) directly.

I like it either way, so we can proceed.

Bonus item (non-blocker): I wonder is it a good idea to add a TTL to FilePersistentDict, so that old .json files on disk also gets purged? Right now I do not see a mechanism for purging old user JSON files...