backend: cache game metadata between scans

[Splaser]

Adds a lightweight game metadata cache to speed up startup with large libraries.

Cache invalidation

The cache is automatically invalidated when:

• metadata files change (metadata.txt, metadata.pegasus.txt, *.metadata.txt, *.metadata.pegasus.txt)
• enabled providers change
• cache schema version changes

The cache is intended to speed up repeated launches of an unchanged library.

Changes to ROM files or media assets without corresponding metadata changes currently require a manual refresh/reload.

Testing

Tested on Android 15 (Odin3) with a large ROM collection (~1TB).

Verified:

• initial scan creates cache successfully
• subsequent launches restore from cache
• favorites persist correctly after cache restoration
• metadata modifications invalidate the cache and trigger a full rescan

[mmatyas]

Interesting! How much time does this save for you?

[Splaser]

Interesting! How much time does this save for you?

On my Android handheld (AYN Odin3), a library of about 9,500 ROMs takes roughly 70 seconds to scan on first launch.
Subsequent launches restore from cache in about 1–2 seconds.

Since my library is relatively stable, most launches are repeated launches rather than rescans, so the cache makes a significant difference in day-to-day use.

[mmatyas]

Really nice. The cache implementation itself looks OK on a quick glance, two points I'm concerned about:

• If you have a metadata file that adds all files with a matching extension, and you drop new files into the directory, then Pegasus will not find new games, because the metadata file itself is unchanged
• Similarly, if any third party provider adds or removes a game (Steam, GOG, etc.), that won't reflect in Pegasus. The cache currently overrides every provider, but only checks the metadata provider's files

[Splaser]

Really nice. The cache implementation itself looks OK on a quick glance, two points I'm concerned about:

* If you have a metadata file that adds all files with a matching extension, and you drop new files into the directory, then Pegasus will not find new games, because the metadata file itself is unchanged

* Similarly, if any third party provider adds or removes a game (Steam, GOG, etc.), that won't reflect in Pegasus. The cache currently overrides every provider, but only checks the metadata provider's files

Thanks, both points make sense.

My main test case is an Android handheld setup with a mostly metadata-based ROM library, so I focused on repeated launches of a relatively stable collection. I did not test the Steam/GOG providers much.

For the first point, I agree the current invalidation is not enough for metadata entries that collect files by extension. I can add a lightweight directory fingerprint for the configured game directories, eg. file count + newest mtime, so dropping new ROMs into a scanned directory invalidates the cache.

For the second point, I also agree that the current cache should not override dynamic third-party providers unless their state is part of the fingerprint. A conservative fix could be to only use this cache when the enabled provider set is limited to Pegasus metadata/media/favorites/playtime, or alternatively skip the cache when Steam/GOG/Lutris/etc. providers are enabled.

Would you prefer a conservative metadata-only cache for now, or a provider-level fingerprint approach?

[mmatyas]

I'm okay with a cache covering (as in, substituting) only a subset of providers, as long as the points above are resolved. We'll probably need a somewhat more complex way to detect directory changes though, as mtime for a file might not reflect when the file was placed into the directory, and for a directory, might not reflect modification in its contents. Symlinks may also make things difficult, especially if they loop, so you might want an upper limit if you do recursive checks. Note that you must also consider assets, eg. adding or removing media files for the games. So unfortunately I'm not sure how feasible a lightweight solution is here.

Alternatively, you could also decide that the cache is only for metadata files without such "wildcards", though that might need some redesign in the current solution.

If you do find a good way to accurately detect the need for a rescan, you could use both cached and uncached data, if possible, or disable the cache if not. It's probably not too scalable to do if (provider_name == "steam" || ...) checks, but if it helps, you could eg. introduce a CACHEABLE flag here.

Also if you have a good solution, I'd be interested in how large the cache file grows for a dataset like yours, and also how it compares in time to the current Pegasus, both when "Validate game files" under Settings is turned off, and when turned on.

[Splaser]

I'm okay with a cache covering (as in, substituting) only a subset of providers, as long as the points above are resolved. We'll probably need a somewhat more complex way to detect directory changes though, as mtime for a file might not reflect when the file was placed into the directory, and for a directory, might not reflect modification in its contents. Symlinks may also make things difficult, especially if they loop, so you might want an upper limit if you do recursive checks. Note that you must also consider assets, eg. adding or removing media files for the games. So unfortunately I'm not sure how feasible a lightweight solution is here.

Alternatively, you could also decide that the cache is only for metadata files without such "wildcards", though that might need some redesign in the current solution.

If you do find a good way to accurately detect the need for a rescan, you could use both cached and uncached data, if possible, or disable the cache if not. It's probably not too scalable to do if (provider_name == "steam" || ...) checks, but if it helps, you could eg. introduce a CACHEABLE flag here.

Also if you have a good solution, I'd be interested in how large the cache file grows for a dataset like yours, and also how it compares in time to the current Pegasus, both when "Validate game files" under Settings is turned off, and when turned on.

Thanks, that makes sense.

I agree that trying to perfectly detect every possible library change can quickly become as expensive and complex as a full scan itself, especially when considering wildcard metadata entries, media assets, symlinks and third-party providers.

I like the idea of introducing a CACHEABLE flag on providers. That feels cleaner than hardcoding provider names and would allow the cache to be limited to providers that can reliably participate in cache invalidation.

I also collected some numbers from my Android test setup. I manually deleted the cache file between runs (/data/data/org.pegasus_frontend.android/cache/gameindex-v1.json) and measured fresh scans:

Android 15 (Odin3), metadata-based ROM library

Validate game files ON:

• Games: 9437
• Full scan after cache deletion: ~19–23 s
• Cache file: 17 MB
• Cache restore: ~1 s

Validate game files OFF:

• Games: 12322
• Full scan after cache deletion: ~17–19 s
• Cache file: 22 MB
• Cache restore: ~1 s

The OFF case contains more games because my metadata includes entries for games that are not currently present locally.

One thing I noticed while profiling this is that a large part of the scan time seems to come from resolving media assets (box fronts, logos, videos, etc.), which may explain why cache restoration is so much faster than a normal scan on Android.

One correction regarding my earlier timing estimate: the previously mentioned ~60–70s startup time was not a carefully measured benchmark and is probably not reliable. After manually deleting the cache file and repeating the tests, I would consider the ~20s vs ~1s numbers above the more accurate comparison.

[Splaser]

Force refresh improvement

This new commit also introduces an optional force_refresh scan mode.

When enabled, the game index cache is bypassed and a full provider scan is executed, regardless of cache validity. This is used to support explicit user-triggered rescans.

Default behavior remains unchanged.

With this addition, the behavior remains fully aligned with the existing cache system.

The force refresh mode does not change cache semantics or invalidation rules — it only provides a user-triggered override to bypass the cache and execute a full rescan.

In practice, this ensures that any unexpected or stale results can always be resolved by an explicit manual refresh, without requiring changes to cache fingerprint logic.

[mmatyas]

Hm, so if I understand correctly, by introducing a force refresh flag, you shift the responsibility of detecting changes on the file system to the user: if there's a change in the directories, or in the third party providers, it is the user's responsibility to manually run a refresh. That is a valid strategy, but probably not the best default behavior.

If we go this route, I'd like an option under Settings, next to "Validate game files", with a label like "Scan for games on launch", a description like "When enabled, ... . When disabled, ...something something cache...", and it should be enabled by default.

[Splaser]

Hm, so if I understand correctly, by introducing a force refresh flag, you shift the responsibility of detecting changes on the file system to the user: if there's a change in the directories, or in the third party providers, it is the user's responsibility to manually run a refresh. That is a valid strategy, but probably not the best default behavior.

If we go this route, I'd like an option under Settings, next to "Validate game files", with a label like "Scan for games on launch", a description like "When enabled, ... . When disabled, ...something something cache...", and it should be enabled by default.

That makes sense. I agree the safest default is to preserve the current behavior and scan on every launch.

I’ll add a “Scan for games on launch” setting next to “Validate game files”, enabled by default.

When enabled, Pegasus will perform the normal full scan on startup. When disabled, it will restore the cached game index when valid, while a manual reload will still always bypass the cache and perform a full scan.

[Splaser]

Update

Implemented the suggested Scan for games on launch option next to Validate game files.

• Enabled by default, preserving the existing startup behavior
• When enabled, Pegasus performs a full provider scan on launch
• When disabled, Pegasus restores the cached game list when possible
• Reload all games always bypasses the cache and performs a full scan
• Cache misses or invalid fingerprints still fall back to a full scan

Tested on Android 15 with an AYN Odin3:

• setting enabled: full scan runs on launch
• setting disabled: cached game data is restored
• manual reload correctly bypasses the cache
• the setting is persisted in settings.txt

The new UI strings are marked with QT_TR_NOOP, but translations have not yet been added to the separate translations repository.

[mmatyas]

Thanks! Wonderful code, it was a pleasure to read. I've only found some minor things, but otherwise I think we can go with this. I'll also do some testing locally. And sorry again for the late reply!

[mmatyas]

A possible optimization: if an array is empty, we should not add it to the gameindex JSON. Pegasus supports eg. many asset types, but it is unlikely that a game will have all of them, so the JSON will be full of empty arrays. (I'm fine with implementing this later in a follow up PR.)

[Splaser]

Thanks for the review!

I've implemented both suggestions:

• Replaced the provider-name based cache handling with explicit PROVIDER_FLAG_CACHEABLE flags and kept cache-hit logging limited to actual skipped scans.
• Updated GameDataCache to omit empty arrays from gameindex-v1.json while keeping missing fields backward compatible during loading.

Thanks again for the helpful feedback!

[mmatyas]

Ok, I think this looks good!

Now I was about to merge, but I noticed some commits have different author information, for both name and email. Could you double check which ones are correct?

[Splaser]

Ok, I think this looks good!

Now I was about to merge, but I noticed some commits have different author information, for both name and email. Could you double check which ones are correct?

Both are the same guy -- me. There are 2 working PCs for me from my home and office with different emails.

[mmatyas]

Ok, looks good to me then, thanks!