Make the cache configurable by cpp API by mgautierfr · Pull Request #950 · openzim/libzim

mgautierfr · 2025-02-07T09:39:10Z

This fix #946.

Contrary to what proposed in #946, this doesn't introduce a cache config structure but use simple method:

This follow comment Make cache configuration available from cpp api. #946 (comment)
As we will introduce global cache based on memory, the cluster cache configuration will be move to a plain function.

This PR probably have some small conflict with #949, I will resolve them as soon as a PR is merged.

mgautierfr · 2025-02-07T10:17:47Z

CI seems to be broken for other reasons than this PR.

include/zim/archive.h

src/fileimpl.cpp

test/archive.cpp

src/fileimpl.cpp

src/fileimpl.h

veloman-yunkan

Let's have #949 merged first. I will review the rebased&fixed-up version of this PR again after conflict resolution.

mgautierfr · 2025-02-12T14:31:06Z

PR is now rebased&fixed-up and conflict are resolved.

veloman-yunkan

All my feedback was restricted to the test code, which means that generally the PR is good to merge.

test/lrucache.cpp

test/archive.cpp

mgautierfr · 2025-02-17T14:31:40Z

Last push updates with your comments.
I have directly modified the commits to be ready to merge as soon as you agree (which I hope is soon)

Only test/lru_cache.cpp and test/archive.cpp files have been modified.

test/archive.cpp

mgautierfr · 2025-02-17T16:01:24Z

I have added a small comment about no reset the ref_it.
Please note the test will kind of become obsolete with PR #956 which introduce a global cluster cache. So I don't it worth have a perfect test here.

mgautierfr · 2025-02-26T18:09:52Z

This (almost) last push rename all introduced method from snake_case to camelCase as this is our coding style. fixup commits are interleaved in the git history.

@veloman-yunkan please give a formal approval before I rebase-fixup (and reword commit messages) and merge without further approval.

veloman-yunkan

I only checked the full diff from the previous version without inspecting each fixup commit separately. LGTM.

- DirentLookup becoming polymorphic - Possibility to disable lookup caching altogether - Once created, this cache size cannot be modified

Probably nobody never set this, and we don't use lzma anymore.

We don't use it anymore.

This make fail the test as it appears we create the dirent lookup cache at initialization. This will be fixed in next commits.

There is no reason this setting is done while reading the mimetype.

…ctor.

Let's use a temp dirent lookup (without cache) to access titleOrdered v1 and 'C' namespace.

Reading and writing in the same time to a unique_ptr is not thread safe. We introduce a atomicBool to know if the unique_ptr has been initialized. Fix #945

mgautierfr requested review from kelson42 and veloman-yunkan February 7, 2025 09:39

mgautierfr force-pushed the cache_config branch from 433e1b4 to 6c5f25e Compare February 7, 2025 10:04

veloman-yunkan requested changes Feb 10, 2025

View reviewed changes

mgautierfr force-pushed the cache_config branch from 6c5f25e to 2c34dbf Compare February 10, 2025 20:35

mgautierfr requested a review from veloman-yunkan February 11, 2025 08:53

veloman-yunkan reviewed Feb 11, 2025

View reviewed changes

mgautierfr force-pushed the cache_config branch from 2c34dbf to e6fbbbf Compare February 12, 2025 14:30

mgautierfr force-pushed the cache_config branch 2 times, most recently from fe9ee9d to 4a1881a Compare February 12, 2025 14:38

veloman-yunkan reviewed Feb 15, 2025

View reviewed changes

test/lrucache.cpp Outdated Show resolved Hide resolved

test/archive.cpp Outdated Show resolved Hide resolved

test/archive.cpp Show resolved Hide resolved

mgautierfr force-pushed the cache_config branch from 4a1881a to 117868f Compare February 17, 2025 14:28

mgautierfr requested a review from veloman-yunkan February 17, 2025 14:31

veloman-yunkan reviewed Feb 17, 2025

View reviewed changes

test/archive.cpp Show resolved Hide resolved

test/archive.cpp Show resolved Hide resolved

test/archive.cpp Show resolved Hide resolved

test/archive.cpp Outdated Show resolved Hide resolved

mgautierfr force-pushed the cache_config branch 2 times, most recently from 55616af to f3ea99f Compare February 17, 2025 15:59

mgautierfr force-pushed the cache_config branch from f3ea99f to cc02757 Compare February 17, 2025 16:02

veloman-yunkan approved these changes Feb 17, 2025

View reviewed changes

mgautierfr force-pushed the cache_config branch from cc02757 to c853517 Compare February 26, 2025 18:07

mgautierfr requested a review from veloman-yunkan February 26, 2025 18:09

veloman-yunkan approved these changes Feb 27, 2025

View reviewed changes

mgautierfr added 4 commits February 27, 2025 13:45

Introduce dropLast method in lru_cache.

0e300f8

Introduce helper macro to test cache

7d7f830

Make lrucache size configurable.

bb3cb89

Make concurrent cache size configurable.

df83972

mgautierfr added 12 commits February 27, 2025 13:45

Make cluster cache size configurable.

6df5d30

Make dirent cache size configurable

8677a5a

Make dirent lookup cache size configurable.

2c216e7

- DirentLookup becoming polymorphic - Possibility to disable lookup caching altogether - Once created, this cache size cannot be modified

Do not use env var to limit the maximum size used by lzma.

2a62877

Probably nobody never set this, and we don't use lzma anymore.

Remove envvalue source code.

1ec73f6

We don't use it anymore.

Add testing of cache

e5256b5

Ensure that cache configuration is taken into account

e9c3b7b

This make fail the test as it appears we create the dirent lookup cache at initialization. This will be fixed in next commits.

Move setting of UserEntry range out of mimetype.

8ce38e1

There is no reason this setting is done while reading the mimetype.

Don't hide access to direntLookup in subfonctions in FileImpl constru…

2371453

…ctor.

Don't create the cache dirent lookup at initialization.

4e9f37b

Let's use a temp dirent lookup (without cache) to access titleOrdered v1 and 'C' namespace.

Fix potential race condition on m_direntLookup initialization.

51f9e72

Reading and writing in the same time to a unique_ptr is not thread safe. We introduce a atomicBool to know if the unique_ptr has been initialized. Fix #945

Correctly test that changing the cache size on live archive has effect.

8dd1508

mgautierfr force-pushed the cache_config branch from c853517 to 8dd1508 Compare February 27, 2025 12:46

mgautierfr merged commit 2bde511 into main Feb 27, 2025
24 of 28 checks passed

mgautierfr deleted the cache_config branch February 27, 2025 12:56

This was referenced Feb 27, 2025

Limit the cluster cache by memory instead of number of clusters #956

Closed

Preload Xapian database #958

Merged

benoit74 mentioned this pull request Apr 19, 2025

kiwix-serve : provide a solution to customize cache settings kiwix/kiwix-tools#738

Closed

This was referenced Jun 3, 2025

Expose new libzim settings to handle memory consumption kiwix/libkiwix#1199

Open

Expose new libzim settings to handle memory consumption kiwix/kiwix-tools#745

Open

Expose new libzim settings to handle memory consumption openzim/zim-tools#470

Open

Uh oh!

Conversation

mgautierfr commented Feb 7, 2025

Uh oh!

mgautierfr commented Feb 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

veloman-yunkan left a comment

Choose a reason for hiding this comment

Uh oh!

mgautierfr commented Feb 12, 2025

Uh oh!

veloman-yunkan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgautierfr commented Feb 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgautierfr commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgautierfr commented Feb 26, 2025

Uh oh!

veloman-yunkan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mgautierfr commented Feb 17, 2025 •

edited

Loading