Skip to content
Merged
Show file tree
Hide file tree
Changes from 51 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
9c95a15
Update Ansible deployment to allow for unprivileged deployment using …
maxstack Jan 17, 2025
c98d16a
WIP: cached crate experiments
sd109 Jan 24, 2025
99e61ba
Add Bytes serde feature
sd109 Jan 24, 2025
df2ed74
Functional disk cache implementation
sd109 Jan 24, 2025
0d70386
Add prometheus counter for cache misses
sd109 Jan 24, 2025
f809321
Ensure client ID is included in chunk cache lookup key
sd109 Jan 24, 2025
d9dfd2d
Refactor to allow reinstating resource manager
sd109 Feb 7, 2025
528d532
Update the podman installation play to enable linger on the unprivile…
maxstack Feb 13, 2025
f3b994d
Allow Prometheus and HAProxy to be installed as the non-privileged user.
maxstack Feb 14, 2025
8fb4a95
Implement an object chunk cache on disk where we can configure:
maxstack Feb 27, 2025
2882c47
Refactor the chunk cache so it stores each chunk in a separate file.
Mar 4, 2025
a414f0c
Add example of simplified cache implementation
sd109 Mar 4, 2025
3e6ed6d
Add SimpleChunkCache pruning, with tests, and integrate with the Redu…
maxstack Mar 10, 2025
37f0c95
Remove unneeded dependencies, added during development of the caching…
maxstack Mar 10, 2025
4715e3e
Provide a S3 download handler and a cached S3 download handler with t…
maxstack Mar 11, 2025
5d408c8
Panic if the cache directory already exists.
maxstack Mar 11, 2025
c549b29
Remove test portion relating to cache wipe on init.
maxstack Mar 11, 2025
979d8a3
Channel all cache commits through a mpsc channel to serialise commits…
maxstack Mar 13, 2025
04e1f55
Allow configuration of the pruning interval, in seconds, of cache chu…
maxstack Mar 13, 2025
91eb805
Make "cargo clippy" happy.
maxstack Mar 13, 2025
c405c84
Merge remote-tracking branch 'origin/feat/disk-cache' into deployment…
maxstack Mar 13, 2025
4cb4ae8
Configure chunk cache test deployment.
maxstack Mar 13, 2025
093e9e4
Deploy a local build of the Reductionist.
maxstack Mar 13, 2025
ec5464a
Update Reductionist tag.
maxstack Mar 13, 2025
416cce3
Tweak what I think I think is needed to deploy our local build.
maxstack Mar 13, 2025
6fa2203
Correct the volume setup for the chunk cache.
maxstack Mar 13, 2025
46a3162
Tweak Reductionist deployment in group vars.
maxstack Mar 14, 2025
4b34f33
Tweak the Reductionist's ansible build so we can:
maxstack Mar 14, 2025
cfcbaf3
Allow the tokio mpsc buffer size to be configured, the number of chun…
maxstack Mar 14, 2025
0486393
Merge branch 'feat/disk-cache' into deployment/test
maxstack Mar 14, 2025
07e5e60
Add the chunk cache queue size to the group_vars.
maxstack Mar 14, 2025
15e3e60
Merge branch 'deployment/test' of https://github.com/stackhpc/reducti…
maxstack Mar 14, 2025
53d2a4e
Print out our command line args.
maxstack Mar 14, 2025
988e196
Add "headroom_bytes" to the pruning so we can ensure this many bytes …
maxstack Mar 17, 2025
a00c942
Tidy up adding code documentation.
maxstack Mar 18, 2025
72fc70a
Change over to tempfile::TempDir for temporary directory creation due…
maxstack Mar 19, 2025
9e4d08a
Change boolean assert_eq to assert to keep clippy happy.
maxstack Mar 19, 2025
4902ae1
Remove dupplicated cache error.
maxstack Mar 19, 2025
b20e193
Debug trait no longer needed on ResourceManager.
maxstack Mar 19, 2025
36a655e
Merge remote-tracking branch 'origin/main' into feat/disk-cache
maxstack Mar 19, 2025
cd6876b
Add chunk cache configuration to group_vars/all.
maxstack Mar 19, 2025
f1659c8
Fix for group_vars/all
maxstack Mar 19, 2025
9a7023e
More sensible default for the cache path that works with our without …
maxstack Mar 19, 2025
063a084
Misc fixes
sd109 Mar 24, 2025
e0c911b
Re-adopt an existing cache.
maxstack Mar 25, 2025
71b18de
Fix broken resource manager memory permits implementation
sd109 Mar 26, 2025
1bad961
Add S3 client auth check to cached object
sd109 Mar 27, 2025
195e663
Run compliance suite with cache enabled as part of CI test suite
sd109 Mar 27, 2025
b7f7201
Add chunk_cache_bypass_auth flag
sd109 Mar 27, 2025
d394384
Merge branch 'main' into feat/disk-cache
sd109 Mar 28, 2025
06286b2
Update documentation inline with the chunk cache additions
maxstack Mar 31, 2025
43fa08f
Fix minor typos
sd109 Apr 1, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 40 additions & 6 deletions .github/workflows/pull-request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,13 @@ jobs:
sleep 1;
done

- name: Create artifacts directory
run: mkdir artifacts

#####
# Test without local disk cache
#####

- name: Run active storage container
run: make run

Expand All @@ -108,16 +115,45 @@ jobs:
sleep 1;
done

- name: Create artifacts directory
run: mkdir artifacts

- name: Run compliance test suite
run: pytest -s > artifacts/pytest.log

- name: Get active storage logs
run: docker logs reductionist > artifacts/reductionist.log
if: always()

- name: Stop active storage container
run: make stop
if: always()

#####
# Test with local disk cache
#####

- name: Run active storage container with local disk cache
run: make run-with-cache

- name: Wait for active storage server to start
run: |
until curl -if http://localhost:8080/.well-known/reductionist-schema; do
sleep 1;
done

- name: Run compliance test suite
run: pytest -s > artifacts/pytest-with-cache.log

- name: Get active storage logs
run: docker logs reductionist > artifacts/reductionist-with-cache.log
if: always()

- name: Stop active storage container
run: make stop
if: always()

#####
# Clean up steps
#####

- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
Expand All @@ -129,9 +165,6 @@ jobs:
run: scripts/minio-stop
if: always()

- name: Stop active storage container
run: make stop
if: always()
deployment-test:
runs-on: ubuntu-latest
steps:
Expand Down Expand Up @@ -183,6 +216,7 @@ jobs:
sudo ip a
sudo ip r
if: failure()

dependency-review:
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
Expand Down
Loading
Loading