Dataset cache purge cronjob

[ponyisi]

Address #1167 , except the cache lifetime is reduced to one day.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.20%. Comparing base (f343c26) to head (c069ca4).
⚠️ Report is 10 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1211      +/-   ##
===========================================
+ Coverage    86.02%   86.20%   +0.18%     
===========================================
  Files           94       95       +1     
  Lines         3256     3284      +28     
  Branches       373      377       +4     
===========================================
+ Hits          2801     2831      +30     
+ Misses         380      378       -2     
  Partials        75       75              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull Request Overview

This PR implements a dataset cache purge cronjob that automatically marks datasets as stale after they've exceeded a configurable age threshold (default 24 hours). The implementation adds a new internal API endpoint for dataset lifecycle management and a Kubernetes CronJob to periodically trigger cache cleanup.

Key changes:

• Added new internal API endpoint /servicex/internal/dataset-lifecycle for dataset cache purging
• Introduced Kubernetes CronJob to automatically purge old cached datasets every hour
• Refactored dataset deletion logic to enable reuse between manual and automated operations

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
servicex_app/routes.py	Registers the new DatasetLifecycleOps resource endpoint
servicex_app/resources/internal/dataset_lifecycle_ops.py	New endpoint implementation that purges datasets older than specified age
servicex_app/resources/datasets/get_all.py	Extracts dataset fetching logic into reusable function
servicex_app/resources/datasets/delete_dataset.py	Extracts deletion logic into reusable function
helm/servicex/values.yaml	Adds configuration parameters for dataset lifecycle cronjob
helm/servicex/templates/dataset-lifecycle/cronjob.yaml	Defines Kubernetes CronJob for periodic dataset cache cleanup
docs/deployment/reference.md	Documents new configuration parameters

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The command should be formatted as an array with shell invocation, similar to the data-lifecycle cronjob. Change to use '/bin/sh' with '-c' flag for better error handling and consistency: command: ['/bin/sh', '-c', 'curl --request POST \"http://{{ .Release.Name }}-servicex-app:8000/servicex/internal/dataset-lifecycle?age=$LIFETIME\"']

Suggested change

	command:
	command:
	- /bin/sh
	- -c

[Copilot]

The curl command lacks error handling flags. Add '-f' (fail on HTTP errors) and '--silent' or '--show-error' for better error reporting: curl -f --silent --show-error --request POST ... This ensures the job fails appropriately when the API returns an error.

Suggested change

	- curl --request POST "http://{{ .Release.Name }}-servicex-app:8000/servicex/internal/dataset-lifecycle?age=$LIFETIME"
	- curl -f --silent --show-error --request POST "http://{{ .Release.Name }}-servicex-app:8000/servicex/internal/dataset-lifecycle?age=$LIFETIME"

[Copilot]

Pull Request Overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 9 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

@ponyisi I've opened a new pull request, #1214, to work on those changes. Once the pull request is ready, I'll request review from you.

[BenGalewsky]

Ok - glad you were able to tackle this.
Just a couple of issues:

1. Don't call methods in one resource from another. Migrate common database operations to models.py
2. Make Transform cleanup and dataset cleanup siblings in the helm values dataLifecycle properties

[BenGalewsky]

Thanks, this looks great!