Add test coverage for cache scitoken overrides using Xrootd.ScitokensConfig by Copilot · Pull Request #3028 · PelicanPlatform/pelican

Copilot · 2026-01-23T23:55:12Z

Changes

Reverted the addition of Cache.ScitokensUserOverride parameter because the functionality already exists via Xrootd.ScitokensConfig, which is marked as usable by both cache and origin components.

New comprehensive E2E test: TestCacheScitokensConfigOverride demonstrates:

Sets up a full federation with private reads and pulls a file through cache
Simulates origin downtime by removing namespace ads from cache
Triggers cache authz refresh by overwriting Xrootd.ScitokensConfig with unrelated issuer
Verifies data is no longer accessible through cache
Triggers another authz refresh with proper authorization for the test prefix
Verifies cached object is now accessible even without origin

This test validates that Xrootd.ScitokensConfig provides the override mechanism needed for caches to serve objects during origin downtime.

All review feedback addressed including linter fixes.

Original prompt

This section details on the original issue you should resolve

<issue_title>Allow scitoken overrides in the cache</issue_title>
<issue_description>ONLY is the cache is unregistered, we should allow the administrator to specify a scitoken overrides file that gets merged into the calculated scitoken file.

This is analogous to what is done for the origin.

When adding the new configuration parameter, mark it as runtime reloadable. Make sure the file to use (if a file is used) can indeed be changed at runtime.

The goal is to have this be a mechanism where specific caches can continue to serve objects even when the origin is in downtime. Add an integration test along those lines.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes Allow scitoken overrides in the cache #3027

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

bbockelm

It looks like Xrootd.ScitokensConfig works for both caches and origins, meaning this functionality already exists. Let's focus on providing test coverage for this fact in cache mode. I like having both a unit test and an e2e test as is done in this PR.

I'd additionally like a new E2E test that:

Sets up a full federation and pulls a file through the cache.
Places the origin in downtime and removes corresponding namespace ads.
Triggers a cache authz refresh. This can be done by overwriting the Xrootd.ScitokensConfig file with some new issuer covering an unrelated path. You can verify the refresh by seeing new data show up in the corresponding generated file.
Verify you can no longer access the data through the cache.
Trigger another cache authz refresh, this time adding the authorization information explicitly for the test prefix.
Verify you can now access the cached object, even with the origin offline.

This new test will demonstrate that the Xrootd.ScitokensConfig override is the only thing needed to access offline data in a cache.

bbockelm · 2026-01-26T20:58:03Z

e2e_fed_tests/cache_scitoken_override_test.go

+
+/***************************************************************
+*
+* Copyright (C) 2025, Pelican Project, Morgridge Institute for Research


Fixed in commit 0bdf17e

docs/parameters.yaml

e2e_fed_tests/cache_scitoken_override_test.go

xrootd/authorization.go

Copilot · 2026-01-26T21:34:44Z

It looks like Xrootd.ScitokensConfig works for both caches and origins, meaning this functionality already exists. Let's focus on providing test coverage for this fact in cache mode. I like having...

Added comprehensive E2E test as requested in commit 0bdf17e. The test:

Sets up federation and pulls file through cache
Removes namespace ads to simulate origin downtime
Triggers cache authz refresh with unrelated issuer
Verifies data is no longer accessible
Triggers another refresh with proper authorization for test prefix
Verifies cached object is accessible despite "offline" origin

The test demonstrates that Xrootd.ScitokensConfig override is sufficient for accessing offline data in a cache.

turetske

The linked task specifies that "ONLY if the cache is unregistered, we should allow the administrator to specify a scitoken overrides file that gets merged into the calculated scitoken file" which makes sense as it could potentially override security in certain places. I don't see any check for if the cache is unregistered to allow this.

Edit: Nevermind, had a discussion with Brian about this. Ignore this comment

e2e_fed_tests/cache_scitokens_config_test.go

client/handle_http.go

e2e_fed_tests/cache_scitokens_config_test.go

turetske · 2026-02-26T18:17:47Z

@copilot Please address the review comments that are tagging you.

e2e_fed_tests/cache_scitokens_config_test.go

turetske · 2026-02-26T19:31:48Z

@copilot Please fix the linter errors

Copilot · 2026-02-26T19:36:08Z

@copilot Please fix the linter errors

Fixed in commit 9122618 - removed unused assert import (all assertions were changed to require.* in earlier commits)

turetske

@bbockelm Throwing this back to you.

This helps refresh information quickly in unit tests. Includes a "floor" to prevent this from happening too quickly in production.

If the director failed - but the user specified a cache to use - continue trying the request against that cache. Use the default token; in this case, it will not select the right token automatically. This allows a cache to be leveraged and Pelican to work while everything else in the federation is 100% offline.

Shows that data can be accessed at a cache even when the origin is offline!

- Change assert.Equal/Error to require.Equal/Error for critical assertions - Increase test timeouts from 5s/10s to 20s to handle CI load - Use URL construction instead of string concatenation for cache URL - Create resource file for origin config (single-private-full.yml) - Fix tj.token.DirResp not being set when director fails with explicit caches Co-authored-by: turetske <498529+turetske@users.noreply.github.com>

All items addressed!

Copilot AI assigned Copilot and bbockelm Jan 23, 2026

Copilot started work on behalf of bbockelm January 23, 2026 23:55 View session

Copilot AI changed the title ~~[WIP] Add support for scitoken overrides in the cache~~ Add scitoken override support for site-local caches Jan 24, 2026

Copilot AI requested a review from bbockelm January 24, 2026 00:17

Copilot finished work on behalf of bbockelm January 24, 2026 00:17

bbockelm previously requested changes Jan 26, 2026

View reviewed changes

Copilot started work on behalf of bbockelm January 26, 2026 21:31 View session

Copilot AI changed the title ~~Add scitoken override support for site-local caches~~ Add test coverage for cache scitoken overrides using Xrootd.ScitokensConfig Jan 26, 2026

Copilot AI requested a review from bbockelm January 26, 2026 21:35

Copilot finished work on behalf of bbockelm January 26, 2026 21:35

bbockelm force-pushed the copilot/allow-scitoken-overrides-cache branch from 8b9d74c to 996bf60 Compare January 27, 2026 02:04

bbockelm added client Issue affecting the OSDF client cache Issue relating to the cache component user-request labels Feb 21, 2026

bbockelm requested a review from turetske February 21, 2026 16:22

bbockelm force-pushed the copilot/allow-scitoken-overrides-cache branch from 996bf60 to a259732 Compare February 21, 2026 16:58

bbockelm marked this pull request as ready for review February 21, 2026 17:17

bbockelm assigned turetske Feb 24, 2026

turetske requested changes Feb 25, 2026

View reviewed changes

turetske self-requested a review February 25, 2026 17:54

turetske requested changes Feb 26, 2026

View reviewed changes

Copilot started work on behalf of turetske February 26, 2026 18:18 View session

Copilot finished work on behalf of turetske February 26, 2026 18:22

Copilot AI requested a review from turetske February 26, 2026 18:22

turetske reviewed Feb 26, 2026

View reviewed changes

e2e_fed_tests/cache_scitokens_config_test.go Outdated Show resolved Hide resolved

Copilot started work on behalf of turetske February 26, 2026 19:18 View session

Copilot AI requested a review from turetske February 26, 2026 19:20

Copilot finished work on behalf of turetske February 26, 2026 19:20

Copilot started work on behalf of turetske February 26, 2026 19:32 View session

Copilot finished work on behalf of turetske February 26, 2026 19:37

turetske approved these changes Feb 26, 2026

View reviewed changes

bbockelm and others added 4 commits February 28, 2026 09:12

Automatically refresh the namespaces when scitokens.cfg changes

7a717d2

This helps refresh information quickly in unit tests. Includes a "floor" to prevent this from happening too quickly in production.

Provide an E2E cache test for offline origins

7f6de36

Shows that data can be accessed at a cache even when the origin is offline!

bbockelm force-pushed the copilot/allow-scitoken-overrides-cache branch from 9122618 to 1e02d4d Compare February 28, 2026 15:13

bbockelm merged commit 894da13 into main Mar 1, 2026
31 of 32 checks passed

Conversation

Copilot AI commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Comments on the Issue (you are @copilot in this section)

Uh oh!

bbockelm left a comment

Choose a reason for hiding this comment

Uh oh!

bbockelm Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI commented Jan 26, 2026

Uh oh!

turetske left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

turetske commented Feb 26, 2026

Uh oh!

Uh oh!

turetske commented Feb 26, 2026

Uh oh!

Copilot AI commented Feb 26, 2026

Uh oh!

turetske left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jan 23, 2026 •

edited

Loading

turetske left a comment •

edited

Loading