Convert from Sqlite to Redis, Add Bulk Requests, Queues, and Genre Caching by tsmarvin · Pull Request #274 · tsmarvin/BridgeBeats

tsmarvin · 2026-02-04T10:18:55Z

Significant overhaul to the BridgeBeats process to

Convert from Sqlite to Redis for all lookup cache opertaions
Add Bulk Lookup request pieces
Add Queues for each of the music service providers endpoints to track rate limits on a per-provider/per-endpoint basis.
Add Genre data caching for future use.
Cleanup and re-organization of the apphost setup and related projects.

Bumps caddy from `efb93f7` to `b6424b4`. --- updated-dependencies: - dependency-name: caddy dependency-version: '2' dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/setup-dotnet](https://github.com/actions/setup-dotnet) from 5.0.1 to 5.1.0. - [Release notes](https://github.com/actions/setup-dotnet/releases) - [Commits](actions/setup-dotnet@2016bd2...baa11fb) --- updated-dependencies: - dependency-name: actions/setup-dotnet dependency-version: 5.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

) * Remove FluentAssertions and convert assertions to MSTest

…sal (#237)

* Fix Apple Music link parser to properly handle query parameters

…the queuing system for requests.

…umentation for Redis migration (#242)

…encies, and credential validation (#253)

- Run formatter, fix line endings and encodings

… and comprehensive tests (#254)

In-Progress merge to reduce the number of PRs see the fan out of PRs for more context. - Add initial ATProto user OAuth configuration - Add genre caching functionality - Overhaul ATProto process to add a unified session manager to vastly reduce the number of refreshSession and getSession calls we're making - Update package lock files for CI with linux-x64 runtime (#256) - Implement Initial ATProto OAuth token exchange with DPoP and fix base branch issues - Add genre caching for playlist organization (#258) - Fix additional bugs

…#259) * - Add initial ATProto user OAuth configuration - Add genre caching functionality * Overhaul ATProto process to add a unified session manager to vastly reduce the number of refreshSession and getSession calls we're making * Update package lock files for CI with linux-x64 runtime (#256) * Initial plan * Address non-resolved review comments: OAuth methods and Dockerfile improvements Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Move ATProtoOAuthResult to its own file in Records directory Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Implement full OAuth token exchange with DPoP support Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Improve redirect URI handling with constants and validation Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Address code review feedback: sealed modifier, error handling, and redirect URI fix Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Refactor: narrow try-catch scope and simplify JWT return Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Address review feedback: validation, error handling, and DPoP improvements Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Refine error handling and nbf timing for clock skew tolerance Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Clarify error messages for token response parsing Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Add comprehensive unit tests for ATProto OAuth service and DPoP JWT creation Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Fix HttpClient anti-pattern: use IHttpClientFactory instead of storing HttpClient Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Run formatter * Enhance error handling and JWK validation, reorganize utility methods Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * refactor: standardize logging with Serilog and consolidate cache layer - Migrate all workers to Serilog via ServiceDefaults - Remove MediaLinkCacheRepository, consolidate into RedisMediaLinkCache - Enhance SagaCoordinator with comprehensive logging and retry logic - Add integration tests for cache layer - Update documentation and deployment scripts * Add resilience configuration, documentation, and TODOs for OAuth improvements Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * 1. Increase maintainability index scores by a. Breaking out shared logic into the BridgeBeats.Worker.Common project b. reducing top-level namespaces and breaking code up into methods and classes. * Implement ATProto OAuth token exchange with DPoP and fix base branch issues (#257) * Move dependabot file back out of workflows. * [WIP] Add genre caching for playlist organization (#258) * Initial plan * Fix Dockerfile ARG, encrypt DPoP key storage, add OAuth cleanup service Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Replace silent OAuth session failure with explicit NotImplementedException Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Remove old unencrypted DPoP key property and add migration Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Run initial OAuth cleanup on startup and simplify exception message Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Fix Data Protection config and scoped service dependency in cleanup service Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Fix Data Protection wiring and update comments to reflect actual behavior Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Document plain text storage reality and security risks honestly Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Fix ASP0015 issues * - Resolve CA1041 issues - Fix tests - Run formatter - Prefer string.Empty to "" * - Resolve CA1822 issues * - Resolve CA1848 and add event Ids. - Resolve IDE0305 too because I forgot to hit commit before updating these. * Additional IDE recommended improvements. * Adjustments to address MSTEST0049 * Queues seem functional * Regenerate package lock files for linux-x64 runtime Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> --------- Co-authored-by: Taylor Marvin <taylor@marvin.email> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com>

…est forgery token validation Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

…ilds (#270)

…ry test helper, break up TestUtilities file into individual class files.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

… location.

Copilot

Pull request overview

This PR converts BridgeBeats from SQLite to Redis for caching, introduces queue-based processing with per-provider rate limit tracking, adds bulk lookup capabilities, and implements genre data caching. The changes reorganize the project structure around .NET Aspire orchestration with separate worker services for each music provider.

Changes:

Replaced SQLite with Redis for all lookup cache operations and introduced distributed request deduplication
Added queue-based request processing with priority weights and per-endpoint rate limit tracking
Introduced separate worker services for Spotify, Apple Music, Tidal, Discord bot, and saga coordination
Added genre caching infrastructure for future metadata enrichment features

Reviewed changes

Copilot reviewed 123 out of 375 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/BridgeBeats.Core/Domain/Extensions/AspireServiceExtensionsLog.cs	New logging definitions for HTTP retry operations
src/BridgeBeats.Core/Domain/Extensions/AspireServiceExtensions.cs	Enhanced resilience configuration with rate limit handling and file logging
src/BridgeBeats.Contracts/packages.lock.json	Updated idunno.AtProto packages to version 1.5.0
src/BridgeBeats.Contracts/Records/*.cs	New record types for worker API requests/responses and queue management
src/BridgeBeats.Contracts/Interfaces/*.cs	New interfaces for queue management, rate limiting, and saga coordination
src/BridgeBeats.Contracts/Exceptions/RetryAfterExceededException.cs	New exception for handling excessive rate limit delays
src/BridgeBeats.Contracts/Constants/SpotifyConstants.cs	Batch operation limits for Spotify API
src/BridgeBeats.AppHost/.	Aspire orchestration configuration for Redis and worker services
docs/*.md	Updated documentation for Redis, queue system, and worker architecture
containers/.	Updated Docker configuration for Aspire AppHost and multi-service deployment
Tests/*.cs	Comprehensive test updates for Redis integration and new services

src/BridgeBeats.Core/Domain/Extensions/AspireServiceExtensions.cs

containers/Dockerfile.caddy

containers/Test-LockFileChanges.ps1

Tests/SharedTestInfrastructure.cs

…Beats into refactor-develop

- services start up and stay running

Copilot

Pull request overview

Copilot reviewed 86 out of 385 changed files in this pull request and generated no new comments.

…Beats into refactor-develop

Copilot

Pull request overview

Copilot reviewed 85 out of 385 changed files in this pull request and generated 2 comments.

Copilot · 2026-02-06T01:54:41Z

containers/entrypoint.sh

-# 2) Create new appsettings.json
-cat > /app/appsettings.json <<EOF
+# 3) Create new appsettings.json
+cat > /src/BridgeBeats.Web/appsettings.json <<EOF


The script writes to /src/BridgeBeats.Web/appsettings.json but earlier removed /app/appsettings.json. Ensure the paths are consistent with the new orchestration model. If projects are now in /src/, update the comment on line 95 to reflect the correct path being removed.

Copilot · 2026-02-06T01:54:42Z

containers/docker-compose.yml

+      - internal
+    command: >
+      sh -c "if [ -f /run/secrets/redis_password ]; then
+        redis-server --appendonly yes --requirepass \"$$(cat /run/secrets/redis_password)\";


The Redis password is read from /run/secrets/redis_password but passed directly to the command line via --requirepass. This exposes the password in process listings. Consider using Redis ACL config files or environment variables with proper escaping instead.

Suggested change

redis-server --appendonly yes --requirepass \"$$(cat /run/secrets/redis_password)\";

mkdir -p /usr/local/etc/redis &&

printf 'requirepass %s\nappendonly yes\n' \"$(cat /run/secrets/redis_password)\" > /usr/local/etc/redis/redis.conf &&

redis-server /usr/local/etc/redis/redis.conf;

Copilot

Pull request overview

Copilot reviewed 85 out of 391 changed files in this pull request and generated 2 comments.

Copilot · 2026-02-06T03:40:10Z

containers/entrypoint.sh


 # Logging configuration
-LOG_FILE_PATH="${LOG_FILE_PATH:-/app/data/logs/bridgebeats-.log}"
+LOG_DIR_PATH="${LOG_DIR_PATH:-/app/data/logs}"


The variable name changed from LOG_FILE_PATH to LOG_DIR_PATH, but the comment in docs/CONFIGURATION.md line 54 still references LOG_FILE_PATH in the configuration table. Ensure all documentation is updated to reflect this naming change.

Copilot · 2026-02-06T03:40:10Z

Tests/Unit/AppleMusicLinkParserTests.cs

@@ -0,0 +1,154 @@
+using BridgeBeats.Providers.AppleMusic;


The import references BridgeBeats.Providers.AppleMusic but based on other test files in this PR, the namespace should be BridgeBeats.Core.Domain.Providers.AppleMusic to match the new project structure mentioned in the PR description. Verify the correct namespace.

Suggested change

using BridgeBeats.Providers.AppleMusic;

using BridgeBeats.Core.Domain.Providers.AppleMusic;

Copilot

Pull request overview

Copilot reviewed 85 out of 392 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 85 out of 392 changed files in this pull request and generated 4 comments.

containers/entrypoint.sh

Copilot · 2026-02-06T05:15:28Z

src/BridgeBeats.Contracts/DTOs/ATProtoPlaylistDto.cs

+
+    /// <summary>
+    /// Ordered list of track record keys (rkeys) referencing link.bridgebeats.lookup records.
+    /// Maximum 16,384 entries.


The comment mentions a maximum of 16,384 entries but there is no validation in the code to enforce this limit. Consider adding validation in the setter or a method to check the count before adding tracks.

Copilot · 2026-02-06T05:15:28Z

Tests/Unit/RecordKeyGeneratorTests.cs

 using BridgeBeats.Contracts.Enums;
-using BridgeBeats.Infrastructure.Storage;
-using FluentAssertions;
+using BridgeBeats.Core.Infrastructure.Storage;


The namespace has changed from BridgeBeats.Infrastructure.Storage to BridgeBeats.Core.Infrastructure.Storage. Ensure all references to RecordKeyGenerator throughout the codebase have been updated to use the new namespace.

Copilot · 2026-02-06T05:15:28Z

containers/entrypoint.sh

+REDIS_HOST="${REDIS_HOST:-redis}"
+REDIS_PORT="${REDIS_PORT:-6379}"
+# Try to read Redis password from Docker secret
+REDIS_PASSWORD="$(read_secret "redis_password")"


Redis password is being read from a secrets file but is stored in an environment variable. Ensure that this environment variable is not logged or exposed in any diagnostic outputs.

wip oauth public -> confidential client update

…Beats into refactor-develop

Copilot

Pull request overview

Copilot reviewed 78 out of 397 changed files in this pull request and generated 1 comment.

Copilot · 2026-02-06T06:06:16Z

containers/entrypoint.sh

+# Read from Docker secret if available, otherwise generate a random one
+INTERNAL_SERVICE_KEY="$(read_secret "internal_service_key")"
+if [ -z "$INTERNAL_SERVICE_KEY" ]; then
+    INTERNAL_SERVICE_KEY="$(head -c 32 /dev/urandom | base64)"
+fi


Generating a random internal service key on every container start creates authentication inconsistency across container restarts and prevents worker services from authenticating with the main API. The key should be read from a Docker secret or environment variable to persist across restarts and be shared with worker containers.

Suggested change

# Read from Docker secret if available, otherwise generate a random one

INTERNAL_SERVICE_KEY="$(read_secret "internal_service_key")"

if [ -z "$INTERNAL_SERVICE_KEY" ]; then

INTERNAL_SERVICE_KEY="$(head -c 32 /dev/urandom | base64)"

fi

# Must be provided via environment variable or Docker secret to remain stable across restarts

INTERNAL_SERVICE_KEY="${INTERNAL_SERVICE_KEY:-}"

if [ -z "$INTERNAL_SERVICE_KEY" ]; then

INTERNAL_SERVICE_KEY="$(read_secret "internal_service_key")"

fi

if [ -z "$INTERNAL_SERVICE_KEY" ]; then

echo "ERROR: INTERNAL_SERVICE_KEY is not set. Configure it via environment variable or Docker secret 'internal_service_key'." >&2

exit 1

fi

tsmarvin · 2026-02-06T06:31:01Z

@copilot open a new pull request to apply changes based on the comments in this thread, this thread, and this thread

Copilot · 2026-02-06T06:31:11Z

@tsmarvin I've opened a new pull request, #285, to work on those changes. Once the pull request is ready, I'll request review from you.

…285) * Initial plan * Address review comments: fix path comment, secure Redis password, require persistent internal service key Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Improve Redis password handling: add error checking and prevent command injection Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Use head -n 1 to ensure only first line of password file is used Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Improve error handling and use safer awk print instead of printf Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Fix Redis config generation, healthcheck auth, signal handling, and secret newlines Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Add --no-auth-warning flag and clarify secret handling comment Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> * Use head -n 1 in healthcheck for consistency and security Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tsmarvin <57049894+tsmarvin@users.noreply.github.com> Co-authored-by: Taylor Marvin <Taylor@Marvin.Email>

dependabot bot and others added 23 commits January 17, 2026 19:57

Remove FluentAssertions and re-enable disabled authorization tests (#235

46b90d7

) * Remove FluentAssertions and convert assertions to MSTest

Fix code review feedback: null-forgiving operator and DbContext dispo…

135d178

…sal (#237)

Fix Apple Music link parser query parameter handling (#240)

01abbeb

* Fix Apple Music link parser to properly handle query parameters

Mostly complete conversion from sqlite to redis, and the addition of …

d50594a

…the queuing system for requests.

Fix Redis queue message parsing, add log sanitization, and update doc…

c17c7ad

…umentation for Redis migration (#242)

Fix consumer ID generation, memory streaming, normalization inconsist…

341ed95

…encies, and credential validation (#253)

- Add QR code functionality to embed endpoints.

b58a2a2

- Run formatter, fix line endings and encodings

Refactor ParseMessageId and URL extraction with shared implementation…

a44bccf

… and comprehensive tests (#254)

Fix missing namespace import for AppSettings in test files (#260)

a4d7a95

Merge branch 'develop' into feature/convert2redis_addqueue

20ceaec

Potential fix for code scanning alert no. 93: Missing cross-site requ…

7f11fbc

…est forgery token validation Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Potential fix for code scanning alert no. 89: Missing cross-site requ…

3ef1e90

…est forgery token validation Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Fix package lock file runtime identifier mismatch in CI and Docker bu…

3b2a5f8

…ilds (#270)

Implement three-tier authentication for music lookup endpoints (#271)

cc73e7d

Fix antiforgery for home controller tests, standarize using antiforge…

5b0172f

…ry test helper, break up TestUtilities file into individual class files.

Ditch unnecessary platform specific lock pieces.

5f8f4a8

Apply suggestion from @Copilot

c975b37

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update Tests/Unit/OpenGraphExtensionsTests.cs

37c2c10

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update entrypoint to ensure we're starting the process from the right…

165d8a1

… location.

Copilot AI review requested due to automatic review settings February 4, 2026 10:18

Refresh NuGet lockfile for refactor-develop dependencies (#273)

7cda39f

Copilot AI reviewed Feb 4, 2026

View reviewed changes

Copilot AI and others added 3 commits February 4, 2026 02:31

Update NuGet lock files for linux-x64 locked restore (#275)

df88285

Fix docker publish

bbc3b87

Merge branch 'refactor-develop' of https://github.com/tsmarvin/Bridge…

ad9866b

…Beats into refactor-develop

Copilot AI review requested due to automatic review settings February 4, 2026 10:35

- ports are accessible

527bf89

- services start up and stay running

Copilot AI review requested due to automatic review settings February 6, 2026 01:37

Copilot AI reviewed Feb 6, 2026

View reviewed changes

tsmarvin and others added 3 commits February 5, 2026 17:42

Merge branch 'develop' into refactor-develop

a753b38

Fixes and cleanup

84800ba

Merge branch 'refactor-develop' of https://github.com/tsmarvin/Bridge…

f2b8dfb

…Beats into refactor-develop

Copilot AI review requested due to automatic review settings February 6, 2026 01:53

Copilot AI reviewed Feb 6, 2026

View reviewed changes

tsmarvin added 2 commits February 5, 2026 19:33

Fix anti-forgery issues preventing login.

a9afea7

Fix embed CSP policy

4191461

Copilot AI review requested due to automatic review settings February 6, 2026 03:38

Copilot AI reviewed Feb 6, 2026

View reviewed changes

tsmarvin added 2 commits February 5, 2026 20:00

Actually include antiforgery.js

f262d51

Continued iteration on CSP fixes for embeds.

c79ecfa

Copilot AI review requested due to automatic review settings February 6, 2026 04:10

Copilot AI reviewed Feb 6, 2026

View reviewed changes

tsmarvin added 2 commits February 5, 2026 20:34

Fix the qr code routing

4f89235

Fix embed qrs to point at actual card not embed

3f04224

Copilot AI review requested due to automatic review settings February 6, 2026 05:14

Copilot AI reviewed Feb 6, 2026

View reviewed changes

tsmarvin and others added 4 commits February 5, 2026 21:47

Point discord worker service discovery in the right direction

36f2adf

In progress syncing saga data work

72b0d41

wip discord service auth fixes

bd57fb7

wip oauth public -> confidential client update

Merge branch 'refactor-develop' of https://github.com/tsmarvin/Bridge…

4b6f600

…Beats into refactor-develop

Copilot AI review requested due to automatic review settings February 6, 2026 06:05

Copilot AI reviewed Feb 6, 2026

View reviewed changes

Copilot AI mentioned this pull request Feb 6, 2026

Fix container security and configuration issues from PR #274 review #285

Merged

-        redis-server --appendonly yes --requirepass \"$$(cat /run/secrets/redis_password)\";
+        mkdir -p /usr/local/etc/redis &&
+        printf 'requirepass %s\nappendonly yes\n' \"$(cat /run/secrets/redis_password)\" > /usr/local/etc/redis/redis.conf &&
+        redis-server /usr/local/etc/redis/redis.conf;

	using BridgeBeats.Providers.AppleMusic;
	using BridgeBeats.Core.Domain.Providers.AppleMusic;

Conversation

tsmarvin commented Feb 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

tsmarvin commented Feb 6, 2026

Uh oh!

Copilot AI commented Feb 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants