-
Notifications
You must be signed in to change notification settings - Fork 2.9k
HybridCache: Add locks to prevent potential SQL Server deadlocks #21294
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Add ReadLock to database read operations in HybridCache services to prevent deadlocks during concurrent content/media save operations. This extends the fix from PR #21156 to other methods that were missing lock protection. Methods updated with ReadLock: - DocumentCacheService: GetNodeAsync, GetByContentType, SeedAsync, RebuildMemoryCacheByContentTypeAsync - MediaCacheService: GetNodeAsync, GetByContentType, SeedAsync, RefreshMemoryCacheAsync, RebuildMemoryCacheByContentTypeAsync - ContentTypeSeedKeyProvider: GetSeedKeys
47646e5 to
751d70d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR extends the SQL Server deadlock prevention mechanism from PR #21156 to additional database read operations in the HybridCache project. The changes add ReadLock calls to protect concurrent database reads from conflicting with write operations (such as content/media saves), preventing potential SQL Server deadlocks under heavy load.
Key changes:
- Added
ReadLockprotection to cache miss callbacks, seeding operations, and memory cache rebuild methods - Ensured consistent lock acquisition pattern across Document, Media, and ContentType seed providers
- Added necessary import (
Umbraco.Cms.Core) for Constants.Locks access in ContentTypeSeedKeyProvider
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| MediaCacheService.cs | Added ReadLock to GetNodeAsync cache miss callback, SeedAsync, RefreshMemoryCacheAsync, RebuildMemoryCacheByContentTypeAsync, and GetByContentType methods to protect database read operations |
| DocumentCacheService.cs | Added ReadLock to GetNodeAsync cache miss callback, GetByContentType, SeedAsync, and RebuildMemoryCacheByContentTypeAsync methods to protect database read operations |
| ContentTypeSeedKeyProvider.cs | Added ReadLock to GetSeedKeys method and imported Umbraco.Cms.Core namespace for Constants.Locks access |
|
@AndyButland Do you have a large database we could test this with? For reference, these changes were suggested by Claude when @Migaroez asked it to find possible similar issues to #21156 in the hybrid cache project. |
|
The updates all look sensible to me @lauraneto. I've shared a large test database with you, so if that's able to show the problems on |
| public ISet<Guid> GetSeedKeys() | ||
| { | ||
| using ICoreScope scope = _scopeProvider.CreateCoreScope(); | ||
| scope.ReadLock(Constants.Locks.ContentTree); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure that this lock should be necessary, or, if we do need one, should it be a new lock ID that is dedicated for reads from the cmsContentNu table? What I'm thinking is that it shouldn't be necessary to lock reads for the content tree - which will be umbracoNode, umbracoContent, umbracoDocument etc., to read from a table that's unrelated in the sense of the database schema.
I think the same applies for all the others added in the PR.
So - of course if in the testing you've since done or can do with a large database you see improvements, then probably we should discount my point noted here. But if we can't see any issue, then I'm not sure there's value in adding these locks as they are.
Summary
This PR extends the deadlock prevention fix from #21156 to other methods in the HybridCache project that could potentially cause similar deadlocks under concurrent load.
Add
ReadLockto database read operations inDocumentCacheService,MediaCacheService, andContentTypeSeedKeyProviderto prevent SQL Server deadlocks when these read operations compete with concurrent content/media save operations.Methods updated with ReadLock
DocumentCacheService:
GetNodeAsync(cache miss callback)GetByContentTypeSeedAsyncRebuildMemoryCacheByContentTypeAsyncMediaCacheService:
GetNodeAsync(cache miss callback)GetByContentTypeSeedAsyncRefreshMemoryCacheAsyncRebuildMemoryCacheByContentTypeAsyncContentTypeSeedKeyProvider:
GetSeedKeysTesting
Deadlocks are race conditions that are difficult to reproduce reliably. They depend on timing, SQL Server lock escalation, and concurrent operation patterns. There are no specific reproduction steps - the issue manifests under heavy concurrent load on SQL Server.