Skip to content

[APIView] Fix background service crashing APIView #13469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tjprescott wants to merge 2 commits into from
Closed

[APIView] Fix background service crashing APIView #13469

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a34f1b3
Fix background service crashing APIView.
tjprescott Jan 6, 2026
713e0ae
Add error handling to background services.
tjprescott Jan 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 31 additions & 8 deletions src/dotnet/APIView/APIViewWeb/HostedServices/CopilotPollingBackgroundHostedService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,41 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken)

while (!stoppingToken.IsCancellationRequested)
{
if (_pollingJobQueueManager.TryDequeue(out AIReviewJobInfoModel jobInfo))
try
{
var task = Task.Run(async () =>
if (_pollingJobQueueManager.TryDequeue(out AIReviewJobInfoModel jobInfo))
{
await _jobProcessor.ProcessJobAsync(jobInfo, stoppingToken);
}, stoppingToken);
var task = Task.Run(async () =>
{
try
{
await _jobProcessor.ProcessJobAsync(jobInfo, stoppingToken);
}
catch (Exception ex) when (!(ex is OperationCanceledException))
{
_logger.LogError(ex, "Error processing Copilot job {JobId} for revision {RevisionId}", jobInfo?.JobId, jobInfo?.APIRevision?.Id);
}
}, stoppingToken);

runningTasks.Add(task);
}

runningTasks.Add(task);
runningTasks.RemoveAll(t => t.IsCompleted);
await Task.Delay(1000, stoppingToken);
}
catch (Exception ex) when (!(ex is OperationCanceledException))
{
_logger.LogError(ex, "Error in CopilotPollingBackgroundHostedService main loop");
// Brief delay to avoid tight loop on persistent errors
try
{
await Task.Delay(5000, stoppingToken);
}
catch (OperationCanceledException)
{
break;
}
}

runningTasks.RemoveAll(t => t.IsCompleted);
await Task.Delay(1000, stoppingToken);
}

try
Expand Down
59 changes: 36 additions & 23 deletions src/dotnet/APIView/APIViewWeb/HostedServices/LinesWithDiffBackgroundHostedService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,34 +39,47 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
if (!_isDisabled)
{
var requestTelemetry = new RequestTelemetry { Name = "Computing Line Number of Sections with Diff" };
var operation = _telemetryClient.StartOperation(requestTelemetry);
try
while (!stoppingToken.IsCancellationRequested)
{
var reviews = await _reviewManager.GetReviewsAsync(language: ApiViewConstants.SwaggerLanguage);
reviews = reviews.OrderBy(r => r.CreatedOn).Reverse();
int index = 1;
int total = reviews.Count();
foreach (var review in reviews)
var requestTelemetry = new RequestTelemetry { Name = "Computing Line Number of Sections with Diff" };
var operation = _telemetryClient.StartOperation(requestTelemetry);
try
{
_telemetryClient.TrackTrace($"Computing Line Number of Sections with Diff for Review {review.Id}, processing {index}/{total}.");
var apiRevisions = await _apiRevisionManager.GetAPIRevisionsAsync(reviewId: review.Id);
var processedRevisions = new HashSet<string>();
foreach (var apiRevision in apiRevisions)
var reviews = await _reviewManager.GetReviewsAsync(language: ApiViewConstants.SwaggerLanguage);
reviews = reviews.OrderBy(r => r.CreatedOn).Reverse();
int index = 1;
int total = reviews.Count();
foreach (var review in reviews)
{
processedRevisions.Add(apiRevision.Id);
await _apiRevisionManager.GetLineNumbersOfHeadingsOfSectionsWithDiff(reviewId: review.Id, apiRevision: apiRevision, apiRevisions: apiRevisions.Where(r => !processedRevisions.Contains(r.Id)));
_telemetryClient.TrackTrace($"Computing Line Number of Sections with Diff for Review {review.Id}, processing {index}/{total}.");
var apiRevisions = await _apiRevisionManager.GetAPIRevisionsAsync(reviewId: review.Id);
var processedRevisions = new HashSet<string>();
foreach (var apiRevision in apiRevisions)
{
processedRevisions.Add(apiRevision.Id);
await _apiRevisionManager.GetLineNumbersOfHeadingsOfSectionsWithDiff(reviewId: review.Id, apiRevision: apiRevision, apiRevisions: apiRevisions.Where(r => !processedRevisions.Contains(r.Id)));
}
index++;
}
index++;
break; // Exit the loop after successful completion
}
catch (Exception ex) when (!(ex is OperationCanceledException))
{
_telemetryClient.TrackException(ex);
// Wait before retrying to avoid tight loop on persistent errors
try
{
await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
}
catch (OperationCanceledException)
{
break;
}
}
finally
{
_telemetryClient.StopOperation(operation);
}
}
catch (Exception ex)
{
_telemetryClient.TrackException(ex);
}
finally
{
_telemetryClient.StopOperation(operation);
}
}
}
Expand Down
27 changes: 20 additions & 7 deletions src/dotnet/APIView/APIViewWeb/HostedServices/ReviewBackgroundHostedService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,14 +72,27 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
if (!_isDisabled)
{
try
{
await _reviewManager.UpdateReviewsInBackground(_upgradeDisabledLangs, _backgroundBatchProcessCount, _isUpgradeTestEnabled, _packageNameFilterForUpgrade);
await ArchiveInactiveAPIReviews(stoppingToken, _autoArchiveInactiveGracePeriodMonths);
}
catch (Exception ex)
while (!stoppingToken.IsCancellationRequested)
Copy link
Member

@praveenkuttappan praveenkuttappan Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we running this in loop? This is a onetime task to update reviews when a new version is deployed. right?

{
_telemetryClient.TrackException(ex);
try
{
await _reviewManager.UpdateReviewsInBackground(_upgradeDisabledLangs, _backgroundBatchProcessCount, _isUpgradeTestEnabled, _packageNameFilterForUpgrade);
await ArchiveInactiveAPIReviews(stoppingToken, _autoArchiveInactiveGracePeriodMonths);
break; // Exit the loop after successful completion
}
catch (Exception ex) when (!(ex is OperationCanceledException))
{
_telemetryClient.TrackException(ex);
// Wait before retrying to avoid tight loop on persistent errors
try
{
await Task.Delay(TimeSpan.FromMinutes(5), stoppingToken);
}
catch (OperationCanceledException)
{
break;
}
}
}
}
}
Expand Down
7 changes: 7 additions & 0 deletions src/dotnet/APIView/APIViewWeb/Startup.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,13 @@ public Startup(IConfiguration configuration, IWebHostEnvironment environment)
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
// Configure host options to prevent background service exceptions from crashing the app
services.Configure<HostOptions>(options =>
{
options.BackgroundServiceExceptionBehavior = BackgroundServiceExceptionBehavior.Ignore;
Copy link

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting BackgroundServiceExceptionBehavior.Ignore silently suppresses all exceptions from background services, which masks the root cause instead of fixing it. This approach has several operational concerns:

  1. Loss of observability: Exceptions that crash the app are visible and actionable. Ignored exceptions may go unnoticed until they cause data inconsistencies or silent failures.

  2. Existing exception handling: All background services in this codebase already have proper try-catch blocks with telemetry tracking (e.g., CopilotPollingBackgroundHostedService lines 50-57, ReviewBackgroundHostedService lines 75-83, QueuedHostedService lines 43-46). If the app is still crashing, it suggests exceptions are escaping these handlers, which indicates a bug that should be fixed rather than suppressed.

  3. Better alternatives:

    • Investigate which specific background service is throwing unhandled exceptions and fix the exception handling in that service
    • Use BackgroundServiceExceptionBehavior.StopHost (the default) to maintain fail-fast behavior while fixing the root cause
    • If you must use Ignore temporarily, add comprehensive logging to track when exceptions occur

Recommendation: Instead of ignoring exceptions globally, identify the specific background service causing crashes and add proper exception handling there. This maintains observability and follows the existing pattern used in the other services.

Copilot uses AI. Check for mistakes.
Copy link
Member

@AlitzelMendez AlitzelMendez Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think that the best things would be actually identity from which line/call/process the exceptions is being thrown and see if we can fix it, in this case ignoring exception behavior will make debugging harder in the future, I will try to find the root cause

options.ShutdownTimeout = TimeSpan.FromSeconds(30);
});

services.AddApplicationInsightsTelemetry();
services.AddApplicationInsightsTelemetryProcessor<TelemetryIpAddressFilter>();
services.AddAzureAppConfiguration();
Expand Down