Problem Analysis

The problem is related to a CI job that was causing build failures or instability, which led to it being disabled in PR #63005. The specific job that was disabled is located at:

• File: .azure/pipelines/ci-public.yml
• Line: 632

There's also a template test job that needs to work in conjunction with this disabled job:

• File: .azure/pipelines/jobs/default-build.yml
• Line: 210

The Goal

The objective is to:

1. Run local development validation tests using the steps for job template in .azure/pipelines/ci-public.yml line 603
2. Run template tests using the script from .azure/pipelines/jobs/default-build.yml line 210.
3. Ensure both 1 and 2 pass without errors.
4. Do not edit any code located in src\ProjectTemplates\Web.ProjectTemplates directory
5. Do not edit the contents of project templates tests.

Solution and Testing Instructions

Local development validation

"Run Blazor web app test script" step has to fail without errors. The prevous steps are needed to build packages for the test.

Template tests

Both build and tests are included in one command:

Check for:

1. Proper job dependencies
2. Correct artifact passing between jobs
3. Environment variables that both jobs might share
4. Timeout configurations

Testing Process

1. Monitor both jobs specifically:

   • Job 1: The re-enabled job from ci-public.yml:632
   • Job 2: The template test job from default-build.yml:210

2. Validation criteria (both must be true):

   • ✅ Both sets of commands start and run to completion
   • ✅ Both pass (no errors)
   • ✅ No timeout issues occur
   • ✅ Proper artifact flow between jobs (if applicable)
   • ✅ No resource conflicts between the jobs

Intermediate Solution Testing

If the commands don't pass with the same state of repository, follow this iterative approach focused on the specific failure:

1. Isolate the Problem

Test each command individually:

• Identify if the failure is related to the interaction between these two sets of commands

Add targeted debugging for the specific failure:

Focus logging on the exact failure scenario from issue #62851:

• Add verbose logging around the specific operations that are failing
• Log the state of shared components or dependencies that both sets of commands might affect
• Capture timing information for operations that might have race conditions
• Log any shared file system operations, database connections, or network resources

2. Failure-Specific Debugging Approaches

Based on the actual failure pattern:

• Add logging before and after the specific operations that cause the failure
• Capture stack traces and detailed error messages when the failure occurs
• Log the sequence of operations in both sets of commands to identify conflicting actions
• Monitor any shared state that could be corrupted by concurrent access

Common ASP.NET Core failure patterns to debug:

• Port conflicts when both sets of commands try to start test servers
• File locking problems with shared build artifacts or test files
• Test isolation failures when tests affect shared global state

3. Gradual Re-enabling with Failure-Focused Logging

Phase 1: Identify the Exact Failure Point

• Add detailed logging around the specific line or operation where the failure occurs
• Log input parameters and system state at the failure point
• Capture the exact error codes, exception types, and error messages
• Document whether the failure is deterministic or intermittent

Phase 2: Add Synchronization and Conflict Detection

• Log when each sets of commands starts and ends critical sections
• Add logging to detect if both sets of commands are trying to access the same resources simultaneously
• Monitor for any cleanup operations that might interfere between sets of commands
• Log dependency resolution and version conflicts

4. Advanced Debugging for the Specific Issue

Error Context Capture:

• Log the full exception details including inner exceptions
• Capture the call stack at the point of failure
• Log any configuration differences between individual vs. combined runs
• Record the exact timing when failures occur relative to both sets of commands' execution

Resource Conflict Detection:

• Log attempts to access shared resources (files, ports, databases, services)
• Monitor for cleanup conflicts where one sets of commands interferes with another's teardown
• Track any global state modifications that could affect the other sets of commands
• Log dependency injection container state if both sets of commands share DI configuration

Failure Pattern Analysis:
When sets of commands fail at specific repository state together but pass in different states:

• Compare the exact error messages and stack traces
• Analyze whether the failure happens at the same point in execution
• Check if the failure is related to test ordering or parallelization
• Monitor for any shared test fixtures or static state that gets corrupted
• Log the sequence of critical operations to identify race conditions

Issue-Specific Monitoring:
Based on the nature of the ASP.NET Core issue:

• Log test server startup and shutdown sequences
• Monitor for conflicts in test data setup and teardown
• Track any shared configuration files or environment variables
• Log HTTP client creation and disposal if web tests are involved
• Monitor for conflicts in temporary file creation or cleanup

This targeted approach focuses on identifying the root cause of the specific failure rather than general system monitoring, making it much more likely to pinpoint and resolve the actual issue.

Success Criteria

The solution is complete when:

• Both sets of commands pass in the same state of codebase
• No change was done to locations that were listed as forbidden to change

Documentation for Future Reference

Create a summary document explaining:

• What the original issue was
• Why the job was disabled
• What changes were made to re-enable it
• How to test similar issues in the future
• Monitoring points for the re-enabled functionality

This approach ensures a systematic validation of the fix while minimizing risk to the main CI pipeline.