Skip to content

Commit 51c9e1a

Browse files
antonsyndclaude
andcommitted
test: exclude flaky wall-clock performance tests from CI and default test runs
Timing-based tests (CachedDiscoveryPerformanceTests, cancellation promptness checks) are inherently flaky under CI load. Tag them with [Trait("Category", "Benchmark")] and filter them out of the default `dotnet test` run in CI and the /run-tests skill. They can still be run explicitly with `--filter "Category=Benchmark"`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent d0f9a5e commit 51c9e1a

5 files changed

Lines changed: 9 additions & 3 deletions

File tree

.claude/skills/run-tests/SKILL.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,8 +28,8 @@ Run Sharpy tests with optional filter. Output is smart-truncated to avoid token
2828
1. Run `mkdir -p .claude/tmp` to ensure log directory exists
2929
2. Clear the old log with `rm -f .claude/tmp/last-test-run.log`
3030
3. Build first: `dotnet build sharpy.sln --nologo -v q >> .claude/tmp/last-test-run.log 2>&1`. If build fails, print "=== BUILD FAILED — cannot run tests ===" then `tail -30 .claude/tmp/last-test-run.log` and stop.
31-
4. If `$ARGUMENTS` is non-empty, run: `dotnet test --filter "FullyQualifiedName~$ARGUMENTS" --no-build --logger "console;verbosity=normal" >> .claude/tmp/last-test-run.log 2>&1`
32-
5. Otherwise run: `dotnet test --no-build --logger "console;verbosity=normal" >> .claude/tmp/last-test-run.log 2>&1`
31+
4. If `$ARGUMENTS` is non-empty, run: `dotnet test --filter "FullyQualifiedName~$ARGUMENTS&Category!=Benchmark" --no-build --logger "console;verbosity=normal" >> .claude/tmp/last-test-run.log 2>&1`
32+
5. Otherwise run: `dotnet test --filter "Category!=Benchmark" --no-build --logger "console;verbosity=normal" >> .claude/tmp/last-test-run.log 2>&1`
3333
6. Check exit code:
3434
- Exit 0: Print "=== TESTS PASSED ===" then `tail -20 .claude/tmp/last-test-run.log`
3535
- Exit non-zero: Print "=== TESTS FAILED (last 80 lines) ===" then `tail -80 .claude/tmp/last-test-run.log`, then echo "Full log: .claude/tmp/last-test-run.log"

.github/workflows/dotnet10.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,4 +43,4 @@ jobs:
4343
- name: Build
4444
run: dotnet build --no-restore
4545
- name: Test
46-
run: dotnet test --no-build --verbosity normal
46+
run: dotnet test --no-build --verbosity normal --filter "Category!=Benchmark"

src/Sharpy.Compiler.Tests/CancellationTests.cs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ public void Compile_WithAlreadyCancelledToken_ThrowsOperationCanceledException()
6464
}
6565

6666
[Fact]
67+
[Trait("Category", "Benchmark")]
6768
public void Compile_WithAlreadyCancelledToken_ReturnsQuickly()
6869
{
6970
// Arrange
@@ -85,6 +86,7 @@ public void Compile_WithAlreadyCancelledToken_ReturnsQuickly()
8586
}
8687

8788
[Fact]
89+
[Trait("Category", "Benchmark")]
8890
public void Compile_WithDelayedCancellation_TerminatesPromptly()
8991
{
9092
// Arrange - generate a large program that takes nontrivial time to compile

src/Sharpy.Compiler.Tests/CancellationTokenComponentTests.cs

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ public void Lexer_PreCancelledToken_WithSourceText_ThrowsDuringTokenizeAll()
5959
}
6060

6161
[Fact]
62+
[Trait("Category", "Benchmark")]
6263
public void Lexer_PreCancelledToken_LargeSource_TerminatesPromptly()
6364
{
6465
// Even with a large source, a pre-cancelled token should terminate quickly
@@ -137,6 +138,7 @@ public void Parser_PreCancelledToken_SmallProgram_CompletesWithoutCrash()
137138
}
138139

139140
[Fact]
141+
[Trait("Category", "Benchmark")]
140142
public void Parser_PreCancelledToken_LargeTokenList_TerminatesPromptly()
141143
{
142144
// Generate a large program and tokenize it
@@ -214,6 +216,7 @@ public void Compile_PreCancelledToken_DoesNotProduceGeneratedCode()
214216
}
215217

216218
[Fact]
219+
[Trait("Category", "Benchmark")]
217220
public void Compile_CancelledDuringLargeProgram_TerminatesPromptly()
218221
{
219222
// Use a token that cancels after a short delay

src/Sharpy.Compiler.Tests/Performance/CachedDiscoveryPerformanceTests.cs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ namespace Sharpy.Compiler.Tests.Performance;
1212
/// <summary>
1313
/// Performance benchmarks for the cached overload discovery system
1414
/// </summary>
15+
[Trait("Category", "Benchmark")]
1516
public class CachedDiscoveryPerformanceTests : IDisposable
1617
{
1718
private readonly ITestOutputHelper _output;

0 commit comments

Comments
 (0)