Skip to content

feat: record Javascript error types #565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
a2937 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

feat: record Javascript error types #565

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5960da4
feat: record Javascript error types
a2937 Feb 9, 2026
15d3824
Apply suggestions from code review
a2937 Feb 10, 2026
51d36d6
broken tests
a2937 Feb 10, 2026
22a3a2e
some kind of test
a2937 Feb 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/dom-evaluator/src/dom-test-evaluator.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ export class DOMTestEvaluator implements TestEvaluator {
stack: error.stack,
...(!!error.expected && { expected: error.expected }),
...(!!error.actual && { actual: error.actual }),
name: error.name,
},
};
}
Expand Down
1 change: 1 addition & 0 deletions packages/javascript-evaluator/src/javascript-test-evaluator.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ export class JavascriptTestEvaluator implements TestEvaluator {
stack: error.stack,
...(!!error.expected && { expected: error.expected }),
...(!!error.actual && { actual: error.actual }),
name: error.name,
},
};
}
Expand Down
1 change: 1 addition & 0 deletions packages/shared/src/interfaces/test-evaluator.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ export interface TestError {
expected?: unknown;
actual?: unknown;
type?: string;
name?: string;
}

export interface Fail extends Logged {
Expand Down
2 changes: 2 additions & 0 deletions packages/tests/dom-test-evaluator.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ describe("DOMTestEvaluator", () => {
expect(result).toStrictEqual({
err: {
message: "test error",
name: "Error",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching("Error: test error"),
},
Expand All @@ -54,6 +55,7 @@ describe("DOMTestEvaluator", () => {
message: "test error: expected 'actual' to equal 'expected'",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching("Error: test error"),
name: "AssertionError",
expected: "expected",
actual: "actual",
},
Expand Down
7 changes: 7 additions & 0 deletions packages/tests/integration-tests/index.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,9 +140,10 @@
return runner.runTest("throw new Error('test error')");
}, type);

expect(result).toEqual({

Check failure on line 143 in packages/tests/integration-tests/index.test.ts

View workflow job for this annotation

GitHub Actions / Node CI

packages/tests/integration-tests/index.test.ts > Test Runner > 'python' test evaluator > should handle tests that throw errors 

AssertionError: expected { Object (err) } to deeply equal { Object (err) }

- Expected
+ Received

  {
    "err": {
      "message": "test error",
-     "name": "Error",
      "stack": StringMatching /Error: test error/,
    },
  }

 ❯ packages/tests/integration-tests/index.test.ts:143:22
err: {
message: "test error",
name: "Error",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching("Error: test error"),
},
Expand Down Expand Up @@ -236,9 +237,10 @@
return runner.runTest("throw new Error('test error')");
}, type);

expect(result).toEqual({

Check failure on line 240 in packages/tests/integration-tests/index.test.ts

View workflow job for this annotation

GitHub Actions / Node CI

packages/tests/integration-tests/index.test.ts > Test Runner > 'python' test evaluator > should NOT return error logs generated after the test has finished 

AssertionError: expected { Object (err) } to deeply equal { Object (err) }

- Expected
+ Received

  {
    "err": {
      "message": "test error",
-     "name": "Error",
      "stack": StringMatching /Error: test error/,
    },
  }

 ❯ packages/tests/integration-tests/index.test.ts:240:22
err: {
message: "test error",
name: "Error",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching("Error: test error"),
},
Expand Down Expand Up @@ -434,7 +436,7 @@
return runner.runAllTests(tests);
}, type);

expect(result).toEqual([

Check failure on line 439 in packages/tests/integration-tests/index.test.ts

View workflow job for this annotation

GitHub Actions / Node CI

packages/tests/integration-tests/index.test.ts > Test Runner > 'javascript' test evaluator > should handle running multiple tests with runAllTests 

AssertionError: expected [ Array(4) ] to deeply equal [ Array(4) ]

- Expected
+ Received

@@ -8,10 +8,11 @@
    {
      "err": {
        "actual": 1,
        "expected": 2,
        "message": "expected 1 to equal 2",
+       "name": "AssertionError",
        "stack": StringMatching /AssertionError: expected 1 to equal 2/,
      },
    },
    {
      "logs": [

 ❯ packages/tests/integration-tests/index.test.ts:439:22

Check failure on line 439 in packages/tests/integration-tests/index.test.ts

View workflow job for this annotation

GitHub Actions / Node CI

packages/tests/integration-tests/index.test.ts > Test Runner > 'dom' test evaluator > should handle running multiple tests with runAllTests 

AssertionError: expected [ Array(4) ] to deeply equal [ Array(4) ]

- Expected
+ Received

@@ -8,10 +8,11 @@
    {
      "err": {
        "actual": 1,
        "expected": 2,
        "message": "expected 1 to equal 2",
+       "name": "AssertionError",
        "stack": StringMatching /AssertionError: expected 1 to equal 2/,
      },
    },
    {
      "logs": [

 ❯ packages/tests/integration-tests/index.test.ts:439:22
{ pass: true },
{ pass: true },
{
Expand Down Expand Up @@ -583,9 +585,10 @@
return runner.runTest("");
}, type);

expect(result).toEqual({

Check failure on line 588 in packages/tests/integration-tests/index.test.ts

View workflow job for this annotation

GitHub Actions / Node CI

packages/tests/integration-tests/index.test.ts > Test Runner > 'python' test evaluator > should fail the test if the afterEach hook throws an error 

AssertionError: expected { err: { …(2) } } to deeply equal { err: { …(3) } }

- Expected
+ Received

  {
    "err": {
      "message": "afterEach error",
-     "name": "Error",
      "stack": StringMatching /Error: afterEach error/,
    },
  }

 ❯ packages/tests/integration-tests/index.test.ts:588:22
err: {
message: "afterEach error",
name: "Error",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching("Error: afterEach error"),
},
Expand Down Expand Up @@ -1033,6 +1036,7 @@
err: {
actual: "test",
expected: "tes",
name: "AssertionError",
message: "expected 'test' to equal 'tes'",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching(
Expand Down Expand Up @@ -1108,6 +1112,7 @@
expect(results[0]).toEqual({
err: {
actual: "NodeList []",
name: "AssertionError",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching(
// Yes, there's a discrepancy between the error message and the
Expand Down Expand Up @@ -1144,6 +1149,7 @@
message: expect.stringMatching(
/The element has no supported sources.|Failed to load because no supported source was found./,
),
name: "NotSupportedError",
},
});
});
Expand Down Expand Up @@ -1466,6 +1472,7 @@
err: {
actual: "Symbol(foo)",
expected: "Symbol(bar)",
name: "AssertionError",
message: "expected Symbol(foo) to equal Symbol(bar)",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching(
Expand Down
30 changes: 30 additions & 0 deletions packages/tests/javascript-test-evaluator.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ const x = 2;
message: "expected 2 to equal 1",
expected: 1,
actual: 2,
name: "AssertionError",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching("AssertionError: expected"),
},
Expand Down Expand Up @@ -68,6 +69,7 @@ const x = 1;
expect(result).toStrictEqual({
err: {
message: "test error",
name: "Error",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching("Error: test error"),
},
Expand All @@ -82,6 +84,7 @@ const x = 1;
expect(result).toStrictEqual({
err: {
message: "expected 'actual' to equal 'expected'",
name: "AssertionError",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching("AssertionError: expected"),
expected: "expected",
Expand All @@ -99,6 +102,7 @@ const x = 1;
expect(result).toStrictEqual({
err: {
message: "expected 1 to equal 2",
name: "AssertionError",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching("AssertionError: expected"),
expected: 2,
Expand Down Expand Up @@ -198,5 +202,31 @@ assert.equal(x, 1)`;
pass: true,
});
});

it("error response should include the name of IndexOfBoundsError when thrown in user code", async () => {
evaluator.init({
code: {},
source: `
x = 0;`,
});

const test = "assert.equal(x, 0)";
const result = await evaluator.runTest(test);

expect(result).toStrictEqual({
err: {
message: "x is not defined",
name: "ReferenceError",
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
stack: expect.stringMatching("ReferenceError: x"),
},
logs: [
{
level: "error",
msg: "ReferenceError: x is not defined",
},
],
});
});
});
});
Loading