Testing Standards: Integration vs Unit Tests, TypeScript vs JavaScript, CJS vs ESM

[Ethan-Arrowood]

Context

We need to establish clear testing standards for ourselves and contributors.

Here are some important details about the state of our repo today:

• Legacy tests (from pre-open source): Mix of JS/TS, CJS/ESM, both unit and integration
  • Migration Notes
    • Migrated JS ESM based integration tests (https://github.com/HarperFast/harper/tree/main/integrationTests/apiTests)
    • Still actively migrating old unit-tests (https://github.com/HarperFast/harper/tree/main/unitTests & More and better unit tests #15)
• Starting to write new tests (Improve Harper Application Management with lockfile  #11)
• Codebase source is mix of JS/CommonJS and newer TS (compiled to CommonJS)
  • Future goals for source code is ESM, stricter types, and Node.js type stripping during development (implies executing our import-syntax TS as ESM)

This inconsistency creates an unnecessary maintenance burden, confusion for all contributors, and challenges with reliability and scalability.

Integration Tests vs Unit Tests

First, let's clarify what we mean by these test types in the context of our application platform.

Integration Tests

Integration tests verify the full application workflow - how users actually interact with the platform:

• Testing CLI commands end-to-end
• Testing Operations API with full request/response cycles
• Testing complete application lifecycle

These tests invoke the application externally, not by importing source code directly. Harper must be running for these test to work (or they may test how the Harper application as a whole boots-up, reloads, shuts down, etc.)

Unit Tests

Unit tests verify the programmatic APIs that users import and use in their code:

• Global APIs used in jsResource application code
• New plugin API
• Logger instance, Resource API, Server middleware, etc.

These tests import and call code directly, the same way users would in plugins or applications.

Why this distinction matters for our standards:

Integration tests don't depend on source code directly; thus, can freely use whatever language, module type, or test runner regardless of source code state.

Unit tests import source code directly; thus, need to consider (language/module) compatibility, and furthermore what we're actually testing (runtime behavior vs type-checked behavior)

What Should We Unit Test?

Our primary focus for unit tests should be the user-facing programmatic APIs (logger, database client, HTTP server, etc.) - the code that users import and interact with in their plugins and applications.

However, there are valid cases for unit testing internal logic:

• Complex algorithms with many edge cases
• Error conditions difficult to trigger through an integration test
• Frequently-changing core logic where fast feedback is valuable
• Internal modules heavily reused across the codebase

Solution: use your best judgement

If an internal module is simple and fully covered by integration tests, additional unit tests may not provide value. If it's complex or has hard-to-reach edge cases, unit tests are worthwhile. Integration tests should be preferred over unit tests, but more tests the merrier (as long as they aren't slow or flaky).

Proposal

Integration Tests

All integration tests should be TypeScript + ESM

• These tests invoke the application externally (don't import source code)
• Already working independently
• Aligns with our codebase direction (TypeScript + ESM migration)
• Allows us to start using Node.js Test Runner and Type Stripping today for Node.js v22+
• Node.js v20 doesn't support Type Stripping natively so we must figure out an appropriate solution; likely can use Tsx as a loader and it'll work fine.

We only have to update the existing apiTests/ to TS and all new tests can follow patterns being created in #11

Unit Tests (Two Options)

This is where it gets interesting. Unit tests need to import our source code, and we need to consider what we're actually testing.

Option 1: JavaScript + ESM

Pros:

✅ Easier to test runtime behavior without type constraints
✅ Can test "what happens when users ignore types" naturally (e.g., passing a number to a function expecting a string)
✅ Lower barrier for contributors unfamiliar with TypeScript
✅ Tests the lowest common denominator - actual runtime behavior

Cons:

❌ Miss TypeScript benefits (autocomplete, type checking, refactoring support)
❌ Doesn't align with codebase direction
❌ Less maintainable as codebase becomes more TypeScript-heavy

Option 2: TypeScript + ESM (with pragmatic runtime testing)

Pros:

✅ Aligns with codebase migration direction
✅ Better DX: autocomplete, type checking, easier refactoring
✅ Matches what we're building toward (Node.js Test Runner and Type Stripping)
✅ Can still test runtime behavior using @ts-expect-error, as any, or type escape hatches
✅ Consistency with integration test standard

Cons:

❌ Slightly more verbose for invalid input testing
❌ Requires contributors to use type escape hatches when testing runtime behavior

Example:

test('logger.info handles invalid input gracefully', () => {
  // @ts-expect-error - testing runtime behavior with wrong type
  assert.throws(() => logger.info(123)).toThrow('Expected string');
});

My thoughts

I'm leaning toward TypeScript + ESM for all tests (both integration and unit) because of the consistency throughout all aspects of the repo. Yes we might have to use some TS hacks to validate runtime behavior, but I'm okay with that as we'll still benefit from all the other aspects of TS. The slight verbosity of @ts-expect-error is a small price to pay for the benefits of TypeScript tooling and consistency.

Next Steps

• Decide on the standard - get team consensus
• Document in CONTRIBUTING.md - clear guidance for contributors
  • Create test templates/examples - show both happy path and runtime testing patterns
• Migrate opportunistically - no big rewrite needed, update tests as we touch them
  • Maybe tracking issues?
• Update CI/tooling - ensure test runner supports the chosen standard

[cb1kenobi]

+1 for TypeScript + ESM for all tests.

[kriszyp]

This is great work, fantastic guidelines and suggestions. I do have some questions/commentary about the TS suggestion:

Legacy tests (from pre-open source): Mix of JS/TS

I am not aware of any TypeScript tests that we are importing, which ones are you referring to?

And the "legacy tests" are very important part of this whole effort. While there are some old tests in there, there are also a lot of recently created, high-quality, extremely valuable tests, that we will be importing and are crucial for maintaining stability. And these should be imported as is (to preserve cherry-picking functionality, and for maximum openness and visibility of the changes we make). The effort and timeline of converting tests should be a key consideration in this (which I think has been pretty clearly outlined in the "Next Steps")

For the immediate future, before any changes, all tests will be in JavaScript. If we begin creating or migrating tests, it is at that point that we will introduce a "inconsistency" or "mix" of JS and TS. Also, applying "inconsistency" to JS vs TS is somewhat of vague concept; TS is superset of JS, not an incompatible language. It seems like one could apply the same conceptual definition of "inconsistency" to JS with classes vs JS without classes, which doesn't seem like a meaningful distinction. The slightly more meaningful distinction might be demanding that tests (new ones?, migrated ones?) meet the demands of the type checker (which has differing demands depending on the TS options).

This inconsistency creates an unnecessary maintenance burden

Can you quantify this?

As far as TypeScript, I think there are a number of reasons why TypeScript design, purpose, and goal is exactly the opposite of the goals of testing:

• TypeScript has extra build step, that is slower, unit test should facilitate fast, iterative feedback.
• TypeScript is designed to prevent mistakes, tests are intended to trigger and find them.
• TypeScript is designed to constrain the possibilities types of data types through code and push developers to follow those constraints, tests are designed to test all paths, and specifically to encourage all of those paths.
• TypeScript takes extra work. TypeScript has a real and substantial cost that we daily experience of making the type checker happy. For reusable code, TypeScript provides a valuable benefit though: it has an excellent syntax for explicitly declaring types that inform the expected flow of data types for consumers of code. Which is exactly the opposite of the role of testing. Unit tests are not code to be consumed, they are consuming other code to ensure it works. This means bearing the burden of extra work, without the benefit.
• For external APIs, we fully anticipate our code will be consumed by other TypeScript and JavaScript. JavaScript is the lowest common demoninator, and if are tests show that it works correctly with JavaScript, then by definition it will work with TypeScript. This is not true in the reverse direction.

As far as suggested benefits:

autocomplete, type checking, refactoring support

These are all fully available in JavaScript in any semi-modern IDE (at least the ones I have used), and have been for years. And with AI, requiring specific syntactical declarations with AST processing/type-checking is becoming increasingly irrelevant and unnecessary for smart code assistance.

Doesn't align with codebase direction

If the direction is source in TypeScript and tests in JS, this aligns. If it doesn't align because the tests should be in TS, that seems like begging the question.

Basically, I don't want to fall for the law of the instrument. Each tool has an appropriate use, and effective tool usage is about knowing the right and wrong situations for each tool, including TypeScript.

Unit tests verify the programmatic APIs that users import and use in their code:

This isn't related to TS, but I think there are a lot of internal modules that don't necessarily have APIs that are external and exported from the package (or available in the runtime environment for Harper apps), that are still extremely important to unit tests, and being able to test their specific functionality is of great value (and we have plenty of such tests).

For integration tests, I think it is appropriate that those that are doing the majority integration test maintenance are central in any decisions on how those tests are maintained. I will defer to @mihaiharperdb and @T-Brougham on that. If they prefer to use TS, that's certainly fine with me, I'm fine either way.

But again, I think these are excellent guidelines. I would love to start using the Node.js test runner more. And I think this provides great advice for building more and better tests.

[Ethan-Arrowood]

I am not aware of any TypeScript tests that we are importing, which ones are you referring to?

You're absolutely right; that was my mistake. I got confused by the first set of unit tests that got migrated over here (https://github.com/HarperFast/harper/tree/main/unitTests/components) and didn't realize (or manually go and check) that they, and everything else, is just JavaScript originally.

there are also a lot of recently created, high-quality, extremely valuable tests, that we will be importing and are crucial for maintaining stability. And these should be imported as is (to preserve cherry-picking functionality, and for maximum openness and visibility of the changes we make). The effort and timeline of converting tests should be a key consideration in this (which I think has been pretty clearly outlined in the "Next Steps")

Yes totally; I should have included this detail about migration in my original post. I'm 100% in support of maintaining what we have. For all the relatively new tests I definitely don't see a need to rush to rewrite them. I was primarily thinking about some of the really broken ones like the rewire based tests. Furthermore, again I should've expanded on this detail in the original post, but with our target future of TS+ESM codebase it makes all of the stubbing and mocking we do in tests complicated, if not just broken. So yes, we should still bring over new tests, but if we intend for our source code to eventually be ESM then there may be some maintenance work needed for those same CJS-compat tests at least eventually.

"inconsistency" or "mix" of JS and TS.

Can you quantify this?

The particular inconsistency I was seeing is the constant switching from JS to TS across unit tests, integration tests, and source code. It's hard for me to put it into words, but the best way I can quantify this is how we currently consider our level of TS strictness. I know the tests don't necessarily impact how strict we use TS in our source code, but since its one less place that we don't care for the quality of our source types, then its adding to the overall sentiment that we don't care about the strictness of our source TS types. I know TS doesn't have to be an all or nothing thing, but we've definitely just accepted that 700+ TS errors on build is "okay" and I think that comes from the lack of consistency.

TypeScript has extra build step

Yes, but we have (a) a relatively small TS project that builds very fast, (b) typescript building is faster than ever with incremental builds, (c) some TS tools don't even have to build anymore and are lightning fast as a result!

TypeScript is designed to prevent mistakes, tests are intended to trigger and find them.

Okay, but what about mistakes in the tests?

This means bearing the burden of extra work, without the benefit.

Until you have thousands of lines of spaghetti test code with poor consistency. Sure it still works, but I believe treating our test code with the same expectations of our source code is imperative for a large engineering team.

if are tests show that it works correctly with JavaScript, then by definition it will work with TypeScript. This is not true in the reverse direction.

I don't completely agree here. I can have unit tests in JavaScript, but if I don't actually have appropriate coverage then it doesn't matter and is no more confident than TS. I think this is where the argument about "what are unit tests actually testing" is significantly more important. (And caveat, I'm not talking about like literal code coverage; i think thats a BS metric. I'm talking about actual conceptual coverage. Like have you tried passing every primitive type to the arguments of the function you're testing? Do we have appropriate error messages covering those API entrypoints? etc.)

Basically, I don't want to fall for the law of the instrument. Each tool has an appropriate use, and effective tool usage is about knowing the right and wrong situations for each tool, including TypeScript.

And if you asked me this even a year ago, I'd likely completely agree with you. But truly, with the advent of Node.js type stripping and the simplicity of literally 0 extra tools involved to write and run Node.js tests in Typescript, it has been such a pleasant experience. I actually remember when the Node test runner first came out I immediately started experimenting with how to plug TS into the mix (using Node.js loaders) to execute pure TS Node tests. Once type stripping came out... blew my mind! So maybe I'm just a little bit baised here?

This isn't related to TS, but I think there are a lot of internal modules that don't necessarily have APIs that are external and exported from the package (or available in the runtime environment for Harper apps), that are still extremely important to unit tests, and being able to test their specific functionality is of great value (and we have plenty of such tests).

Yes thats exactly why I included the extra details on it being a judgement call. Just wanted to highlight it because I've felt like I've written a lot of unit tests that really should be integration tests.

For integration tests, I think it is appropriate that those that are doing the majority integration test maintenance are central in any decisions on how those tests are maintained.

Definitely want to hear their thoughts as well, but I do think once we make integration test patterns and tooling better, more core engineers (and external contributors too) will be able to contribute to them. Speaking from personal experience, I cannot wait to write so many new integration tests. Furthermore, since integration tests complexity is a lot higher, and they don't actually depend on the Harper source, I think it makes a stronger case to use TypeScript. I almost see our integration tests as an application on their own. So you'd want all the consistency and correctness that TS brings.

But again, I think these are excellent guidelines.

Thank you! I'm glad we are finally having this discussion and making progress on tests.

I would love to start using the Node.js test runner more. And I think this provides great advice for building more and better tests.

Agreed; Node.js test runner is awesome!

[cb1kenobi]

I don't want to reiterate when Ethan said, so I'll just say I agree with him with on exception: the Node.js test runner.

Node.js' test runner is not "awesome". It's slow and cumbersome to use. It feels half baked. I switched to it for a side project because I wanted to try it and found it super easy to get coverage reporting working, but concurrency didn't work at all. The --test-only flag is super annoying. If your tests are written in TypeScript, you will need to run the tests through tsx (don't use ts-node). It does run tests in isolation via processes. I just have a hard time recommended the Node.js test runner over the superior Vitest which has built-in support for TypeScript, benchmarking, and can run tests isolated in either a process or worker thread (worker thread being faster).