Merge pull request #870 from BitGo/DX-658

feat: support custom codec files in host repositories

Author: ad-world

packages/openapi-generator/README.md

@@ -5,7 +5,7 @@ API specification into an OpenAPI specification.
 
 ## Install
 
-```
+```shell
 npm install --save-dev @api-ts/openapi-generator
 ```
 
@@ -15,7 +15,7 @@ The **openapi-generator** assumes the io-ts-http `apiSpec` is exported in the to
 of the Typescript file passed as an input parameter. The OpenAPI specification will be
 written to stdout.
 
-```
+```shell
 ARGUMENTS:
   <file> - API route definition file
 
@@ -35,32 +35,125 @@ For example:
 npx openapi-generator src/index.ts
 ```
 
-## Custom codec file
-
-`openapi-generator` only reads files in the specified package, and stops at the module
-boundary. This allows it to work even without `node_modules` installed. It has built-in
-support for `io-ts`, `io-ts-types`, and `@api-ts/io-ts-http` imports. If your package
-imports codecs from another external library, then you will have to define them in a
-custom configuration file so that `openapi-generator` will understand them. To do so,
-create a JS file with the following format:
-
-```typescript
-module.exports = (E) => {
-  return {
-    'io-ts-bigint': {
-      BigIntFromString: () => E.right({ type: 'string' }),
-      NonZeroBigInt: () => E.right({ type: 'number' }),
-      NonZeroBigIntFromString: () => E.right({ type: 'string' }),
-      NegativeBigIntFromString: () => E.right({ type: 'string' }),
-      NonNegativeBigIntFromString: () => E.right({ type: 'string' }),
-      PositiveBigIntFromString: () => E.right({ type: 'string' }),
-    },
-    // ... and so on for other packages
-  };
-};
-```
+## Preparing a types package for reusable codecs
+
+In order to use types from external `io-ts` types packages, you must ensure two things
+are done.
+
+1. The package source code must be included in the bundle, as the generator is built to
+   generate specs based from the Typescript AST. It is not set up to work with
+   transpiled js code. You can do this by modifying your `package.json` to include your
+   source code in the bundle. For example, if the source code is present in the `src/`
+   directory, then add `src/` to the files array in the `package.json` of your project.
+2. After Step 1, change the `types` field in the `package.json` to be the entry point of
+   the types in the source code. For example, if the entrypoint is `src/index.ts`, then
+   set `"types": "src/index.ts"` in the `package.json`
+
+## Defining Custom Codecs
+
+When working with `openapi-generator`, you may encounter challenges with handling custom
+codecs that require JavaScript interpretation or aren't natively supported by the
+generator. These issues typically arise with codecs such as `new t.Type(...)` and other
+primitives that aren't directly supported. However, there are two solutions to address
+these challenges effectively. Click [here](#list-of-supported-io-ts-primitives) for the
+list of supported primitives.
+
+### Solution 1: Defining Custom Codec Schemas in the Types Package (recommended)
+
+`openapi-generator` now offers the ability to define the schema of custom codecs
+directly within the types package that defines them, rather than the downstream package
+that uses them. This approach is particularly useful for codecs that are used in many
+different types packages. Here’s how you can define schemas for your custom codecs in
+the upstream repository:
+
+1. Create a file named `openapi-gen.config.js` in the root of your repository.
+
+2. Add the following line to the `package.json` of the types package:
+
+   ```json
+   "customCodecFile": "openapi-gen.config.js"
+   ```
+
+   You must also add `"openapi-gen.config.js"` to the files field in the package.json,
+   so that it is included in the final bundle.
+
+3. In the `openapi-gen.config.js` file, define your custom codecs:
+
+   ```javascript
+   module.exports = (E) => {
+     return {
+       SampleCodecDefinition: () =>
+         E.right({
+           type: 'string',
+           default: 'defaultString',
+           minLength: 1,
+         }),
+       // ... rest of your custom codec definitions
+     };
+   };
+   ```
+
+By following these steps, the schemas for your custom codecs will be included in the
+generated API docs for any endpoints that use the respective codecs. The input parameter
+`E` is the namespace import of `fp-ts/Either`, and the return type should be a `Record`
+containing AST definitions for external libraries. For more details, see
+[KNOWN_IMPORTS](./src/knownImports.ts).
+
+### Solution 2: Using a Custom Codec Configuration File
+
+`openapi-generator` supports importing codecs from other packages in `node_modules`, but
+it struggles with `io-ts` primitives that need JavaScript interpretation, such as
+`new t.Type(...)`. To work around this, you can define schemas for these codecs in a
+configuration file within your downstream types package (where you generate the API
+docs). This allows the generator to understand and use these schemas where necessary.
+Follow these steps to create and use a custom codec configuration file:
+
+1. Create a JavaScript file with the following format:
+
+   ```javascript
+   module.exports = (E) => {
+     return {
+       'io-ts-bigint': {
+         BigIntFromString: () => E.right({ type: 'string' }),
+         NonZeroBigInt: () => E.right({ type: 'number' }),
+         NonZeroBigIntFromString: () => E.right({ type: 'string' }),
+         NegativeBigIntFromString: () => E.right({ type: 'string' }),
+         NonNegativeBigIntFromString: () => E.right({ type: 'string' }),
+         PositiveBigIntFromString: () => E.right({ type: 'string' }),
+       },
+       // ... and so on for other packages
+     };
+   };
+   ```
+
+2. The input parameter `E` is the namespace import of `fp-ts/Either`, which avoids
+   issues with `require`. The return type should be a `Record` containing AST
+   definitions for external libraries. For more information on the structure, refer to
+   [KNOWN_IMPORTS](./src/knownImports.ts).
+
+## List of supported io-ts primitives
 
-The input parameter `E` is the namespace import of `fp-ts/Either` (so that trying to
-`require` it from the config file isn't an issue), and the return type is a `Record`
-containing AST definitions for external libraries.
-[Refer to KNOWN_IMPORTS here for info on the structure](./src/knownImports.ts)
+- string
+- number
+- bigint
+- boolean
+- null
+- nullType
+- undefined
+- unknown
+- any
+- array
+- readonlyArray
+- object
+- type
+- partial
+- exact
+- strict
+- record
+- union
+- intersection
+- literal
+- keyof
+- brand
+- UnknownRecord
+- void

packages/openapi-generator/src/cli.ts

@@ -125,7 +125,7 @@ const app = command({
       } else if (!isApiSpec(entryPoint, symbol.init.callee)) {
         continue;
       }
-      logInfo(`[INFO] Found API spec in ${symbol.name}`);
+      logInfo(`Found API spec in ${symbol.name}`);
 
       const result = parseApiSpec(
         project.right,
@@ -171,17 +171,17 @@ const app = command({
 
         const initE = findSymbolInitializer(project.right, sourceFile, ref.name);
         if (E.isLeft(initE)) {
-          console.error(
-            `[ERROR] Could not find symbol '${ref.name}' in '${ref.location}': ${initE.left}`,
+          logError(
+            `Could not find symbol '${ref.name}' in '${ref.location}': ${initE.left}`,
           );
           process.exit(1);
         }
         const [newSourceFile, init, comment] = initE.right;
 
         const codecE = parseCodecInitializer(project.right, newSourceFile, init);
         if (E.isLeft(codecE)) {
-          console.error(
-            `[ERROR] Could not parse codec '${ref.name}' in '${ref.location}': ${codecE.left}`,
+          logError(
+            `Could not parse codec '${ref.name}' in '${ref.location}': ${codecE.left}`,
           );
           process.exit(1);
         }

packages/openapi-generator/src/project.ts

@@ -6,7 +6,7 @@ import resolve from 'resolve';
 
 import { KNOWN_IMPORTS, type KnownCodec } from './knownImports';
 import { parseSource, type SourceFile } from './sourceFile';
-import { errorLeft } from './error';
+import { errorLeft, logInfo } from './error';
 
 const readFile = promisify(fs.readFile);
 
@@ -37,6 +37,7 @@ export class Project {
   async parseEntryPoint(entryPoint: string): Promise<E.Either<string, Project>> {
     const queue: string[] = [entryPoint];
     let path: string | undefined;
+    const visitedPackages = new Set<string>();
     while (((path = queue.pop()), path !== undefined)) {
       if (!['.ts', '.js'].includes(p.extname(path))) {
         continue;
@@ -59,6 +60,26 @@ export class Project {
           // If we are not resolving a relative path, we need to resolve the entry point
           const baseDir = p.dirname(sourceFile.path);
           let entryPoint = this.resolveEntryPoint(baseDir, sym.from);
+
+          if (!visitedPackages.has(sym.from)) {
+            // This is a step that checks if this import has custom codecs, and loads them into known imports
+            const codecs = await this.getCustomCodecs(baseDir, sym.from);
+            if (E.isLeft(codecs)) {
+              return codecs;
+            }
+
+            if (Object.keys(codecs.right).length > 0) {
+              this.knownImports[sym.from] = {
+                ...codecs.right,
+                ...this.knownImports[sym.from],
+              };
+
+              logInfo(`Loaded custom codecs for ${sym.from}`);
+            }
+          }
+
+          visitedPackages.add(sym.from);
+
           if (E.isLeft(entryPoint)) {
             continue;
           } else if (!this.has(entryPoint.right)) {
@@ -148,4 +169,45 @@ export class Project {
   getTypes() {
     return this.types;
   }
+
+  private async getCustomCodecs(
+    basedir: string,
+    packageName: string,
+  ): Promise<E.Either<string, Record<string, KnownCodec>>> {
+    let packageJsonPath = '';
+
+    try {
+      packageJsonPath = resolve.sync(`${packageName}/package.json`, {
+        basedir,
+        extensions: ['.json'],
+      });
+    } catch (e) {
+      // This should not lead to the failure of the entire project, so return an empty record
+      return E.right({});
+    }
+
+    const packageInfo = JSON.parse(fs.readFileSync(packageJsonPath, 'utf8'));
+
+    if (packageInfo['customCodecFile']) {
+      // The package defines their own custom codecs
+      const customCodecPath = resolve.sync(
+        `${packageName}/${packageInfo['customCodecFile']}`,
+        {
+          basedir,
+          extensions: ['.ts', '.js'],
+        },
+      );
+
+      const module = await import(customCodecPath);
+      if (module.default === undefined) {
+        // Package does not have a default export so we can't use it. Format of the custom codec file is incorrect
+        return errorLeft(`Could not find default export in ${customCodecPath}`);
+      }
+
+      const customCodecs = module.default(E);
+      return E.right(customCodecs);
+    }
+
+    return E.right({});
+  }
 }

packages/openapi-generator/src/sourceFile.ts

@@ -1,6 +1,7 @@
 import * as swc from '@swc/core';
 
 import { parseTopLevelSymbols, type SymbolTable } from './symbol';
+import { logWarn } from './error';
 
 export type SourceFile = {
   path: string;
@@ -41,7 +42,7 @@ export async function parseSource(
       span: module.span,
     };
   } catch (e: unknown) {
-    console.error(`Error parsing source file: ${path}`, e);
+    logWarn(`Error parsing source file: ${path}, ${e}`);
     return undefined;
   }
 }

packages/openapi-generator/test/externalModuleApiSpec.test.ts

@@ -126,8 +126,8 @@ async function testCase(
       components,
     );
 
-    assert.deepEqual(errors, expectedErrors);
-    assert.deepEqual(openapi, expected);
+    assert.deepStrictEqual(errors, expectedErrors); 
+    assert.deepStrictEqual(openapi, expected);
   });
 }
 
@@ -319,3 +319,48 @@ testCase(
   },
   []
 )
+
+testCase("simple api spec with custom codec", "test/sample-types/apiSpecWithCustomCodec.ts", {
+  openapi: "3.0.3",
+  info: {
+    title: "simple api spec with custom codec",
+    version: "4.7.4",
+    description: "simple api spec with custom codec"
+  },
+  paths: {
+    "/test": {
+      get: {
+        parameters: [],
+        responses: {
+          200: {
+            description: "OK",
+            content: {
+              'application/json': {
+                schema: {
+                  type: 'string',
+                  description: 'Sample custom codec',
+                  example: 'sample',
+                  format: 'sample'
+                }
+              }
+            }
+          },
+          201: {
+            description: 'Created',
+            content: {
+              'application/json': {
+                schema: {
+                  type: 'number',
+                  description: 'Another sample codec',
+                }
+              }
+            }
+          }
+        }
+      }
+    }
+  },
+  components: {
+    schemas: {}
+  }
+}, []);

packages/openapi-generator/test/sample-types/apiSpecWithCustomCodec.ts

@@ -0,0 +1,16 @@
+import { SampleCustomCodec, AnotherSampleCodec } from '@bitgo/custom-codecs';
+import * as h from '@api-ts/io-ts-http';
+
+export const apiSpec = h.apiSpec({
+    'api.get.test': {
+        get: h.httpRoute({
+            path: '/test',
+            method: 'GET',
+            request: h.httpRequest({}),
+            response: {
+                200: SampleCustomCodec,
+                201: AnotherSampleCodec,
+            },
+        }),
+    },
+})