Simplify the build process

[justinfagnani]

I think there are a number of changes that can be made to the way projects are built to simplify the process and make the system easier to understand and maintain for part-time maintainers.

The current system is designed to enable streaming builds, but the builds aren't streaming in practice mainly because we must build all the project files together with the synchronous TypeScript compiler.

Instead I think we should do project file builds in a single async call, and then transform dependency files (for module resolution) on-demand.

The new build flow would roughly look like:

1. Find and parse, or generate, a package.json
2. Perform a virtual npm install
3. Fetch the types based on the node modules layout
4. Build the TypeScript files, resolving module specifiers with a compiler transform
5. Resolve module specifiers in non-TypeScript project files with es-module-lexer
6. Return the built files, syntax diagnostics, and promise of semantic diagnostics to the Playground project
7. Transform bare specifiers of npm dependencies on-demand

Steps to get there:

1. Change the typescript-builder and bare-module-transform to deal in promises of objects rather than async iterables of heterogenous build outputs. This makes it easier to know when the build is done and keeps files and diagnostics separate, simplifying their interfaces a bit. A build result is a:

   interface {
     files: File;
     diagnostics: Array<{filename: string, diagnostic: Diagnostic}>,
     // These are delivered after the syntax transforms
     semanticDiagnostics: Promise<Array<{filename: string, diagnostic: Diagnostic}>>;
   }

   1. Find the root package.json early and pass it to build steps. This lets us add any JSON parsing diagnostic in an obvious place and removes the need to pass a getPackageJson function.

2. Use NodeModulesDirectory for all Node module resolution.

   The NodeModulesDirectory is used to fetch types from the correct package versions, and it has all the information needed to do Node module resolution, but bare-module-transform doesn't use it, instead fetching package.json files from the CDN itself.

   1. Build the NodeModulesDirectory earlier, in the typescript-worker. Building it earlier is akin to doing an npm install before a build in a local dev setup. This will include building the dependency graph up out of the types fetcher as well.
   2. Refactor out a method to resolve a specifier given a NodeModulesDirectory and referrer within that directory.
   3. Add specifier resolution caching
   4. Use the specifier resolver in a typescript transform for compiled files.

3. Do not add dependencies files to the build in bare-module-transform. This seems like a mixin of concerns for the bare-module-transform to discover and fetch new nodes in the module graph. Dependency files can be transformed on the fly as they are requested. The worker can still run ahead of requests and fetch and transform static imports before they are requested.

[justinfagnani]

I think a change that would need to be done in order to not have the bare-module-transform do a recursive crawl and fetch of dependencies is to make the typescript-worker more stateful and remember the files, package-json, node modules layout, and build state (running, canceled, done, etc). Right now the PlaygroundBuild keeps track of some state on the main thread side of things.

This is because in order to resolve specifiers you need at least the package.json, and currently that means getting all the files. We don't want to send that with every request for a file. We also don't want to generate a node modules layout every request.