Fixes to make tsc -b and incremental work for memory issues

[sheetalkamat]

• Buildinfo was on "toBuildInfo" object made it separate allocation so that it doesnt form cycle with incremental program and snapshot
• Do not store incremental program past compilation as we are storing buildInfo anyways
• handled single threaded build to build in order to ensure that it does not create a deadlock
• Handled the case where same file exists multiple times in program
• optimized to not calculate "referenceMap" and other incremental state when we are building for tsc -b for non incremental programs

[Copilot]

Pull Request Overview

This PR implements memory optimizations for TypeScript incremental compilation and build functionality. The changes focus on reducing memory usage by breaking cycles between incremental programs and snapshots, avoiding unnecessary state tracking for non-incremental builds, and ensuring proper cleanup of resources.

Key changes include:

• Refactored buildInfo allocation to be separate from incremental program snapshots
• Optimized incremental state computation to only run when needed
• Restructured build task execution to support both single-threaded and multi-threaded modes

Reviewed Changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
internal/execute/incremental/snapshottobuildinfo.go	Separates buildInfo allocation and handles duplicate files in programs
internal/execute/incremental/snapshot.go	Adds method to determine when incremental state tracking is needed
internal/execute/incremental/programtosnapshot.go	Conditionally computes incremental state based on build requirements
internal/execute/incremental/program.go	Removes MakeReadonly method and optimizes diagnostic collection
internal/execute/incremental/emitfileshandler.go	Restructures emit handling to support both incremental and non-incremental paths
internal/execute/build/orchestrator.go	Refactors build execution to handle single-threaded builds sequentially
internal/execute/build/buildtask.go	Restructures task state management and reporting
internal/execute/tsc/statistics.go	Updates statistics aggregation to work incrementally
testdata/baselines/reference/*	Updates test baselines reflecting changes in incremental behavior

[mjames-c]

FYI I tried this out (commit 21e97b927) with an uncached build of our (Canva's) monorepo TS project with 5k+ transitive project ref deps (see #1622) and the build finished (before this change it would OOM)! However it still used extreme amounts of memory (116GB of my 124GB machines RAM).

./built/local/tsgo --build ../canva/<omitted>/tsconfig.json   2180.76s user 103.13s system 1424% cpu 2:40.29 total

[andrewbranch]

This change doesn’t do anything. Before:

internal/execute/incremental/programtosnapshot.go:20:8: &toProgramSnapshot{...} escapes to heap in programToSnapshot:
internal/execute/incremental/programtosnapshot.go:20:8:   flow: to ← &{storage for &toProgramSnapshot{...}}:
internal/execute/incremental/programtosnapshot.go:20:8:     from &toProgramSnapshot{...} (spill) at internal/execute/incremental/programtosnapshot.go:20:8
internal/execute/incremental/programtosnapshot.go:20:8:     from to := &toProgramSnapshot{...} (assign) at internal/execute/incremental/programtosnapshot.go:20:5
internal/execute/incremental/programtosnapshot.go:20:8:   flow: {heap} ← to:
internal/execute/incremental/programtosnapshot.go:20:8:     from (*toProgramSnapshot).computeProgramFileChanges(to) (call parameter) at internal/execute/incremental/programtosnapshot.go:30:30

After:

internal/execute/incremental/programtosnapshot.go:24:8: &toProgramSnapshot{...} escapes to heap in programToSnapshot:
internal/execute/incremental/programtosnapshot.go:24:8:   flow: to ← &{storage for &toProgramSnapshot{...}}:
internal/execute/incremental/programtosnapshot.go:24:8:     from &toProgramSnapshot{...} (spill) at internal/execute/incremental/programtosnapshot.go:24:8
internal/execute/incremental/programtosnapshot.go:24:8:     from to := &toProgramSnapshot{...} (assign) at internal/execute/incremental/programtosnapshot.go:24:5
internal/execute/incremental/programtosnapshot.go:24:8:   flow: {heap} ← to:
internal/execute/incremental/programtosnapshot.go:24:8:     from (*toProgramSnapshot).computeProgramFileChanges(to) (call parameter) at internal/execute/incremental/programtosnapshot.go:32:31

In contrast, the similar change in programtobuildinfo.go was effectual because the buildInfo field changed to a pointer, so you’re no longer returning an interior pointer. Before:

internal/execute/incremental/snapshottobuildinfo.go:19:8: &toBuildInfo{...} escapes to heap in snapshotToBuildInfo:
internal/execute/incremental/snapshottobuildinfo.go:19:8:   flow: to ← &{storage for &toBuildInfo{...}}:
internal/execute/incremental/snapshottobuildinfo.go:19:8:     from &toBuildInfo{...} (spill) at internal/execute/incremental/snapshottobuildinfo.go:19:8
internal/execute/incremental/snapshottobuildinfo.go:19:8:     from to := &toBuildInfo{...} (assign) at internal/execute/incremental/snapshottobuildinfo.go:19:5
internal/execute/incremental/snapshottobuildinfo.go:19:8:   flow: ~r0 ← to:
internal/execute/incremental/snapshottobuildinfo.go:19:8:     from to.buildInfo (dot of pointer) at internal/execute/incremental/snapshottobuildinfo.go:52:12
internal/execute/incremental/snapshottobuildinfo.go:19:8:     from &to.buildInfo (address-of) at internal/execute/incremental/snapshottobuildinfo.go:52:9
internal/execute/incremental/snapshottobuildinfo.go:19:8:     from return &to.buildInfo (return) at internal/execute/incremental/snapshottobuildinfo.go:52:2

After:

internal/execute/incremental/snapshottobuildinfo.go:22:8: &toBuildInfo{...} does not escape

The important bit in the before dump was flow: ~r0 ← to: that shows the return value as a reason for the container struct escaping. &toProgramSnapshot{} escapes to the heap either way, but the return value doesn’t retain it before or after this change.