support pnp resolver #1876

gun-yu · 2025-10-11T18:56:59Z

This PR adds support for PnP resolution.

#1875 in the previous discussion, I have internalized pnp-go into the repository.

Please note that the PR has become quite large due to the inclusion of the PnP source and test code. I apologize for the length and appreciate your understanding.

what is pnp

Yarn Plug’n’Play (PnP) is a dependency resolution system that removes the need for a traditional node_modules folder.
describe at #460

how to support

add pnp resolution config in host

type host struct {
	orchestrator *Orchestrator
	host         compiler.CompilerHost
	// Caches that last only for build cycle and then cleared out
	extendedConfigCache tsc.ExtendedConfigCache
	sourceFiles         parseCache[ast.SourceFileParseOptions, *ast.SourceFile]
	configTimes         collections.SyncMap[tspath.Path, time.Duration]

	// caches that stay as long as they are needed
	resolvedReferences parseCache[tspath.Path, *tsoptions.ParsedCommandLine]
	mTimes             *collections.SyncMap[tspath.Path, time.Time]
	resolvedReferences  parseCache[tspath.Path, *tsoptions.ParsedCommandLine]
	mTimes              *collections.SyncMap[tspath.Path, time.Time]
	pnpResolutionConfig *pnp.ResolutionConfig
}

add pnp branch in resolver.go

// resolveTypeReferenceDirective
// skip typeRoots because PnP knows exactly where each @types is located.
	if r.resolver.pnpResolutionConfig != nil {
		if resolvedFromNearestNodeModulesDirectory := r.loadModuleFromNearestNodeModulesDirectory(true /*typesScopeOnly*/); !resolvedFromNearestNodeModulesDirectory.shouldContinueSearching() {
			return r.createResolvedTypeReferenceDirective(resolvedFromNearestNodeModulesDirectory, true /*primary*/)
		}
	} else {
		if len(typeRoots) > 0 {
			if r.tracer != nil {
				r.tracer.write(diagnostics.Resolving_with_primary_search_path_0.Format(strings.Join(typeRoots, ", ")))
			}

// loadModuleFromNearestNodeModulesDirectory
if r.resolver.pnpResolutionConfig != nil {
			if result := r.loadModuleFromPNP(priorityExtensions, typesScopeOnly); !result.shouldContinueSearching() {
				return result
			}
		} else {
			if result := r.loadModuleFromNearestNodeModulesDirectoryWorker(priorityExtensions, mode, typesScopeOnly); !result.shouldContinueSearching() {
				return result
			}
		}

gun-yu · 2025-10-11T18:59:20Z

internal/project/ata/ata.go

 		logger.Log(fmt.Sprintf("ATA:: Installed typings %v", packageNames))
 		var installedTypingFiles []string
-		resolver := module.NewResolver(ti.host, &core.CompilerOptions{ModuleResolution: core.ModuleResolutionKindNodeNext}, "", "")
+		resolver := module.NewResolver(ti.host, &core.CompilerOptions{ModuleResolution: core.ModuleResolutionKindNodeNext}, "", "", nil)


todo: add pnp support later

gun-yu · 2025-10-11T19:07:00Z

internal/module/resolver.go

-	packageName, rest := ParsePackageName(moduleName)
-	packageDirectory := tspath.CombinePaths(nodeModulesDirectory, packageName)
-
+func (r *resolutionState) loadModuleFromSpecificNodeModulesDirectory(ext extensions, candidate string, packageDirectory string, rest string, nodeModulesDirectoryExists bool) *resolved {


I change this function's interface

gun-yu · 2025-10-11T19:17:57Z

internal/module/pnp/lib_test.go

+}
+
+// need fixtures to be yarn install and make global cache
+func TestGlobalCache(t *testing.T) {


These test cases were taken from pnp-rs, but since they are relatively complex to run and maintain.
I think it would be fine to remove them if they’re not considered necessary.
Please let me know your thoughts!

gun-yu · 2025-10-12T14:32:09Z

@microsoft-github-policy-service agree

gun-yu · 2025-10-13T01:17:58Z

internal/compiler/host.go

+	pnpResolutionConfig := TryGetPnpResolutionConfig(currentDirectory)
+
+	if pnpResolutionConfig != nil {
+		fs = pnpvfs.From(fs)


On second thought, this way I wouldn’t be able to take advantage of the cached VFS, so I’m thinking of moving its location instead.

jakebailey · 2025-10-13T04:43:45Z

You said some of this comes from pnp-rs. Is it a pure port? What is the license of that code? Do you own it? (You signed the CLA, but that may not be enough if it's not yours.)

gun-yu · 2025-10-13T07:00:55Z

@jakebailey
The original source code, pnp-rs, is licensed under the BSD 2-Clause License. As the ported code is derived from that source, it should also fall under the BSD 2-Clause License — I missed that part, thank you for pointing it out.

As far as I understand, this means I need to include the original author’s license and copyright notice in the comments.

Would this cause any issues? If everything is fine, I’ll go ahead and add the appropriate BSD 2-Clause License notice.

jakebailey · 2025-10-13T07:21:08Z

We could pull it, but would need to explicitly declare that dependence in some extra metadata to generate NOTICE.txt (something we have avoided entirely by ensuring all code was written by us from spec or tests, but it's not really different than other deps).

But you definitely need to declare that there are files derived from code under a different license.

…escript-go into feat/add-pnp-resolver

gun-yu · 2025-10-13T18:56:05Z

@jakebailey
Sorry for the late reply. I just want to confirm my understanding—would it be sufficient to directly add a notice for the BSD 2-Clause License of pnp-rs in the NOTICE.txt file?

jakebailey · 2025-10-13T19:46:18Z

No, the files containing the code need to have headers with said license. For example, these tests:

typescript-go/internal/jsnum/ryu_test.go

Line 3 in a05e479

…escript-go into feat/add-pnp-resolver

gun-yu · 2025-10-14T02:59:14Z

Thank you. I’ve added the license notice as suggested. @jakebailey
b368c1c

jakebailey

A "few" comments; my overall worry is that the code style is not so much similar to our code, but maybe that's okay.

(My reviewing it is not really saying yes or no to the PR, we haven't discussed it)

jakebailey · 2025-10-15T18:27:53Z

internal/module/pnp/utils/normalize_path.go

+	"strings"
+)
+
+func NormalizePath(original string) string {


I would think we don't need this sort of thing; the tspath package already defines the way we typically normalize paths, which is already forward slashed and I think compatible with accessing zips.

305fd49
I replaced normalizePath with tspath.normalizePath.

jakebailey · 2025-10-15T18:28:22Z

internal/module/pnp/utils/portable_path.go

+var (
+	reWindowsPath     = regexp.MustCompile(`^([a-zA-Z]:.*)$`)
+	reUNCWindowsPath  = regexp.MustCompile(`^[\/\\][\/\\](\.[\/\\])?(.*)$`)
+	rePortablePath    = regexp.MustCompile(`^\/([a-zA-Z]:.*)$`)
+	reUNCPortablePath = regexp.MustCompile(`^\/unc\/(\.dot\/)?(.*)$`)
+)


Using regexes is definitely not going to be a good idea; these are notoriously slow. I would hope we could avoid needing any of this.

305fd49
I removed this file as normalizePath has been removed.

jakebailey · 2025-10-15T18:28:47Z

internal/module/pnp/utils/portable_path.go

+}
+
+func fromPortablePath(s string) string {
+	if runtime.GOOS != "windows" {


Depending on runtime.GOOS is a bad idea; all of our other path handling is platform independent.

305fd49
I removed this file as normalizePath has been removed.

internal/module/pnp/utils/portable_path.go

internal/module/pnp/builtins.go

jakebailey · 2025-10-15T18:41:36Z

internal/vfs/pnpvfs/pnpvfs.go

+	files := make([]string, 0)
+	dirs := make([]string, 0)


make([]string, 0) is just an allocate-y version of var files []string; don't intentionally make an empty slice unless nil vs non-nil matters.

jakebailey · 2025-10-15T18:42:18Z

go.mod

 module github.com/microsoft/typescript-go

-go 1.25
+go 1.25.1


Please undo this.

jakebailey · 2025-10-15T18:43:07Z

internal/module/pnp/lib.go

+	return "", false
+}
+
+var rePNP = regexp.MustCompile(`(?s)(const[\ \r\n]+RAW_RUNTIME_STATE[\ \r\n]*=[\ \r\n]*|hydrateRuntimeState\(JSON\.parse\()'`)


This scares me a bit; because of the regex, but also because this reads JS files? Does the spec really require such a thing?

Good point! esbuild uses the JS AST for parsing. Since I was primarily focused on porting pnp-rs, I initially brought the code over as-is, but as you mentioned in another comment, there are definitely some risky areas. I believe the TypeScript Go codebase also includes a JS parser, so I’ll try using that for parsing instead.

According to the official spec, in a Node.js environment, pnp.cjs patches Node’s module resolution, so it’s executed first to override the default behavior. In Go, however, this part needs to be implemented manually.

Ultimately, parsing pnp.cjs is necessary to read the metadata, so this part seems unavoidable.
If we parse it using an AST like esbuild does, it might be a bit more reliable — would that be okay?

That makes more sense to me. Though in general I really thought that PnP had been specified to just emit a plain JSON file...

internal/module/pnp/lib.go

jakebailey · 2025-10-15T18:45:02Z

internal/module/pnp/manifest.go

+}
+
+func (r *RegexDef) compile() (*regexp.Regexp, error) {
+	return regexp.Compile(r.Source)


This will fail because regex is re2 syntax, not ECMAScript. You'd have to use regexp2 in ECMAScript mode.

That being said, is this really ever used? Does the PnP spec really require regex parsing? (How do esbuild and so on do this without pulling on totally different regex implementations?)

Thanks for the feedback!
The original ignorePatternData looks like this:

"ignorePatternData": "(^(?:\\\\.yarn\\\\/sdks(?:\\\\/(?!\\\\.{1,2}(?:\\\\/|$))(?:(?:(?!(?:^|\\\\/)\\\\.{1,2}(?:\\\\/|$)).)*?)|$))$)",

Esbuild also compiles and uses the regex in the same way. Since this is part of the Yarn PnP spec, it seems unavoidable to use a regular expression here.

For now, I’ve optimized this commit to avoid the inefficiency of compiling the regex on every use. f0ee55b

gun-yu · 2025-10-16T00:41:10Z

@jakebailey Thank you for the thoughtful review! I know feedback like this takes a lot of time, and I truly appreciate it. Reading your comments made me realize how unfamiliar I still am with the TypeScript Go codebase—I’m sorry I didn’t reference the existing code more. Consistent code style is especially important in open source, so I’m happy to adopt all of your suggestions. I’ll start by addressing your comments.

gun-yu added 3 commits October 11, 2025 21:11

add pnp resolution support

c315580

remove fmt

99dc36c

pnp internalize

25b13ac

gun-yu commented Oct 11, 2025

View reviewed changes

remove useless code

f24a170

gun-yu changed the title ~~Feat/add pnp resolver~~ support pnp resolver Oct 11, 2025

remove useless code

b4aaac8

gun-yu commented Oct 11, 2025

View reviewed changes

gun-yu and others added 3 commits October 12, 2025 04:30

use go-json-experiment/json

2373552

lint fix

f09cda5

Merge branch 'main' into feat/add-pnp-resolver

d00e36b

gun-yu added 2 commits October 12, 2025 23:43

change test data path

0a0f050

fix hereby format

94a15b7

gun-yu commented Oct 13, 2025

View reviewed changes

gun-yu added 3 commits October 13, 2025 16:40

fix hereby check:format

35d6188

fix window test

35ee2ef

Merge branch 'feat/add-pnp-resolver' of https://github.com/gun-yu/typ…

9463e11

…escript-go into feat/add-pnp-resolver

gun-yu and others added 3 commits October 14, 2025 09:54

Merge branch 'main' into feat/add-pnp-resolver

dd08ae5

add License

b368c1c

Merge branch 'feat/add-pnp-resolver' of https://github.com/gun-yu/typ…

688455e

…escript-go into feat/add-pnp-resolver

jakebailey reviewed Oct 15, 2025

View reviewed changes

gun-yu and others added 4 commits October 16, 2025 10:55

remove filepath and utils

305fd49

make regex more efficient

f0ee55b

fix lint

ef6d713

Merge branch 'main' into feat/add-pnp-resolver

8d165ca

support pnp resolver #1876

Are you sure you want to change the base?

support pnp resolver #1876

Conversation

gun-yu commented Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

what is pnp

how to support

add pnp resolution config in host

add pnp branch in resolver.go

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gun-yu commented Oct 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jakebailey commented Oct 13, 2025

Uh oh!

gun-yu commented Oct 13, 2025

Uh oh!

jakebailey commented Oct 13, 2025

Uh oh!

gun-yu commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakebailey commented Oct 13, 2025

Uh oh!

gun-yu commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakebailey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gun-yu Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gun-yu commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

gun-yu commented Oct 11, 2025 •

edited

Loading

gun-yu commented Oct 13, 2025 •

edited

Loading

gun-yu commented Oct 14, 2025 •

edited

Loading

gun-yu Oct 16, 2025 •

edited

Loading

gun-yu commented Oct 16, 2025 •

edited

Loading