[W-20742893] refactor: make cross-file reference resolution on demand #183

peternhale · 2025-12-31T21:11:38Z

Summary

This PR implements batch workspace loading with compression and on-demand cross-file reference resolution to significantly improve workspace loading performance and reduce queue pressure.

Key Changes

🚀 Batch Workspace Loading with Compression

Client-side: Implements batched file loading with ZIP compression using fflate
Server-side: Adds batch handler that decompresses and enqueues files as didOpen tasks
Configuration: Adds batchSize setting (default: 100 files per batch)
Processing: Batches are sent sequentially to prevent queue overflow
Performance: Reduces connection-level backpressure and improves UI responsiveness

⚡ On-Demand Cross-File Reference Resolution

Separation of concerns: Same-file references processed immediately during addSymbolTable, cross-file references deferred
On-demand resolution: New resolveCrossFileReferencesForFile() method for resolving cross-file references when needed
Integration: Updated DiagnosticProcessingService to resolve cross-file refs before generating diagnostics
Performance: Prevents task explosion during workspace loading while maintaining full functionality

🔧 Queue Management Optimizations

Concurrency limits: Reduced queue processing concurrency to prevent resource exhaustion
Scheduler yielding: Added yield budget to prevent event loop blocking
Queue state notifications: Moved to periodic background task with configurable interval (default: 200ms)
Priority handling: Workspace batch loading uses Priority.Normal to avoid saturating high-priority queue

🧪 Testing & Quality

New tests: Added comprehensive test for cross-file resolution verification
Test updates: Updated tests to reflect separation of same-file and cross-file resolution
Lint fixes: Removed unused imports/variables and fixed formatting issues

📊 Performance Settings UI

Added Workspace Loading section to performance settings UI
Includes controls for batchSize, maxConcurrency, yieldInterval, yieldDelayMs, and enabled

Technical Details

Architecture Changes

Workspace loading: Moved from fire-and-forget didOpen requests to structured batch requests with compression
Symbol processing: Split into same-file (immediate) and cross-file (on-demand) processing
Queue coordination: Improved scheduler initialization and queue state management

Files Changed

52 files changed, 5,314 insertions(+), 1,398 deletions(-)
Major changes in:
- workspace-loader.ts: Batch loading implementation
- WorkspaceBatchHandler.ts: Server-side batch processing
- ApexSymbolManager.ts: On-demand cross-file resolution
- DiagnosticProcessingService.ts: On-demand resolution integration

Benefits

Faster workspace loading: Reduced connection overhead and queue pressure
Better responsiveness: UI remains responsive during large workspace loads
Scalability: Handles large workspaces more efficiently
Maintainability: Clear separation between same-file and cross-file processing

Testing

All existing tests pass
New test added for cross-file resolution verification
Test expectations updated to reflect architectural changes

@W-20742893@

…based orchestration - Convert precompile, compile, lint, bundle, and package tasks from command-based to dependency-based approach - Remove hardcoded package-specific paths from root config - Add explicit cross-package dependency for copy-worker-files on apex-ls:bundle - Follows VSE monorepo pattern for better maintainability and caching

- Add ../apex-parser-ast:compile dependency to ensure parser-ast is compiled before apex-ls - Add ../lsp-compliant-services:compile dependency to ensure proper build order for TypeScript project references - Fixes CI/CD linting failure where apex-ls was compiling before its dependencies

- Change test to check for path.join pattern instead of literal 'dist/resources' string - Test now correctly validates that resourcesDir is constructed using path.join - Fixes CI test failure where test expected literal string match

All package tsconfig.json files extend the root tsconfig.base.json, so Wireit compile tasks should track changes to the base config to ensure proper cache invalidation and rebuilds when the root TypeScript config changes.

- Replace manual file copying functions with esbuild-plugin-copy plugin - Update apex-ls, apex-parser-ast, and apex-lsp-vscode-extension esbuild configs - Update tests to verify plugin configuration instead of manual copy functions - Add esbuild-plugin-copy dependency to affected packages

… hover Fix missing artifact resolution for hover requests when hovering over chained references like FileUtilities.createFile. Previously, the code would extract only the method name (createFile) instead of the full chain (FileUtilities.createFile), causing incorrect search behavior. Changes: - Updated extractReferenceAtPosition to prioritize chained references over individual references when both exist at the same position - Updated generateSearchHints to extract qualifier from chained references' chainNodes - Updated inferQualifierType to treat chained references as class references - Updated resolveQualifierInfo to extract qualifier from chainNodes - Added test cases to verify chained references are prioritized correctly

- Add missing entry points to knip config (index.ts files) - Set ignoreExportsUsedInFile to false for better unused export detection - Remove unused dev dependencies (@jest/test-sequencer, @salesforce/apex-lsp-parser-ast) - Add missing dependencies (effect, @salesforce/apex-lsp-parser-ast, @salesforce/apex-lsp-compliant-services) - Remove redundant entry patterns and non-existent file references - Remove wait from ignoreBinaries and build-config entry pattern

… for Windows compatibility

…st directory error

…p delays

…queue state callbacks

Move scheduler initialization earlier in LCSAdapter startup to ensure it's ready when didOpen events arrive. Add fast path check in LSPQueueManager to avoid redundant initialization checks.

- Add queueStateNotificationIntervalMs config setting (default: 200ms) - Remove callback invocations from processQueuedItem and controllerLoop - Create startQueueStateNotificationTask() for periodic notifications - Update LCSAdapter to start periodic task instead of registering callback - Notifications now sent at configurable intervals independent of queue processing - Simplifies code by removing throttling and race condition logic

- Add workspace state querying to references service - Enhance queue management for workspace loads - Improve hover and references processing services - Add server configuration enhancements - Update dependencies

- Reduce queue processing concurrency limits to prevent resource exhaustion - Add yield budget to scheduler loop to prevent event loop blocking - Replace microtask yields with macrotask yields for better I/O responsiveness - Simplify workspace load coordination using notifications instead of queries - Remove excessive debug logging from queue state notification task - Update default settings for better performance under load

- Add batch workspace loading mechanism using compressed ZIP files - Implement client-side batch creation, compression, and sequential sending - Add server-side batch handler for decompression and didOpen enqueueing - Add batchSize configuration option for workspace loading - Use fflate for compression/decompression with fixed level 6 - Process batches sequentially to prevent queue overflow

- Add addSymbolTableMinimal() method that skips reference processing during file open - Add resolveCrossFileReferencesForFile() for on-demand cross-file ref resolution - Update DocumentProcessingService to use minimal symbol table addition - Update DocumentSaveProcessingService to use minimal symbol table addition - Update DiagnosticProcessingService to resolve cross-file refs on-demand before diagnostics - Add resolveCrossFileReferencesForFile() to ISymbolManager interface This change prevents task explosion during workspace loading by deferring cross-file reference resolution until needed (hover, goto definition, diagnostics). Workspace loading is now faster with lower queue pressure, while maintaining full functionality through on-demand resolution.

…bose log - Add Workspace Loading section to performance settings UI - Include batchSize, maxConcurrency, yieldInterval, yieldDelayMs, and enabled settings - Remove verbose 'Queue metrics changed, calling callback' log entry

- Add comprehensive test verifying cross-file references are resolved and added to graph after resolveCrossFileReferencesForFile - Fix lint errors: remove unused imports and variables, fix formatting - Update test expectations to reflect separation of same-file and cross-file resolution

Same-file references that cannot be resolved during second-pass resolution should be skipped rather than deferred, as deferring won't help for same-file references (the target is in the same file and should have been found during second-pass). This change applies to both: - processSameFileReferenceToGraphEffect (used during addSymbolTable) - processSymbolReferenceToGraphEffect (used during on-demand resolution) Only cross-file references should be deferred, as they may be resolved when the target file is loaded. This reduces unnecessary deferred reference processing during workspace loading and improves UI responsiveness.

…integration analysis

…e status display - Remove automatic queueing of deferred references in ApexSymbolGraph.addSymbol() Deferred references should only be processed on-demand, not automatically during workspace load to maintain lazy behavior - Simplify queue status display to show queued/active/complete format instead of separate sections for each state

- Add EDA workspace performance test suite - Add EDA performance helpers for repository cloning - Fix benchmark suite cleanup and timeout handling - Improve child process security by using spawn instead of exec - Add test filtering for specific performance test suites - Add QUICK mode for faster validation runs feat: add layered symbol collection and progressive enhancement - Add FullSymbolCollectorListener wrapper combining layered listeners - Add ListenerFactory for creating appropriate listeners per service - Add ProgressiveEnhancementService for demand-driven symbol enrichment - Add trigger support and createNewInstance to all layered listeners - Update LSP services to use PublicAPISymbolListener for better performance - Update tests to use FullSymbolCollectorListener where appropriate - Export new components from package index This enables progressive enhancement of symbol tables from least to most expensive operations, improving LSP startup performance while maintaining full symbol collection capabilities when needed. feat(parser): add layered symbol listeners and separate reference collection/resolution - Add PublicAPISymbolListener, ProtectedSymbolListener, and PrivateSymbolListener for progressive symbol collection - Add ApexReferenceCollectorListener for dedicated reference capture - Add ApexReferenceResolver for separated reference resolution - Refactor ApexSymbolCollectorListener to delegate reference resolution to ApexReferenceResolver - Add compileLayered() method to CompilerService for multi-pass compilation - Add ListenerApplicationManager for managing listener dependencies - Add SymbolTable enrichment support via _detailLevel property - Update DocumentStateCache to track detail levels and parse trees - Add comprehensive tests for layered listeners and reference separation - Update performance tests to benchmark layered listener approach - Update README with new architecture documentation

.session-files/batch-workspace-loading-plan.md

kylewalke · 2026-01-06T22:07:17Z

packages/apex-ls/src/server/LCSAdapter.ts

-      async (params: LoadWorkspaceParams): Promise<LoadWorkspaceResult> => {
+    // Register workspace load completion notification handlers
+    this.connection.onNotification(
+      'apex/workspaceLoadComplete',


Not in this PR but one day, I think moving registration of custom registered endpoints in our server vs the standard ones, into their own classes for organization and to reduce the size of LCSAdapter might be a good idea. The class itself is getting pretty big and I think it might be easier to parse down the road that way.

Logged W-20804806

packages/apex-lsp-shared/src/settings/ApexSettingsUtilities.ts

kylewalke · 2026-01-06T23:13:15Z

packages/apex-ls/src/server/WorkspaceBatchHandler.ts

+/**
+ * Decode base64 string to Uint8Array
+ */
+function decodeBase64(base64: string): Uint8Array {


Is there a reason to include the Buffer version at all if the Browser env version of decoding is viable for both? That way we could remove the Buffer reference entirely from the file

packages/apex-lsp-vscode-extension/src/workspace-batch-compressor.ts

kylewalke · 2026-01-06T23:56:45Z

packages/apex-parser-ast/src/queue/priority-scheduler-utils.ts

 }

+const yieldToEventLoop = Effect.async<void>((resume) => {
+  setImmediate(() => resume(Effect.void));


can you explain what this is doing? A chained empty promise that has an immediate timeout to return to void?

I found that a simple Effect.yieldNow or Effect.sleep(0) was not really causing the scheduler to yield to the event loop. The use of set immediate enques a V8 macrotask that is handled independently, at the top of the event loop, where as yieldNow and sleep are dealt with later.

Guaranteess that the scheduler will give the event loop a chance to run.

kylewalke

Just a few questions but nothing blocking.

- Add lint:fix wireit configuration to root and all workspace packages - Fix line length violations in DiagnosticProcessingService - Fix unused variables and catch error parameters across codebase - Update npm packageManager to 11.7.0 - Replace workspace:* protocol with version numbers for npm compatibility

- Remove packageManager field requiring [email protected] - Update engines.npm from >=11.0.0 to >=10.0.0 to match Node.js 20 LTS bundled npm

- Add globalTeardown to root jest.config.cjs to ensure cleanup runs - Add timeouts to jest-teardown.js to prevent hanging on shutdown operations - Add 30-minute timeout to CI test job to catch any remaining hangs - Prevents scheduler shutdown operations from blocking test completion in CI

…dotcom/apex-language-support into phale/more-tweaks-for-refs

- Use path.resolve with __dirname to get absolute path to teardown script - Fixes issue where packages extending base config couldn't find teardown script - Ensures globalTeardown works correctly whether run from root or package directories

…meouts - Remove .perf.ts files from testMatch in apex-lsp-testbed jest config - Performance tests should only run via test:perf command, not during unit tests - Increase performance test timeouts for CI environments (10min for batch, 5min for incremental) - Prevents performance tests from causing timeouts in CI unit test runs

- Increase ResourceLoaderIntegration beforeAll timeout to 5min for CI, 3min for local - Increase PriorityScheduler idle sleep test upper bound to 150ms for CI - Accounts for slower CI environments while maintaining test validity

- Increase timeout to 5min for CI, 3min for local - Fixes timeout failures on Windows CI where full compilation takes longer - Matches timeout pattern used in ResourceLoaderIntegration tests

kylewalke · 2026-01-07T18:10:08Z

package.json

      ]
    },
+    "lint:fix": {
+      "dependencies": [


- Update 12 snapshots for textDocument/documentSymbol requests - Reflects changes in symbol structure (empty children arrays, updated selectionRange values) - All quality tests now passing

peternhale added 24 commits December 30, 2025 10:19

refactor: update lint tasks to depend on compile and use ESLint cache

7907c51

chore: add PR description for Wireit migration

45fdb5b

chore: ignore ESLint cache files

fd774cc

refactor: clean up build configs and update wireit dependencies

68c2744

fix(ci): replace bash-specific syntax in custom-services lint command…

5250685

… for Windows compatibility

fix(ci): lint only src directory in custom-services to avoid empty te…

bc887f3

…st directory error

perf: make queue state callback non-blocking to prevent scheduler loo…

921bee8

…p delays

feat: add scheduler reinitialization on settings change and throttle …

bf23d32

…queue state callbacks

refactor: initialize scheduler early before document handlers

600f4fb

Move scheduler initialization earlier in LCSAdapter startup to ensure it's ready when didOpen events arrive. Add fast path check in LSPQueueManager to avoid redundant initialization checks.

refactor: improve references processing and workspace load coordination

49c7e86

- Add workspace state querying to references service - Enhance queue management for workspace loads - Improve hover and references processing services - Add server configuration enhancements - Update dependencies

docs: add batch workspace loading implementation plan

1995678

peternhale requested a review from mshanemc December 31, 2025 21:12

peternhale added 4 commits January 2, 2026 07:29

refactor: update workspace load batch processing plan with scheduler …

e317a61

…integration analysis

kylewalke reviewed Jan 6, 2026

View reviewed changes

.session-files/batch-workspace-loading-plan.md Outdated Show resolved Hide resolved

kylewalke reviewed Jan 6, 2026

View reviewed changes

packages/apex-lsp-shared/src/settings/ApexSettingsUtilities.ts Show resolved Hide resolved

kylewalke reviewed Jan 6, 2026

View reviewed changes

packages/apex-lsp-vscode-extension/src/workspace-batch-compressor.ts Show resolved Hide resolved

kylewalke reviewed Jan 6, 2026

View reviewed changes

kylewalke approved these changes Jan 7, 2026

View reviewed changes

peternhale marked this pull request as ready for review January 7, 2026 14:07

peternhale requested a review from a team as a code owner January 7, 2026 14:07

peternhale and others added 9 commits January 7, 2026 07:43

build: downgrade npm requirement to 10.x

9b108d5

- Remove packageManager field requiring [email protected] - Update engines.npm from >=11.0.0 to >=10.0.0 to match Node.js 20 LTS bundled npm

Merge branch 'tdx26/main' into phale/more-tweaks-for-refs

b3eeee8

Merge branch 'phale/more-tweaks-for-refs' of https://github.com/force…

8e1e6f1

…dotcom/apex-language-support into phale/more-tweaks-for-refs

test: increase beforeAll timeout for ResourceLoader Compilation tests

aec96bb

- Increase timeout to 5min for CI, 3min for local - Fixes timeout failures on Windows CI where full compilation takes longer - Matches timeout pattern used in ResourceLoaderIntegration tests

style: remove trailing whitespace in ResourceLoaderIntegration test

59fb9e4

kylewalke reviewed Jan 7, 2026

View reviewed changes

package.json

]

},

"lint:fix": {

"dependencies": [

Copy link

kylewalke Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

test: update documentSymbol snapshots to match current implementation

e572416

- Update 12 snapshots for textDocument/documentSymbol requests - Reflects changes in symbol structure (empty children arrays, updated selectionRange values) - All quality tests now passing

peternhale merged commit 24b0b65 into tdx26/main Jan 7, 2026
13 of 14 checks passed

peternhale deleted the phale/more-tweaks-for-refs branch January 7, 2026 18:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[W-20742893] refactor: make cross-file reference resolution on demand #183

[W-20742893] refactor: make cross-file reference resolution on demand #183

Uh oh!

peternhale commented Dec 31, 2025

Uh oh!

Uh oh!

kylewalke Jan 6, 2026

Uh oh!

peternhale Jan 7, 2026

Uh oh!

Uh oh!

kylewalke Jan 6, 2026

Uh oh!

Uh oh!

kylewalke Jan 6, 2026

Uh oh!

peternhale Jan 7, 2026

Uh oh!

kylewalke left a comment

Uh oh!

kylewalke Jan 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[W-20742893] refactor: make cross-file reference resolution on demand #183

[W-20742893] refactor: make cross-file reference resolution on demand #183

Uh oh!

Conversation

peternhale commented Dec 31, 2025

Summary

Key Changes

🚀 Batch Workspace Loading with Compression

⚡ On-Demand Cross-File Reference Resolution

🔧 Queue Management Optimizations

🧪 Testing & Quality

📊 Performance Settings UI

Technical Details

Architecture Changes

Files Changed

Benefits

Testing

Uh oh!

Uh oh!

kylewalke Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

peternhale Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kylewalke Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kylewalke Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

peternhale Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

kylewalke left a comment

Choose a reason for hiding this comment

Uh oh!

kylewalke Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants