Skip to content

Use the VM loader for loading plugin modules as well resource modules #209

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kriszyp merged 9 commits into from
Mar 12, 2026
Merged

Use the VM loader for loading plugin modules as well resource modules #209

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
66f0dc5
Create an ApplicationScope that holds the top level context for the a…
kriszyp Mar 11, 2026
efd52b7
Fix root handling and unit tests
kriszyp Mar 11, 2026
d15dfcf
Respect verifyPath if it is already set
kriszyp Mar 11, 2026
59ef091
Update components/ApplicationScope.ts
kriszyp Mar 11, 2026
23c0505
Remove unused import
kriszyp Mar 11, 2026
0e4f4dc
Apply suggestions from code review
kriszyp Mar 12, 2026
9f04538
Updates for optional verifyPath and assignment of config
kriszyp Mar 12, 2026
592d314
Merge remote-tracking branch 'origin/main' into vm-load-plugins
kriszyp Mar 12, 2026
6588350
Merge remote-tracking branch 'origin/main' into vm-load-plugins
kriszyp Mar 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions components/ApplicationScope.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
import type { Resources } from '../resources/Resources.ts';
import { type Server } from '../server/Server.ts';
import { loggerWithTag } from '../utility/logging/harper_logger.js';
import { scopedImport } from '../security/jsLoader.ts';
import * as env from '../utility/environment/environmentManager.js';
import { CONFIG_PARAMS } from '../utility/hdbTerms.ts';

export class MissingDefaultFilesOptionError extends Error {
constructor() {
super('No default files option exists. Ensure `files` is specified in config.yaml');
this.name = 'MissingDefaultFilesOptionError';
}
}

/**
* This class is used to represent the application scope for the VM context used for loading modules within an application
*/
export class ApplicationScope {
logger: any;
resources: Resources;
server: Server;
mode?: 'none' | 'vm' | 'compartment'; // option to set this from the scope
dependencyContainment?: boolean; // option to set this from the scope
verifyPath?: string;
config: any;
constructor(name: string, resources: Resources, server: Server) {
this.logger = loggerWithTag(name);

this.resources = resources;
this.server = server;

this.mode = env.get(CONFIG_PARAMS.APPLICATIONS_CONTAINMENT) ?? 'vm';
this.dependencyContainment = Boolean(env.get(CONFIG_PARAMS.APPLICATIONS_DEPENDENCYCONTAINMENT));
}

/**
* The compartment that is used for this scope and any imports that it makes
*/
compartment?: Promise<any>;
/**
* Import a file into the scope's sandbox.
* @param filePath - The path of the file to import.
* @returns A promise that resolves with the imported module or value.
*/
async import(filePath: string): Promise<unknown> {
return scopedImport(filePath, this);
}
}
39 changes: 11 additions & 28 deletions components/Scope.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
import { EventEmitter, once } from 'node:events';
import { type Server } from '../server/Server.ts';
import { server, type Server } from '../server/Server.ts';
import { EntryHandler, type EntryHandlerEventMap, type onEntryEventHandler } from './EntryHandler.ts';
import { OptionsWatcher, OptionsWatcherEventMap } from './OptionsWatcher.ts';
import { loggerWithTag } from '../utility/logging/harper_logger.js';
import type { Resources } from '../resources/Resources.ts';
import { resources, type Resources } from '../resources/Resources.ts';
import type { FileAndURLPathConfig } from './Component.ts';
import { FilesOption } from './deriveGlobOptions.ts';
import { requestRestart } from './requestRestart.ts';
import { scopedImport } from '../security/jsLoader.ts';
import * as env from '../utility/environment/environmentManager.js';
import { CONFIG_PARAMS } from '../utility/hdbTerms';
import { ApplicationScope } from './ApplicationScope.ts';

export class MissingDefaultFilesOptionError extends Error {
constructor() {
Expand All @@ -18,12 +16,6 @@ export class MissingDefaultFilesOptionError extends Error {
}
}

export interface ApplicationContainment {
mode?: 'none' | 'vm' | 'compartment'; // option to set this from the scope
dependencyContainment?: boolean; // option to set this from the scope
verifyPath?: string;
}

/**
* This class is what is passed to the `handleApplication` function of an extension.
*
Expand All @@ -39,22 +31,23 @@ export class Scope extends EventEmitter {
#entryHandlers: EntryHandler[];
#logger: any;
#pendingInitialLoads: Set<Promise<void>>;
applicationScope?: ApplicationScope;

options: OptionsWatcher;
resources: Resources;
server: Server;
resources?: Resources;
server?: Server;
ready: Promise<any[]>;
applicationContainment?: ApplicationContainment;
constructor(name: string, directory: string, configFilePath: string, resources: Resources, server: Server) {
constructor(name: string, directory: string, configFilePath: string, applicationScope: ApplicationScope) {
super();

this.#name = name;
this.#directory = directory;
this.#configFilePath = configFilePath;
this.#logger = loggerWithTag(this.#name);

this.resources = resources;
this.server = server;
this.applicationScope = applicationScope;
this.resources = applicationScope?.resources ?? resources;
this.server = applicationScope?.server ?? server;

this.#entryHandlers = [];
this.#pendingInitialLoads = new Set();
Expand All @@ -66,12 +59,6 @@ export class Scope extends EventEmitter {
.on('error', this.#handleError.bind(this))
.on('change', this.#optionsWatcherChangeListener.bind(this)())
.on('ready', this.#handleOptionsWatcherReady.bind(this));

this.applicationContainment = {
mode: env.get(CONFIG_PARAMS.APPLICATIONS_CONTAINMENT) ?? 'vm',
dependencyContainment: Boolean(env.get(CONFIG_PARAMS.APPLICATIONS_DEPENDENCYCONTAINMENT)),
verifyPath: directory,
};
}

get logger(): any {
Expand Down Expand Up @@ -306,16 +293,12 @@ export class Scope extends EventEmitter {
}
}

/**
* The compartment that is used for this scope and any imports that it makes
*/
compartment?: Promise<any>;
/**
* Import a file into the scope's sandbox.
* @param filePath - The path of the file to import.
* @returns A promise that resolves with the imported module or value.
*/
async import(filePath: string): Promise<unknown> {
return scopedImport(filePath, this);
return this.applicationScope ? this.applicationScope.import(filePath) : import(filePath);
}
}
30 changes: 23 additions & 7 deletions components/componentLoader.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@
import { getConfigObj, resolvePath } from '../config/configUtils.js';
import { createReuseportFd } from '../server/serverHelpers/Request.ts';
import { ErrorResource } from '../resources/ErrorResource.ts';
import { Scope, type ApplicationContainment } from './Scope.ts';
import { Scope } from './Scope.ts';
import { ApplicationScope } from './ApplicationScope.ts';
import { ComponentV1, processResourceExtensionComponent } from './ComponentV1.ts';
import * as httpComponent from '../server/http.ts';
import { Status } from '../server/status/index.ts';
Expand Down Expand Up @@ -64,14 +65,14 @@
}
const hdbAppFolder = process.env.RUN_HDB_APP;
if (hdbAppFolder) {
cfsLoaded.push(loadComponent(hdbAppFolder, resources, hdbAppFolder, { autoReload: Boolean(process.env.DEV_MODE) }));

Check failure on line 68 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Build Harper (Node.js v22) 

Argument of type '{ autoReload: boolean; }' is not assignable to parameter of type 'LoadComponentOptions'.

Check failure on line 68 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Build Harper (Node.js v24) 

Argument of type '{ autoReload: boolean; }' is not assignable to parameter of type 'LoadComponentOptions'.

Check failure on line 68 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Build Harper (Node.js v20) 

Argument of type '{ autoReload: boolean; }' is not assignable to parameter of type 'LoadComponentOptions'.

Check failure on line 68 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Unit Test (Node.js v22) 

Argument of type '{ autoReload: boolean; }' is not assignable to parameter of type 'LoadComponentOptions'.

Check failure on line 68 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Unit Test (Node.js v20) 

Argument of type '{ autoReload: boolean; }' is not assignable to parameter of type 'LoadComponentOptions'.
}
return Promise.all(cfsLoaded).then(() => {
watchesSetup = true;
});
}

export const TRUSTED_RESOURCE_PLUGINS = {

Check failure on line 75 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Build Harper (Node.js v22) 

Exported variable 'TRUSTED_RESOURCE_PLUGINS' has or is using name 'AuthAuditLog' from external module "/home/runner/work/harper/harper/utility/logging/harper_logger" but cannot be named.

Check failure on line 75 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Build Harper (Node.js v24) 

Exported variable 'TRUSTED_RESOURCE_PLUGINS' has or is using name 'AuthAuditLog' from external module "/home/runner/work/harper/harper/utility/logging/harper_logger" but cannot be named.

Check failure on line 75 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Build Harper (Node.js v20) 

Exported variable 'TRUSTED_RESOURCE_PLUGINS' has or is using name 'AuthAuditLog' from external module "/home/runner/work/harper/harper/utility/logging/harper_logger" but cannot be named.

Check failure on line 75 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Unit Test (Node.js v22) 

Exported variable 'TRUSTED_RESOURCE_PLUGINS' has or is using name 'AuthAuditLog' from external module "/home/runner/work/harper/harper/utility/logging/harper_logger" but cannot be named.

Check failure on line 75 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Unit Test (Node.js v20) 

Exported variable 'TRUSTED_RESOURCE_PLUGINS' has or is using name 'AuthAuditLog' from external module "/home/runner/work/harper/harper/utility/logging/harper_logger" but cannot be named.
REST, // for backwards compatibility with older configs
rest: REST,
graphql: graphqlQueryHandler,
Expand Down Expand Up @@ -219,8 +220,9 @@

export interface LoadComponentOptions {
isRoot?: boolean;
applicationScope: ApplicationScope;
autoReload?: boolean;
applicationContainment?: ApplicationContainment;

Check failure on line 225 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Build Harper (Node.js v22) 

Cannot find name 'ApplicationContainment'.

Check failure on line 225 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Build Harper (Node.js v24) 

Cannot find name 'ApplicationContainment'.

Check failure on line 225 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Build Harper (Node.js v20) 

Cannot find name 'ApplicationContainment'.

Check failure on line 225 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Unit Test (Node.js v22) 

Cannot find name 'ApplicationContainment'.

Check failure on line 225 in components/componentLoader.ts

View workflow job for this annotation

GitHub Actions / Unit Test (Node.js v20) 

Cannot find name 'ApplicationContainment'.
providedLoadedComponents?: Map<any, any>;
}

Expand All @@ -241,7 +243,14 @@
const resolvedFolder = realpathSync(componentDirectory);
if (loadedPaths.has(resolvedFolder)) return loadedPaths.get(resolvedFolder);
loadedPaths.set(resolvedFolder, true);
const { providedLoadedComponents, isRoot, autoReload } = options;

const {
providedLoadedComponents,
applicationScope = new ApplicationScope(basename(componentDirectory), resources, server),
isRoot,
autoReload,
} = options;
applicationScope.verifyPath ??= componentDirectory;
if (providedLoadedComponents) loadedComponents = providedLoadedComponents;
try {
let config;
Expand Down Expand Up @@ -286,6 +295,8 @@
// Initialize loading status for all components (applications and extensions)
componentLifecycle.loading(componentStatusName);

let subApplicationScope = isRoot ? new ApplicationScope(componentName, resources, server) : applicationScope;

let extensionModule: any;
const pkg = componentConfig.package;
try {
Expand All @@ -306,8 +317,11 @@
}
}
if (componentPath) {
subApplicationScope.verifyPath = componentPath;
if (!process.env.HARPER_SAFE_MODE) {
extensionModule = await loadComponent(componentPath, resources, origin);
extensionModule = await loadComponent(componentPath, resources, origin, {
applicationScope: subApplicationScope,
});
componentFunctionality[componentName] = true;
}
} else {
Expand All @@ -317,7 +331,7 @@
const plugin = TRUSTED_RESOURCE_PLUGINS[componentName];
extensionModule =
typeof plugin === 'string'
? await scopedImport(plugin.startsWith('@/') ? join(PACKAGE_ROOT, plugin.slice(1)) : plugin)
? await import(plugin.startsWith('@/') ? join(PACKAGE_ROOT, plugin.slice(1)) : plugin)
: plugin;
}

Expand Down Expand Up @@ -358,8 +372,7 @@

// New Plugin API (`handleApplication`)
if (resources.isWorker && extensionModule.handleApplication) {
const scope = new Scope(componentName, componentDirectory, configPath, resources, server);
if (options.applicationContainment) scope.applicationContainment = options.applicationContainment;
const scope = new Scope(componentName, componentDirectory, configPath, applicationScope);

await sequentiallyHandleApplication(scope, extensionModule);

Expand Down Expand Up @@ -464,7 +477,10 @@
});
}
if (config.extensionModule || config.pluginModule) {
const extensionModule = await import(join(componentDirectory, config.extensionModule || config.pluginModule));
const extensionModule = await scopedImport(
join(componentDirectory, config.extensionModule || config.pluginModule),
applicationScope
);
loadedPaths.set(resolvedFolder, extensionModule);
return extensionModule;
}
Expand Down
30 changes: 15 additions & 15 deletions security/jsLoader.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { readFileSync } from 'node:fs';
import { dirname, isAbsolute } from 'node:path';
import { pathToFileURL, fileURLToPath } from 'node:url';
import { SourceTextModule, SyntheticModule, createContext, runInContext } from 'node:vm';
import { Scope } from '../components/Scope.ts';
import { ApplicationScope } from '../components/ApplicationScope.ts';
import logger from '../utility/logging/harper_logger.js';
import { createRequire } from 'node:module';
import * as env from '../utility/environment/environmentManager';
Expand All @@ -25,7 +25,7 @@ let lockedDown = false;
* @param moduleUrl
* @param scope
*/
export async function scopedImport(filePath: string | URL, scope?: Scope) {
export async function scopedImport(filePath: string | URL, scope?: ApplicationScope) {
if (!lockedDown && APPLICATIONS_LOCKDOWN && APPLICATIONS_LOCKDOWN !== 'none') {
lockedDown = true;
if (APPLICATIONS_LOCKDOWN === 'ses') {
Expand Down Expand Up @@ -76,7 +76,7 @@ export async function scopedImport(filePath: string | URL, scope?: Scope) {
}
const moduleUrl = (filePath instanceof URL ? filePath : pathToFileURL(filePath)).toString();
try {
const containmentMode = scope?.applicationContainment.mode;
const containmentMode = scope?.mode;
if (scope && containmentMode !== 'none') {
if (containmentMode === 'compartment') {
// use SES Compartments
Expand Down Expand Up @@ -110,7 +110,7 @@ export async function scopedImport(filePath: string | URL, scope?: Scope) {
/**
* Load a module using Node's vm.Module API with (not really secure) sandboxing
*/
async function loadModuleWithVM(moduleUrl: string, scope: Scope) {
async function loadModuleWithVM(moduleUrl: string, scope: ApplicationScope) {
const moduleCache = new Map<string, Promise<SourceTextModule | SyntheticModule>>();

// Create a secure context with limited globals
Expand Down Expand Up @@ -167,7 +167,7 @@ async function loadModuleWithVM(moduleUrl: string, scope: Scope) {
filename: url,
async importModuleDynamically(specifier: string, script) {
const resolvedUrl = resolveModule(specifier, script.sourceURL);
const useContainment = specifier.startsWith('.') || scope.applicationContainment.dependencyContainment;
const useContainment = specifier.startsWith('.') || scope.dependencyContainment;
const dynamicModule = await loadModuleWithCache(resolvedUrl, useContainment);
return dynamicModule;
},
Expand Down Expand Up @@ -213,7 +213,7 @@ async function loadModuleWithVM(moduleUrl: string, scope: Scope) {
return moduleCache.get(resolvedUrl)!;
}

const useContainment = specifier.startsWith('.') || scope.applicationContainment.dependencyContainment;
const useContainment = specifier.startsWith('.') || scope.dependencyContainment;
// Load the module
return await loadModuleWithCache(resolvedUrl, useContainment);
}
Expand Down Expand Up @@ -246,7 +246,7 @@ async function loadModuleWithVM(moduleUrl: string, scope: Scope) {
{ identifier: url, context }
);
} else if (usePrivateGlobal && url.startsWith('file://')) {
checkAllowedModulePath(url, scope.applicationContainment.verifyPath);
checkAllowedModulePath(url, scope.verifyPath);
Copy link
Contributor

@cb1kenobi cb1kenobi Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if verifyPath is undefined? checkAllowedModulePath() expects containingFolder to be defined, so maybe it's easiest to let checkAllowedModulePath() accept an undefined containingFolder and throw a meaningful message?

// Load source text from file
const source = await readFile(new URL(url), { encoding: 'utf-8' });

Expand Down Expand Up @@ -286,7 +286,7 @@ async function loadModuleWithVM(moduleUrl: string, scope: Scope) {
}
}
} else {
checkAllowedModulePath(url, scope.applicationContainment.verifyPath);
checkAllowedModulePath(url, scope.verifyPath);
// For Node.js built-in modules (node:) and npm packages, use dynamic import
const importedModule = await import(url);
const exportNames = Object.keys(importedModule);
Expand Down Expand Up @@ -318,7 +318,7 @@ async function loadModuleWithVM(moduleUrl: string, scope: Scope) {
return entryModule.namespace;
}

async function getCompartment(scope: Scope, globals) {
async function getCompartment(scope: ApplicationScope, globals) {
const { StaticModuleRecord } = await import('@endo/static-module-record');
require('ses');
const compartment: CompartmentOptions = new (Compartment as typeof CompartmentOptions)(
Expand Down Expand Up @@ -351,7 +351,7 @@ async function getCompartment(scope: Scope, globals) {
const moduleText = await readFile(new URL(moduleSpecifier), { encoding: 'utf-8' });
return new StaticModuleRecord(moduleText, moduleSpecifier);
} else {
checkAllowedModulePath(moduleSpecifier, scope.applicationContainment.verifyPath);
checkAllowedModulePath(moduleSpecifier, scope.verifyPath);
const moduleExports = await import(moduleSpecifier);
return {
imports: [],
Expand Down Expand Up @@ -398,7 +398,7 @@ function getDefaultJSGlobalNames() {
/**
* Get the set of global variables that should be available to modules that run in scoped compartments/contexts.
*/
function getGlobalObject(scope: Scope) {
function getGlobalObject(scope: ApplicationScope) {
const appGlobal = {};
// create the new global object, assigning all the global variables from this global
// except those that will be natural intrinsics of the new VM
Expand All @@ -411,20 +411,20 @@ function getGlobalObject(scope: Scope) {
server: scope.server ?? server,
logger: scope.logger ?? logger,
resources: scope.resources,
config: scope.options.getRoot() ?? {},
config: scope.config ?? {},
fetch: APPLICATIONS_LOCKDOWN === 'ses' ? secureOnlyFetch : fetch,
console,
global: appGlobal,
harper: getHarperExports(scope),
});
return appGlobal;
}
function getHarperExports(scope: Scope) {
function getHarperExports(scope: ApplicationScope) {
return {
server: scope.server ?? server,
logger: scope.logger ?? logger,
resources: scope.resources,
config: scope.options.getRoot() ?? {},
config: scope.config ?? {},
Resource,
tables,
databases,
Expand Down Expand Up @@ -468,7 +468,7 @@ function checkAllowedModulePath(moduleUrl: string, containingFolder: string): bo
if (path.startsWith(containingFolder)) {
return true;
}
throw new Error(`Can not load module outside of application folder`);
throw new Error(`Can not load module outside of application folder ${containingFolder}`);
}
let simpleName = moduleUrl.startsWith('node:') ? moduleUrl.slice(5) : moduleUrl;
simpleName = simpleName.split('/')[0];
Expand Down
15 changes: 13 additions & 2 deletions unitTests/components/Scope.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ const { EntryHandler } = require('#src/components/EntryHandler');
const { restartNeeded, resetRestartNeeded } = require('#src/components/requestRestart');
const { writeFile } = require('node:fs/promises');
const { waitFor } = require('./waitFor.js');
const { ApplicationScope } = require('#js/components/ApplicationScope');

describe('Scope', () => {
beforeEach(() => {
Expand All @@ -38,7 +39,12 @@ describe('Scope', () => {
it('should create a default entry handler', async () => {
writeFileSync(this.configFilePath, stringify({ [this.name]: { files: 'test.js' } }));

const scope = new Scope(this.name, this.directory, this.configFilePath, this.resources, this.server);
const scope = new Scope(
this.name,
this.directory,
this.configFilePath,
new ApplicationScope('test', this.resources, this.server)
);

const readySpy = spy();
scope.on('ready', readySpy);
Expand Down Expand Up @@ -88,7 +94,12 @@ describe('Scope', () => {
it('should create a default entry handler with urlPath', async () => {
writeFileSync(this.configFilePath, stringify({ [this.name]: { files: 'test.js', urlPath: 'abc' } }));

const scope = new Scope(this.name, this.directory, this.configFilePath, this.resources, this.server);
const scope = new Scope(
this.name,
this.directory,
this.configFilePath,
new ApplicationScope('test', this.resources, this.server)
);

const readySpy = spy();
scope.on('ready', readySpy);
Expand Down
Loading
Loading