feat: add calculate-memory command to actor (#980)

danpoletaev · web-flow · commit 78eeee79e61b · 2025-12-16T17:05:27.000+01:00
This PR adds new command to cli to allow developers to test their `defaultMemoryMbytes` expressions. Those expressions are used to dynamically calculate memory of the run based on the `input` and `runOptions`. E.g. of usage ```shell apify actor calculate-memory --input ./input.json --maxTotalChargeUsd=25 --timeoutSecs=360 ``` Full specification in Notion 👇 https://www.notion.so/apify/Dynamic-default-Actor-memory-WIP-293f39950a2280b9aa34fa9cd07f52a4?source=copy_link
diff --git a/docs/reference.md b/docs/reference.md
@@ -194,11 +194,13 @@ These commands help you develop Actors locally. Use them to create new Actor pro
 
 ```sh
 DESCRIPTION
-  Creates an Actor project from a template in a new directory.
+  Creates an Actor project from a template in a new directory. The command 
+  automatically initializes a git repository in the newly created Actor 
+  directory.
 
 USAGE
   $ apify create [actorName] [--omit-optional-deps]
-                 [--skip-dependency-install] [-t <value>]
+                 [--skip-dependency-install] [--skip-git-init] [-t <value>]
 
 ARGUMENTS
   actorName  Name of the Actor and its directory
@@ -208,6 +210,8 @@ FLAGS
                                  dependencies.
       --skip-dependency-install  Skip installing Actor
                                  dependencies.
+      --skip-git-init            Skip initializing a git
+                                 repository in the Actor directory.
   -t, --template=<value>         Template for the
                                  Actor. If not provided, the command will prompt for
                                  it. Visit
@@ -373,18 +377,54 @@ DESCRIPTION
   Manages runtime data operations inside of a running Actor.
 
 SUBCOMMANDS
-  actor set-value       Sets or removes record into the
-                        default key-value store associated with the Actor run.
-  actor push-data       Saves data to Actor's run default
-                        dataset.
-  actor get-value       Gets a value from the default
-                        key-value store associated with the Actor run.
-  actor get-public-url  Get an HTTP URL that allows public
-                        access to a key-value store item.
-  actor get-input       Gets the Actor input value from the
-                        default key-value store associated with the Actor run.
-  actor charge          Charge for a specific event in the
-                        pay-per-event Actor run.
+  actor set-value         Sets or removes record into the
+                          default key-value store associated with the Actor run.
+  actor push-data         Saves data to Actor's run
+                          default dataset.
+  actor get-value         Gets a value from the default
+                          key-value store associated with the Actor run.
+  actor get-public-url    Get an HTTP URL that allows
+                          public access to a key-value store item.
+  actor get-input         Gets the Actor input value from
+                          the default key-value store associated with the Actor
+                          run.
+  actor charge            Charge for a specific event in
+                          the pay-per-event Actor run.
+  actor calculate-memory  Calculates the Actor’s dynamic
+                          memory usage based on a memory expression from
+                          actor.json, input data, and run options.
+```
+
+##### `apify actor calculate-memory`
+
+```sh
+DESCRIPTION
+  Calculates the Actor’s dynamic memory usage based on a memory expression from 
+  actor.json, input data, and run options.
+
+USAGE
+  $ apify actor calculate-memory [--build <value>]
+                                 [--default-memory-mbytes <value>]
+                                 [--input <value>] [--max-items <value>]
+                                 [--max-total-charge-usd <value>]
+                                 [--timeout-secs <value>]
+
+FLAGS
+      --build=<value>                  Actor build
+                                       version or build tag to evaluate the
+                                       expression with.
+      --default-memory-mbytes=<value>
+                                       Memory-calculation expression (in MB). If
+                                       omitted, the value is loaded from the
+                                       actor.json file.
+      --input=<value>                  Path to the
+                                       input JSON file used for the calculation.
+      --max-items=<value>              Maximum
+                                       number of items Actor can output.
+      --max-total-charge-usd=<value>   Maximum
+                                       total charge in USD.
+      --timeout-secs=<value>           Maximum run
+                                       timeout, in seconds.
 ```
 
 ##### `apify actor charge`
@@ -517,7 +557,7 @@ DESCRIPTION
 
 USAGE
   $ apify actors push [actorId] [-b <value>] [--dir <value>]
-                      [--force] [--open] [-v <value>] [-w <value>]
+                      [-f] [--open] [-v <value>] [-w <value>]
 
 ARGUMENTS
   actorId  Name or ID of the Actor to push (e.g. "apify/hello-world" or
@@ -530,9 +570,9 @@ FLAGS
                                  it is taken from the '.actor/actor.json' file
       --dir=<value>              Directory where the
                                  Actor is located
-      --force                    Push an Actor even when
-                                 the local files are older than the Actor on the
-                                 platform.
+  -f, --force                    Push an Actor even
+                                 when the local files are older than the Actor on
+                                 the platform.
       --open                     Whether to open the
                                  browser automatically to the Actor details page.
   -v, --version=<value>          Actor version number
diff --git a/package.json b/package.json
@@ -63,6 +63,7 @@
         "node": ">=20"
     },
     "dependencies": {
+        "@apify/actor-memory-expression": "^0.1.3",
         "@apify/actor-templates": "^0.1.5",
         "@apify/consts": "^2.36.0",
         "@apify/input_schema": "^3.17.0",
diff --git a/scripts/generate-cli-docs.ts b/scripts/generate-cli-docs.ts
@@ -29,6 +29,7 @@ const categories: Record<string, CommandsInCategory[]> = {
 		{ command: Commands.actorsRm },
 
 		{ command: Commands.actor },
+		{ command: Commands.actorCalculateMemory },
 		{ command: Commands.actorCharge },
 		{ command: Commands.actorGetInput },
 		{ command: Commands.actorGetPublicUrl },
diff --git a/src/commands/_register.ts b/src/commands/_register.ts
@@ -1,5 +1,6 @@
 import type { BuiltApifyCommand } from '../lib/command-framework/apify-command.js';
 import { ActorIndexCommand } from './actor/_index.js';
+import { ActorCalculateMemoryCommand } from './actor/calculate-memory.js';
 import { ActorChargeCommand } from './actor/charge.js';
 import { ActorGetInputCommand } from './actor/get-input.js';
 import { ActorGetPublicUrlCommand } from './actor/get-public-url.js';
@@ -73,6 +74,7 @@ export const actorCommands = [
 	ActorGetPublicUrlCommand,
 	ActorGetInputCommand,
 	ActorChargeCommand,
+	ActorCalculateMemoryCommand,
 
 	// top-level
 	HelpCommand,
diff --git a/src/commands/actor/_index.ts b/src/commands/actor/_index.ts
@@ -1,4 +1,5 @@
 import { ApifyCommand } from '../../lib/command-framework/apify-command.js';
+import { ActorCalculateMemoryCommand } from './calculate-memory.js';
 import { ActorChargeCommand } from './charge.js';
 import { ActorGetInputCommand } from './get-input.js';
 import { ActorGetPublicUrlCommand } from './get-public-url.js';
@@ -19,6 +20,7 @@ export class ActorIndexCommand extends ApifyCommand<typeof ActorIndexCommand> {
 		ActorGetPublicUrlCommand,
 		ActorGetInputCommand,
 		ActorChargeCommand,
+		ActorCalculateMemoryCommand,
 	];
 
 	async run() {
diff --git a/src/commands/actor/calculate-memory.ts b/src/commands/actor/calculate-memory.ts
@@ -0,0 +1,112 @@
+import { join, resolve } from 'node:path';
+import process from 'node:process';
+
+import { calculateRunDynamicMemory } from '@apify/actor-memory-expression';
+
+import { ApifyCommand } from '../../lib/command-framework/apify-command.js';
+import { Flags } from '../../lib/command-framework/flags.js';
+import { CommandExitCodes } from '../../lib/consts.js';
+import { useActorConfig } from '../../lib/hooks/useActorConfig.js';
+import { error, info, success } from '../../lib/outputs.js';
+import { getJsonFileContent, getLocalKeyValueStorePath } from '../../lib/utils.js';
+
+const DEFAULT_INPUT_PATH = join(getLocalKeyValueStorePath('default'), 'INPUT.json');
+
+/**
+ * This command can be used to test dynamic memory calculation expressions
+ * defined in actor.json or provided via command-line flag.
+ *
+ * Dynamic memory allows Actors to adjust their memory usage based on input data
+ * and run options, optimizing resource allocation and costs.
+ */
+export class ActorCalculateMemoryCommand extends ApifyCommand<typeof ActorCalculateMemoryCommand> {
+	static override name = 'calculate-memory' as const;
+
+	static override description =
+		`Calculates the Actor’s dynamic memory usage based on a memory expression from actor.json, input data, and run options.`;
+
+	/**
+	 * Additional run options exist (e.g., memoryMbytes, disksMbytes, etc.),
+	 * but we intentionally omit them here. These options are rarely needed and
+	 * exposing them would introduce unnecessary confusion for users.
+	 */
+	static override flags = {
+		input: Flags.string({
+			description: 'Path to the input JSON file used for the calculation.',
+			required: false,
+			default: DEFAULT_INPUT_PATH,
+		}),
+		defaultMemoryMbytes: Flags.string({
+			description:
+				'Memory-calculation expression (in MB). If omitted, the value is loaded from the actor.json file.',
+			required: false,
+		}),
+		build: Flags.string({
+			description: 'Actor build version or build tag to evaluate the expression with.',
+			required: false,
+		}),
+		timeoutSecs: Flags.integer({
+			description: 'Maximum run timeout, in seconds.',
+			required: false,
+		}),
+		maxItems: Flags.integer({
+			description: 'Maximum number of items Actor can output.',
+			required: false,
+		}),
+		maxTotalChargeUsd: Flags.integer({
+			description: 'Maximum total charge in USD.',
+			required: false,
+		}),
+	};
+
+	async run() {
+		const { input, defaultMemoryMbytes, ...runOptions } = this.flags;
+
+		let memoryExpression: string | undefined = defaultMemoryMbytes;
+
+		// If not provided via flag, try to load from actor.json
+		if (!memoryExpression) {
+			memoryExpression = await this.getExpressionFromConfig();
+		}
+
+		if (!memoryExpression) {
+			throw new Error(
+				`No memory-calculation expression found. Provide it via the --defaultMemoryMbytes flag or define defaultMemoryMbytes in actor.json.`,
+			);
+		}
+
+		const inputPath = resolve(process.cwd(), this.flags.input);
+		const inputJson = getJsonFileContent(inputPath) ?? {};
+
+		info({ message: `Evaluating memory expression: ${memoryExpression}` });
+
+		try {
+			const result = await calculateRunDynamicMemory(memoryExpression, {
+				input: inputJson,
+				runOptions,
+			});
+			success({ message: `Calculated memory: ${result} MB`, stdout: true });
+		} catch (err) {
+			error({ message: `Memory calculation failed: ${(err as Error).message}` });
+		}
+	}
+
+	/**
+	 * Helper to load the `defaultMemoryMbytes` expression from actor.json.
+	 */
+	private async getExpressionFromConfig(): Promise<string | undefined> {
+		const cwd = process.cwd();
+		const localConfigResult = await useActorConfig({ cwd });
+
+		if (localConfigResult.isErr()) {
+			const { message, cause } = localConfigResult.unwrapErr();
+
+			error({ message: `${message}${cause ? `\n  ${cause.message}` : ''}` });
+			process.exitCode = CommandExitCodes.InvalidActorJson;
+			return;
+		}
+
+		const { config: localConfig } = localConfigResult.unwrap();
+		return localConfig?.defaultMemoryMbytes?.toString();
+	}
+}
diff --git a/test/local/commands/actor/calculate-memory.test.ts b/test/local/commands/actor/calculate-memory.test.ts
@@ -0,0 +1,93 @@
+import { mkdirSync, writeFileSync } from 'node:fs';
+import { mkdir } from 'node:fs/promises';
+import { dirname } from 'node:path';
+
+import { ActorCalculateMemoryCommand } from '../../../../src/commands/actor/calculate-memory.js';
+import { testRunCommand } from '../../../../src/lib/command-framework/apify-command.js';
+import { EMPTY_LOCAL_CONFIG, LOCAL_CONFIG_PATH } from '../../../../src/lib/consts.js';
+import { getLocalKeyValueStorePath } from '../../../../src/lib/utils.js';
+import { useConsoleSpy } from '../../../__setup__/hooks/useConsoleSpy.js';
+import { useTempPath } from '../../../__setup__/hooks/useTempPath.js';
+import { resetCwdCaches } from '../../../__setup__/reset-cwd-caches.js';
+
+const { beforeAllCalls, afterAllCalls, joinPath } = useTempPath('calculate-memory', {
+	create: true,
+	remove: true,
+	cwd: true,
+	cwdParent: false,
+});
+
+const { lastErrorMessage, lastLogMessage } = useConsoleSpy();
+
+const createActorJson = async (overrides: Record<string, unknown> = {}) => {
+	const actorJson = { ...EMPTY_LOCAL_CONFIG, ...overrides };
+
+	await mkdir(joinPath('.actor'), { recursive: true });
+	writeFileSync(joinPath(LOCAL_CONFIG_PATH), JSON.stringify(actorJson, null, '\t'), { flag: 'w' });
+};
+
+describe('apify actor calculate-memory', () => {
+	const START_URLS_LENGTH_BASED_MEMORY_EXPRESSION = "get(input, 'startUrls.length', 1) * 1024";
+	const DEFAULT_INPUT = { startUrls: [1, 2, 3, 4] };
+
+	const inputPath = joinPath(getLocalKeyValueStorePath('default'), 'INPUT.json');
+
+	beforeAll(async () => {
+		mkdirSync(dirname(inputPath), { recursive: true });
+		writeFileSync(inputPath, JSON.stringify(DEFAULT_INPUT), { flag: 'w' });
+		await beforeAllCalls();
+	});
+
+	afterAll(async () => {
+		await afterAllCalls();
+	});
+
+	beforeEach(() => {
+		resetCwdCaches();
+	});
+
+	it('should fail when default memory is not provided in flags or actor.json', async () => {
+		await createActorJson();
+
+		await testRunCommand(ActorCalculateMemoryCommand, {});
+
+		expect(lastErrorMessage()).toMatch(/No memory-calculation expression found./);
+	});
+
+	it('should calculate memory using defaultMemoryMbytes flag', async () => {
+		await testRunCommand(ActorCalculateMemoryCommand, {
+			flags_input: inputPath,
+			flags_defaultMemoryMbytes: START_URLS_LENGTH_BASED_MEMORY_EXPRESSION,
+		});
+
+		expect(lastLogMessage()).toMatch(/4096 MB/);
+	});
+
+	it('should calculate memory using expression from actor.json', async () => {
+		await createActorJson({ defaultMemoryMbytes: START_URLS_LENGTH_BASED_MEMORY_EXPRESSION });
+
+		await testRunCommand(ActorCalculateMemoryCommand, {
+			flags_input: inputPath,
+		});
+
+		expect(lastLogMessage()).toMatch(/4096 MB/);
+	});
+
+	it('should fallback to default input path if input flag is not provided', async () => {
+		await createActorJson({ defaultMemoryMbytes: START_URLS_LENGTH_BASED_MEMORY_EXPRESSION });
+
+		await testRunCommand(ActorCalculateMemoryCommand, {});
+
+		expect(lastLogMessage()).toMatch(/4096 MB/);
+	});
+
+	it('should report error if memory calculation expression is invalid', async () => {
+		await createActorJson({ defaultMemoryMbytes: 'invalid expression' });
+
+		await testRunCommand(ActorCalculateMemoryCommand, {
+			flags_defaultMemoryMbytes: 'invalid expression',
+		});
+
+		expect(lastErrorMessage()).toMatch(/Memory calculation failed: /);
+	});
+});
diff --git a/yarn.lock b/yarn.lock
