portfolio vstorage utils

PLAN.md

@@ -0,0 +1,38 @@
+# Plan: Move portfolio vstorage schema into portfolio-api and add client helpers
+
+## Goals
+- Make the vstorage schema (paths, shapes, and derived state rules) part of `portfolio-api` so external services can depend on it instead of pulling from `portfolio-contract`.
+- Provide client-facing helpers in `portfolio-api` to read vstorage entries and infer coherent state even when data is split across paths or arrives in different orders.
+
+## Current findings
+- Vstorage path builders and pattern guards live in `packages/portfolio-contract/src/type-guards.ts` (`makePortfolioPath/makePositionPath/makeFlowPath`, `PortfolioStatusShapeExt`, `PositionStatusShape`, `FlowStatusShape`, etc.) and are used directly by tests and downstream services (`services/ymax-planner/src/engine.ts` uses `PortfolioStatusShapeExt` and `flowIdFromKey` alongside separate vstorage queries).
+- Flow lifecycle data is split: the contract writes `flowsRunning` on the portfolio node before emitting a `flows.flowN` node, so consumers infer an “init/planning” state by combining `flowsRunning` with the child paths; `engine.ts` currently does this by checking `flowsRunning` versus `flows.*` children.
+- Client/test utilities duplicate path logic: `packages/portfolio-contract/tools/portfolio-actors.ts` builds position paths manually, and `multichain-testing/test/ymax0/ymax-deploy.test.ts` parses portfolio IDs from paths.
+- No `portfolio-api` module currently surfaces the vstorage schema; it only defines TypeScript types and minimal guards (`src/type-guards.ts`).
+
+## Plan
+
+2) **Move existing schema references to the new module**
+   - Refactor `portfolio-contract` to import the schema helpers from `@agoric/portfolio-api` rather than defining them locally; breaking changes to `@aglocal/portfolio-contract/src/type-guards.ts` are acceptable (no compatibility shim needed).
+   - Update downstream consumers (planner `services/ymax-planner`, deploy/tests, and `tools/portfolio-actors.ts`) to import from `portfolio-api` instead of the contract package.
+
+3) **Add derived-state utilities for clients**
+   - Implement helper(s) in `portfolio-api` that read vstorage via `VstorageKit`/`chainStorageWatcher` style interfaces and return a normalized snapshot combining:
+     - Portfolio status (including `flowsRunning`) from the portfolio node.
+     - Flow nodes (`flows.flowN`, `flows.flowN.steps`, `flows.flowN.order`) when present.
+     - Derived flow states that mark flows as `init`/`planning` when present in `flowsRunning` but missing child nodes (covers the “flowsRunning written first” case).
+   - Provide functions to materialize positions using `positionKeys` and `positions.*` nodes, and utilities to join records by pool/chain for UIs and services.
+   - Scope (now): only the latest entry per vstorage node. Future work: optional helpers to walk historical entries for replay / richer test coverage.
+
+4) **Reduce duplicated path handling**
+   - Replace ad hoc path assembly/parsing in test utilities (`portfolio-actors`, `multichain-testing`, planner) with the new helpers to ensure consistent behavior and easier upgrades to path conventions.
+   - Document the expected topology of vstorage under `published.<instance>.portfolios.*` and the event-vs-relational nature of updates (e.g., flow nodes accumulate history, portfolio node is current state).
+
+5) **Tests and documentation**
+   - Add unit tests in `portfolio-api` for the new schema/derived-state helpers using fixture capdata derived from existing contract snapshots.
+   - Update `packages/portfolio-api/README.md` to describe how to read portfolio data, detect in-progress flows, and interpret flow histories; include examples for “init” flows where `flowsRunning` has an entry but `flows.flowN` is not yet present.
+
+## Risks / open questions
+- Avoid circular dependencies: confirm `portfolio-api` can import the necessary pattern utilities without pulling in contract code.
+- Backward compatibility: not required for `@aglocal/*`; we can break `@aglocal/portfolio-contract/src/type-guards.ts` while moving schema into `portfolio-api`.
+- Data retention: current scope is latest-per-node; follow-up work should add history/replay helpers for production vstorage stores and tests.

multichain-testing/scripts/ymax-tool.ts

@@ -12,7 +12,7 @@ import {
   type OfferArgsFor,
   type ProposalType,
   type TargetAllocation,
-} from '@aglocal/portfolio-contract/src/type-guards.ts';
+} from '@agoric/portfolio-api';
 import { makePortfolioSteps } from '@aglocal/portfolio-contract/tools/plan-transfers.ts';
 import { makePortfolioQuery } from '@aglocal/portfolio-contract/tools/portfolio-actors.ts';
 import {

multichain-testing/test/ymax0/ymax-deploy.test.ts

@@ -16,12 +16,9 @@ import {
 } from '@agoric/orchestration';
 import type { TestFn } from 'ava';
 import {
-  makeFlowPath,
-  makePositionPath,
-  portfolioIdOfPath,
   PortfolioStatusShapeExt,
-  type StatusFor,
-} from '@aglocal/portfolio-contract/src/type-guards.ts';
+  readPortfolioLatest,
+} from '@agoric/portfolio-api';
 import { mustMatch } from '@agoric/internal';
 
 const { fromEntries, keys, values } = Object;
@@ -240,8 +237,6 @@ async function* readHistory(
   } while (blockHeight > 0);
 }
 
-const range = (n: number) => [...Array(n).keys()];
-
 // TODO: send an offer before running this test
 test('portfolio-opened', async t => {
   //   t.timeout(1_000);
@@ -262,36 +257,42 @@ test('portfolio-opened', async t => {
 
   const chopPub = (key: string) => key.replace(/^published./, '');
   for (const portfolioKey of portfolioKeys) {
-    const portfolioId = portfolioIdOfPath(portfolioKey);
-    const portfolioInfo = (await vsc.readPublished(
-      chopPub(portfolioKey),
-    )) as StatusFor['portfolio'];
+    const segments = portfolioKey.split('.');
+    const portfolioSegment = segments.pop();
+    portfolioSegment || t.fail('missing portfolio path segment');
+    const portfoliosPathPrefix = segments.join('.');
+    const readLatest = (path: string) => vsc.readPublished(chopPub(path));
+    const snapshot = await readPortfolioLatest({
+      readLatest,
+      listChildren: path => vs.keys(path),
+      portfoliosPathPrefix,
+      portfolioKey: /** @type {`portfolio${number}`} */ (portfolioSegment),
+      includePositions: true,
+    });
+    const portfolioInfo = snapshot.status;
     t.log(portfolioKey, portfolioInfo);
-    const { positionKeys, flowCount } = portfolioInfo;
-    for (const poolKey of positionKeys) {
-      // XXX this makePositionPath API is kinda messy
-      const positionKey = [
-        'ymax0',
-        'portfolios',
-        ...makePositionPath(portfolioId, poolKey),
-      ].join('.');
-      const positionInfo = (await vsc.readPublished(
-        positionKey,
-      )) as StatusFor['position'];
-      t.log(positionKey, positionInfo);
+    mustMatch(portfolioInfo, PortfolioStatusShapeExt, portfolioKey);
+
+    const baseLogPath = chopPub(`${portfoliosPathPrefix}.${snapshot.portfolioKey}`);
+    for (const [poolKey, position] of Object.entries(snapshot.positions || {})) {
+      t.log(
+        `${baseLogPath}.positions.${poolKey}`,
+        position?.status ?? 'missing position node',
+      );
     }
-    for (const flowNum of range(flowCount).map(x => x + 1)) {
-      const flowKey = makeFlowPath(portfolioId, flowNum).join('.');
+    const flowsWithNodes = values(snapshot.flows).filter(flow => flow.status);
+    for (const flowNode of flowsWithNodes) {
+      const flowPath = `${portfoliosPathPrefix}.${snapshot.portfolioKey}.flows.${flowNode.flowKey}`;
       try {
-        for await (const { blockHeight, values } of readHistory(flowKey, {
+        for await (const { blockHeight, values } of readHistory(flowPath, {
           vstorage: vs,
           setTimeout,
           clearTimeout,
         })) {
           try {
             const [jsonCapData] = values as string[];
             const info = vsc.marshaller.fromCapData(JSON.parse(jsonCapData));
-            t.log(flowKey, blockHeight, info);
+            t.log(flowPath, blockHeight, info);
           } catch (err) {
             t.log(blockHeight, values, err);
           }

packages/agoric-cli/src/bin-agops.js

@@ -15,6 +15,7 @@ import { Command, CommanderError, createCommand } from 'commander';
 import { makeOracleCommand } from './commands/oracle.js';
 import { makeGovCommand } from './commands/gov.js';
 import { makePsmCommand } from './commands/psm.js';
+import { makePortfolioCommand } from './commands/portfolio.js';
 import { makeReserveCommand } from './commands/reserve.js';
 import { makeVaultsCommand } from './commands/vaults.js';
 import { makePerfCommand } from './commands/perf.js';
@@ -65,6 +66,7 @@ const procIO = {
 };
 
 program.addCommand(makeOracleCommand(procIO, logger));
+program.addCommand(makePortfolioCommand(procIO));
 program.addCommand(makeReserveCommand(logger, procIO));
 program.addCommand(makeTestCommand(procIO, { fetch }));
 

packages/agoric-cli/src/commands/portfolio.js

@@ -0,0 +1,155 @@
+// @ts-check
+/* eslint-env node */
+import { fetchEnvNetworkConfig, makeVstorageKit } from '@agoric/client-utils';
+import { readPortfolioHistoryEntries } from '@agoric/portfolio-api';
+import { Command } from 'commander';
+
+const networkConfig = await fetchEnvNetworkConfig({ env: process.env, fetch });
+
+const decodeCapDataValue = marshaller => value => {
+  if (
+    value &&
+    typeof value === 'object' &&
+    'body' in value &&
+    'slots' in value
+  ) {
+    return marshaller.fromCapData(value);
+  }
+  if (typeof value === 'string') {
+    return marshaller.fromCapData(JSON.parse(value));
+  }
+  return value;
+};
+
+const formatFlowsRunning = flowsRunning => {
+  const entries = Object.entries(flowsRunning || {});
+  if (!entries.length) return 'none';
+  return entries
+    .map(([key, detail]) =>
+      detail && 'type' in detail ? `${key}:${detail.type}` : key,
+    )
+    .join(', ');
+};
+
+const describePortfolioStatus = status =>
+  `policy=${status.policyVersion} rebalance=${status.rebalanceCount} positionKeys=${status.positionKeys.length} flowsRunning=${formatFlowsRunning(status.flowsRunning)}`;
+
+const toBigIntValue = amount => {
+  if (!amount || typeof amount !== 'object') return undefined;
+  const raw = amount.value;
+  if (typeof raw === 'bigint') return raw;
+  if (typeof raw === 'number') return BigInt(raw);
+  if (typeof raw === 'string') {
+    try {
+      return BigInt(raw);
+    } catch {
+      return undefined;
+    }
+  }
+  return undefined;
+};
+
+const describeFlowEvent = (entry, costBasis) => {
+  const { value } = entry;
+  const detailType = value.type || entry.detail?.type;
+  const amountValue =
+    toBigIntValue(value.amount || entry.detail?.amount) ?? 0n;
+  const { state } = value;
+  const step = 'step' in value ? value.step : undefined;
+  const how = 'how' in value ? value.how : undefined;
+  const parts = [
+    `state=${state}`,
+    detailType ? `type=${detailType}` : undefined,
+    step !== undefined ? `step=${step}` : undefined,
+    how ? `how=${how}` : undefined,
+    amountValue ? `amount=${amountValue}` : undefined,
+  ].filter(Boolean);
+  let delta = 0n;
+  if (amountValue) {
+    if (detailType === 'deposit') delta = amountValue;
+    else if (detailType === 'withdraw') delta = -amountValue;
+  }
+  const nextBasis = costBasis + delta;
+  return { summary: parts.join(' '), delta, nextBasis };
+};
+
+const describePositionStatus = (poolKey, value) => {
+  const totalIn = toBigIntValue(value.totalIn) ?? 0n;
+  const totalOut = toBigIntValue(value.totalOut) ?? 0n;
+  const net = totalIn - totalOut;
+  return { summary: `position=${poolKey} totalIn=${totalIn} totalOut=${totalOut} net=${net}`, totals: { totalIn, totalOut } };
+};
+
+export const makePortfolioCommand = (io = {}) => {
+  const { stdout = process.stdout } = io;
+  const portfolio = new Command('portfolio').description(
+    'Portfolio contract utilities',
+  );
+
+  portfolio
+    .command('history')
+    .description('Print portfolio + flow vstorage history')
+    .option('--instance <string>', 'contract instance name (default ymax1)', 'ymax1')
+    .requiredOption('--id <number>', 'portfolio numeric ID', Number)
+    .option('--limit <number>', 'limit entries (oldest first)', Number)
+    .option('--desc', 'show newest first')
+    .action(async ({ instance, id, limit, desc }) => {
+      const vsk = makeVstorageKit({ fetch }, networkConfig);
+      const decodeValue = decodeCapDataValue(vsk.marshaller);
+      const history = await readPortfolioHistoryEntries({
+        readAt: vsk.vstorage.readAt,
+        listChildren: vsk.vstorage.keys,
+        portfoliosPathPrefix: `published.${instance}.portfolios`,
+        portfolioKey: `portfolio${id}`,
+        decodeValue,
+        sort: desc ? 'desc' : 'asc',
+      });
+      const entries = limit ? history.slice(0, Number(limit)) : history;
+      if (!entries.length) {
+        stdout.write('No history available\n');
+        return;
+      }
+      let costBasis = 0n;
+      const positionTotals = new Map();
+      for (const entry of entries) {
+        let header;
+        let summaryLines = [];
+        if (entry.kind === 'portfolio') {
+          header = 'portfolio';
+          summaryLines = [describePortfolioStatus(entry.value)];
+        } else if (entry.kind === 'position') {
+          header = `position ${entry.poolKey}`;
+          const { summary, totals } = describePositionStatus(
+            entry.poolKey,
+            entry.value,
+          );
+          const prev = positionTotals.get(entry.poolKey) || {
+            totalIn: 0n,
+            totalOut: 0n,
+          };
+          const deltaIn = totals.totalIn - prev.totalIn;
+          const deltaOut = totals.totalOut - prev.totalOut;
+          const delta = deltaIn - deltaOut;
+          costBasis += delta;
+          positionTotals.set(entry.poolKey, totals);
+          summaryLines = [
+            summary,
+            `Δin=${deltaIn} Δout=${deltaOut} costBasis=${costBasis}`,
+          ];
+        } else {
+          header = `flow ${entry.flowKey}`;
+          const { summary, nextBasis } = describeFlowEvent(entry, costBasis);
+          costBasis = nextBasis;
+          summaryLines = [summary, `costBasis=${costBasis}`];
+        }
+        stdout.write(
+          `\n[block ${entry.blockHeight}] ${header}\n  ${summaryLines.join('\n  ')}\n`,
+        );
+      }
+      stdout.write(
+        `\nDisplayed ${entries.length} entr${entries.length === 1 ? 'y' : 'ies'} (of ${history.length})\n`,
+      );
+    });
+
+  return portfolio;
+};

packages/client-utils/src/types.ts

@@ -18,7 +18,7 @@ import type {
   UpdateRecord,
 } from '@agoric/smart-wallet/src/smartWallet.js';
 import type { AssetInfo } from '@agoric/vats/src/vat-bank.js';
-import type { StatusFor } from '@aglocal/portfolio-contract/src/type-guards.js';
+import type { StatusFor } from '@agoric/portfolio-api';
 import type {
   Installation,
   Instance,

packages/portfolio-api/README.md

@@ -1,3 +1,59 @@
 # Portfolio management API
 
 API shared between on-chain contract and external clients
+
+## Vstorage schema and readers
+
+Portfolio vstorage paths, shapes, and helpers now live here (see `src/vstorage-schema.ts`):
+
+- Path builders and parsers: `makePortfolioPath`, `makePositionPath`, `makeFlowPath`, `portfolioIdFromKey`, etc.
+- Shape guards: `PortfolioStatusShapeExt`, `PositionStatusShape`, `FlowStatusShape`, `FlowStepsShape`.
+- Pool mapping: `PoolPlaces` / `BeefyPoolPlaces` and allocation shapes.
+- Derived reader: `readPortfolioLatest({ readLatest, listChildren, portfoliosPathPrefix, portfolioKey })` returns the latest portfolio state plus a joined view of `flowsRunning` and any `flows.flowN` nodes. It marks flows as `phase: 'init'` when they appear in `flowsRunning` but no flow node exists yet (covers the “flowsRunning written first” pattern). Pass `includePositions: true` to also fetch `positions.*` nodes alongside `positionKeys`, returning `positions` keyed by pool and `positionsByChain` keyed by chain. Currently this reads only the latest entry per node; history/replay helpers can be added later.
+- Flow selectors: `selectPendingFlows(snapshot)` pulls the subset of `FlowNodeLatest` entries that still lack a `.flows.flowN` status node (phase `init`), making it easy for planners to find work.
+- History iterator: `iterateVstorageHistory({ readAt, path, minHeight, decodeValue })` wraps `vstorage.readAt` into an async generator so tests and services can replay state transitions with consistent culling.
+- History collector: `readPortfolioHistoryEntries({ readAt, listChildren, portfoliosPathPrefix, portfolioKey, decodeValue, sort })` emits ordered `portfolio`/`flow` events with decoded values.
+- Mock readers: `makeMockVstorageReaders(initialEntries)` provides `readLatest`, `listChildren`, and `writeLatest` helpers backed by a Map for deterministic tests that exercise these utilities without hitting a chain follower.
+- Materializer: `materializePortfolioPositions({ status, positionNodes, poolPlaces })` turns raw `positions.*` entries (or mock data) into the same `positions`/`positionsByChain` view for UIs that already have the node data.
+- Place enumerator: `enumeratePortfolioPlaces({ status, poolPlaces })` returns normalized arrays of chain accounts (e.g., `@noble`) and position entries (e.g., `Aave_Base`) so planners can build balance queries without duplicating `accountIdByChain`/`PoolPlaces` joins.
+
+Example (latest-only):
+
+```js
+import { readPortfolioLatest } from '@agoric/portfolio-api';
+
+const snapshot = await readPortfolioLatest({
+  readLatest: vstorage.readLatest,
+  listChildren: vstorage.keys,
+  portfoliosPathPrefix: 'published.ymax0.portfolios',
+  portfolioKey: 'portfolio3',
+  includePositions: true,
+});
+
+console.log(snapshot.flows.flow2.phase); // 'init' until flow node is written
+console.log(snapshot.positions.USDN.place.chainName); // 'noble'
+console.log(snapshot.positionsByChain.Base.positions.length); // grouped by chain
+
+const pendingFlows = selectPendingFlows(snapshot);
+for (const flow of pendingFlows) {
+  console.log('needs activation', flow.flowKey, flow.detail);
+}
+
+for await (const { blockHeight, values } of iterateVstorageHistory({
+  readAt: vstorage.readAt,
+  path: `${portfoliosPathPrefix}.${snapshot.portfolioKey}.flows.flow1`,
+  minHeight: 5n,
+  decodeValue: json => vstorage.marshaller.fromCapData(JSON.parse(json as string)),
+})) {
+  console.log('flow history entry', blockHeight, values.at(0));
+}
+
+const history = await readPortfolioHistoryEntries({
+  readAt: vstorage.readAt,
+  listChildren: vstorage.keys,
+  portfoliosPathPrefix,
+  portfolioKey: 'portfolio3',
+  decodeValue: json => vstorage.marshaller.fromCapData(JSON.parse(json as string)),
+});
+console.log(`history entries: ${history.length}`);
+```

packages/portfolio-api/package.json

@@ -22,11 +22,15 @@
     "lint:types": "yarn run -T tsc"
   },
   "dependencies": {
+    "@agoric/ertp": "workspace:*",
     "@agoric/orchestration": "workspace:*",
-    "@endo/common": "^1.2.13"
+    "@endo/common": "^1.2.13",
+    "@endo/errors": "^1.2.13",
+    "@endo/patterns": "^1.7.0"
   },
   "devDependencies": {
     "@agoric/internal": "workspace:*",
+    "@endo/init": "^1.1.12",
     "ava": "^5.3.0",
     "c8": "^10.1.3",
     "ts-blank-space": "^0.6.2",