services updates (#1422)

* Service lifecycle hardening

Author: alexcos20

docs/API.md

@@ -2015,6 +2015,39 @@ Array of `ServiceJob` (with `userData` stripped).
 
 ---
 
+### `HTTP` GET /api/services/serviceList
+
+### `P2P` command: serviceList
+
+#### Description
+
+Node-wide service listing. **Authenticated but NOT owner-scoped** — any authenticated
+consumer identity sees every owner's services. By default it returns only the services
+**currently holding a resource reservation** (exactly what the engines count against the
+shared pools): `Running`/`Restarting`/`Stopping`, the mid-start pipeline states, paid
+`Error` (container died, restartable), and explicitly `Stopped` within the paid window.
+`Expired` and never-paid jobs hold nothing and are not listed by default.
+
+#### Query Parameters
+
+| name              | type    | required | description |
+| ----------------- | ------- | -------- | ----------- |
+| consumerAddress   | string  | v        | caller identity (any consumer) |
+| nonce             | string  | v        | request nonce |
+| signature         | string  | v        | signed message (or use an `Authorization` auth-token header) |
+| status            | number  |          | filter to ONE specific `ServiceStatusNumber` (any status, incl. `75` Expired); takes precedence over `includeAllStatuses` |
+| includeAllStatuses | boolean |         | `true` returns services in every status instead of only the resource-holding set |
+| fromTimestamp     | string  |          | only services created at/after this moment — ISO date (`2026-01-15T00:00:00Z`) or Unix timestamp (seconds or milliseconds) |
+
+#### Response (200)
+
+Array of `ServiceJob`, **listing-sanitized**: `userData`, `dockerCmd`, `dockerEntrypoint`,
+`dockerfile` and `additionalDockerFiles` are stripped (identity, status, resources,
+endpoints and payment metadata are kept). Use the owner-scoped `serviceStatus` to see a
+service's own configuration.
+
+---
+
 ### `HTTP` POST /api/services/serviceExtend
 
 ### `P2P` command: serviceExtend

docs/Ocean Node.postman_collection.json

@@ -28,9 +28,10 @@ and `signature` as query parameters (or an auth-token `Authorization` header).
 | --- | --- | --- | --- |
 | `SERVICE_START` | `/api/services/serviceStart` | POST | Validate, persist a `Starting` record, and return the `serviceId` immediately (escrow + image + container happen in the background) |
 | `SERVICE_GET_STATUS` | `/api/services/serviceStatus` | GET | Read job status / endpoints — authenticated, owner-scoped (see notice below); poll this to follow a starting service |
+| `SERVICE_LIST` | `/api/services/serviceList` | GET | Node-wide service listing — authenticated, **not** owner-scoped. Default: only services currently holding a resource reservation; `status=<n>` filters to one specific status, `includeAllStatuses=true` returns everything, `fromTimestamp` keeps services created at/after that moment. Output is listing-sanitized (no `userData`, no `dockerCmd`/`dockerEntrypoint`, no Dockerfile) |
 | `SERVICE_EXTEND` | `/api/services/serviceExtend` | POST | Pay to push the expiry further out |
-| `SERVICE_RESTART` | `/api/services/serviceRestart` | POST | Recreate the container (no extra charge) |
-| `SERVICE_STOP` | `/api/services/serviceStop` | POST | Tear down the container and release resources |
+| `SERVICE_RESTART` | `/api/services/serviceRestart` | POST | Recreate the container (no extra charge); asynchronous like start — returns once the job is `Restarting`, poll `serviceStatus` |
+| `SERVICE_STOP` | `/api/services/serviceStop` | POST | Tear down the container; the paid resource reservation (cpu/ram/gpu + host ports) is kept until `expiresAt`, so the service can be restarted anytime on the same endpoints |
 | `SERVICE_GET_TEMPLATES` | `/api/services/serviceTemplates` | GET | List operator-published service templates |
 | `SERVICE_GET_STREAMABLE_LOGS` | `/api/services/serviceStreamableLogs` | GET | Stream the container's live stdout/stderr logs — authenticated, owner-scoped; available while `Running` or `Error`; optional `since` to skip history |
 
@@ -42,8 +43,11 @@ terminal `*Failed` / `Error`).
 
 **Handler (synchronous, before responding):** signature check → environment + access-list +
 `features.services` check → `userData` decrypt (validity check) → duration cap → resource
-resolution & availability → cost computed from **server-side** environment pricing → persist the
-job as `Starting` (which also reserves its resources) → respond `200` with the `serviceId`.
+resolution & availability → cost computed from **server-side** environment pricing → escrow
+funds pre-check (fail fast with `400 Insufficient escrow funds` when the consumer's available
+escrow visibly can't cover the cost; best-effort — an RPC hiccup skips it and the background
+Locking step remains the authoritative check) → persist the job as `Starting` (which also
+reserves its resources) → respond `200` with the `serviceId`.
 
 **Background pipeline (per the start statuses below):**
 `Starting (10)` → **locking** `Locking (20)`: escrow `createLock` (+ wait for it to mine) →
@@ -57,16 +61,38 @@ container creation fails before the claim, the lock is **cancelled (refunded)**
 in a `*Failed` / `Error` status. This is a change from the previous synchronous flow, which
 locked-then-claimed up front.
 
+**Restart is asynchronous too.** `serviceRestart` performs only the fast validations
+(ownership, environment/access, not expired, payment not refunded), persists the job as
+`Restarting (45)` and responds immediately — the teardown, image re-pull/rebuild and new
+container happen in the background under the same per-service lock. Poll `serviceStatus`
+and watch `Restarting` → `PullImage`/`BuildImage` → `Running` (or `Error` with the failure
+reason in `statusText`). A service whose start payment was **never claimed** — the escrow
+lock failed outright (e.g. insufficient funds) or was refunded before being claimed —
+cannot be restarted: it was never paid for, so restarting it would run the service for
+free. Start a new service instead.
+
+**The reservation lasts the whole paid window — only `Expired` releases it.** The consumer
+paid for the resources for a time interval and may use them as they please within it:
+running the service, stopping it, restarting it. An explicit `SERVICE_STOP` therefore tears
+down the container/network but **keeps** the resource amounts (cpu/ram/gpu) counted and the
+host ports reserved — another consumer cannot take them, and a restart resumes on the same
+endpoints. The reservation is tied to **payment**: an `Error`/`Stopped` job whose payment
+was never claimed (lock failed or refunded) does not reserve anything — otherwise anyone
+could squat a node's GPU for free by starting services against an empty escrow account.
+Once `expiresAt` passes, the expiry sweep tears down whatever is left, marks the
+job `Expired`, and only then releases everything. The sweep refuses to mark `Expired` while
+teardown fails (e.g. Docker unreachable) — the job stays `Error` and is retried every tick,
+so a resource release is never silently skipped.
+
 **`Running` is monitored too.** The same background loop that advances a starting service also
 checks every `Running` service's container on each tick (~every few seconds). If the container
 exits on its own — crash, OOM, or the Docker daemon itself becoming unreachable — the job is
 moved to `Error` immediately instead of waiting for `expiresAt`. This health check does **not**
 release the service's reserved host ports/network/container record, since the consumer already
 paid for them; use `SERVICE_RESTART` to bring the service back on the same endpoints. `Error`
-counts as an active/resource-reserving status just like `Running` does — it still occupies its
-cpu/ram/gpu allocation and keeps its host ports held — until it is restarted, explicitly
-stopped, or swept by the same expiry check once `expiresAt` passes (which then fully releases
-everything, same as a normal expiry).
+counts as an active/resource-reserving status just like `Running` and `Stopped` do — it still
+occupies its cpu/ram/gpu allocation and keeps its host ports held — until it is restarted or
+swept by the expiry check once `expiresAt` passes (which then fully releases everything).
 
 **Restart is self-healing with respect to leftover Docker state.** Each service gets a Docker
 network with the deterministic name `ocean-svc-<serviceId>`. Teardown (restart, stop, expiry
@@ -84,6 +110,24 @@ the network always reflects the current code's configuration instead of silently
 whatever options a previous node version created it with, and it matches restart's overall
 tear-down-and-rebuild semantics (the container is never reused either).
 
+**Lifecycle operations are exclusive per service.** At most one lifecycle operation — the
+background start pipeline, `SERVICE_RESTART`, `SERVICE_STOP`, or the expiry sweep — runs per
+service at a time. A restart or stop issued while another operation is in flight (e.g. a
+restart still pulling the image) is rejected with
+`Service <id> has a start/stop/restart operation in progress — retry shortly`; simply retry
+once the in-flight operation settles. Without this exclusivity, the background loop's
+crash-orphan recovery could tear down the `ocean-svc-<serviceId>` network in the middle of a
+restart that had just created it, failing the restart with
+`network ocean-svc-<id> not found`. If a service expires while such an operation is in
+flight, the expiry sweep simply retries on a later tick.
+
+Exclusivity holds **across node processes** too, not just within one: each operation also
+takes a lease row in the SQLite `service_locks` table, so two processes sharing the same
+`databases/` directory and Docker daemon (e.g. an old container still running during a
+redeploy) cannot run conflicting operations on the same service. Leases are heartbeated
+every 30 s while the operation runs; a lease not refreshed for 2 minutes belongs to a
+crashed process and is stolen automatically, so no manual cleanup is ever needed.
+
 ## Configuration
 
 Service-on-demand is configured per Docker connection under `serviceOnDemand`:

docs/services.md

@@ -78,6 +78,7 @@ export enum ServiceStatusNumber {
   Locking = 20, // escrow createLock in progress (funds locked, not yet claimed)
   Claiming = 30, // payment phase: claimLock on success, or cancelLock if the image step failed
   Running = 40,
+  Restarting = 45, // SERVICE_RESTART accepted; teardown + re-pull/build + new container in progress
   Stopping = 50,
   Stopped = 70,
   Expired = 75,
@@ -95,12 +96,27 @@ export const ServiceStatusText: Record<ServiceStatusNumber, string> = {
   [ServiceStatusNumber.Locking]: 'Locking',
   [ServiceStatusNumber.Claiming]: 'Claiming',
   [ServiceStatusNumber.Running]: 'Running',
+  [ServiceStatusNumber.Restarting]: 'Restarting',
   [ServiceStatusNumber.Stopping]: 'Stopping',
   [ServiceStatusNumber.Stopped]: 'Stopped',
   [ServiceStatusNumber.Expired]: 'Expired',
   [ServiceStatusNumber.Error]: 'Error'
 }
 
+// Statuses of a service job that is mid-start/restart and owned by an exclusive
+// lifecycle operation. Single source of truth for getPendingServiceStarts (DB query) and
+// the pipeline's staleness guard — the two MUST agree or a job can be picked up and then
+// ignored (or vice versa). Restarting is included so a job orphaned by a crash
+// mid-restart is recovered at boot exactly like a crash mid-start.
+export const SERVICE_START_PENDING_STATUSES: readonly ServiceStatusNumber[] = [
+  ServiceStatusNumber.Starting,
+  ServiceStatusNumber.Locking,
+  ServiceStatusNumber.PullImage,
+  ServiceStatusNumber.BuildImage,
+  ServiceStatusNumber.Claiming,
+  ServiceStatusNumber.Restarting
+]
+
 export interface ServiceJob {
   serviceId: string // unique id for a running service — distinct from a compute jobId
   clusterHash: string

src/@types/C2D/ServiceOnDemand.ts

@@ -450,6 +450,24 @@ export interface ServiceGetStatusCommand extends Command {
   serviceId?: string
 }
 
+// Node-wide service listing (SERVICE_LIST), shaped like GetJobsCommand. Authenticated
+// like every other service command (consumerAddress + signature/token), but NOT
+// owner-scoped: it returns every consumer's services. Default (no filters): only the
+// services currently holding a resource reservation.
+export interface GetServicesCommand extends Command {
+  consumerAddress: string
+  nonce: string
+  signature: string
+  // filter to ONE specific status (any ServiceStatusNumber, incl. Expired); takes
+  // precedence over includeAllStatuses
+  status?: number
+  // return services in EVERY status instead of only the resource-holding set
+  includeAllStatuses?: boolean
+  // only services created at/after this moment: ISO date string, or a Unix timestamp
+  // (seconds or milliseconds) as a string
+  fromTimestamp?: string
+}
+
 export interface ServiceRestartCommand extends Command {
   consumerAddress: string
   nonce: string

src/@types/commands.ts

@@ -110,11 +110,25 @@ export abstract class C2DEngine {
   // eslint-disable-next-line @typescript-eslint/no-unused-vars, require-await
   public async processServiceStart(job: ServiceJob): Promise<void> {}
 
+  // onlyIfExpired: expiry-sweep mode — re-validate expiresAt on the fresh row under the
+  // lifecycle lock and skip the teardown when the service was extended in the meantime.
   // eslint-disable-next-line @typescript-eslint/no-unused-vars, require-await
-  public async stopService(serviceId: string, owner: string): Promise<ServiceJob | null> {
+  public async stopService(
+    serviceId: string,
+    owner: string,
+    onlyIfExpired?: boolean
+  ): Promise<ServiceJob | null> {
     return null
   }
 
+  // Runs fn serialized with the engine's per-service lifecycle operations (start
+  // pipeline, restart, stop, expiry sweep). Engines without a lock implementation run
+  // fn directly; C2DEngineDocker overrides this with its lifecycle lock + DB lease.
+  // eslint-disable-next-line @typescript-eslint/no-unused-vars
+  public async runExclusive<T>(serviceId: string, fn: () => Promise<T>): Promise<T> {
+    return await fn()
+  }
+
   // eslint-disable-next-line @typescript-eslint/no-unused-vars, require-await
   public async restartService(
     serviceId: string,