Create a scalable activity system for collecting, storing, and querying audit logs

[scotwells]

Summary

Create a new aggregated apiserver that lets a user search for audit logs that have been generated from the platform. This apiserver will serve as the foundation that will let us develop activity logs as a human readable versions of audit logs.

This will also let us adjust the backend storage system for the audit logs over time without having to update API client. We'll also use this opportunity to move to Clickhouse for audit log storage as it's a better storage backend for structured audit logs.

Motivation

The portals are retrieving audit logs directly from Loki. Loki was intended to be a short-term option for storing audit logs while we developed a hardened backend for audit log storage. Introducing an API in front of the portals and backend will allow us to adjust the backend over time without impacting client integrations. This also will allow us to expose an API to end-users to retrieve audit logs for their projects or organizations.

Goals

• Introduce an API that lets users search audit logs stored in the system
• Move to Clickhouse for audit log storage
• Design such that we can easily bolt on activity log descriptors in the future

[scotwells]

I got a functional aggregated apiserver running in a test environment that offers an Audit Log querying API backed by Clickhouse. A vector deployment is deployed to source events from the NATS queue and write to ClickHouse. Another vector pipeline is responsible for reading from the apiserver audit logs and write to NATS.

graph LR
    APIServer[Activity API Server<br/>Generates audit logs]
    Vector1[Vector Sidecar<br/>Publishes events]
    NATS[NATS JetStream<br/>Event storage & routing]
    Vector2[Vector Aggregator<br/>Batching & persistence]
    CH[ClickHouse<br/>Long-term storage]
    QueryAPI[Activity API Server<br/>Query interface]
    Client[Clients<br/>kubectl/API]

    APIServer -->|writes| Vector1
    Vector1 -->|publish| NATS
    NATS -->|push| Vector2
    Vector2 -->|insert| CH
    Client -->|query| QueryAPI
    QueryAPI -->|CEL → SQL| CH
    CH -->|results| QueryAPI

    style APIServer fill:#e1f5ff
    style NATS fill:#fff3e0
    style CH fill:#f3e5f5
    style QueryAPI fill:#e8f5e9
    style Vector1 fill:#fff9c4
    style Vector2 fill:#fff9c4

Loading

The activity api enables users to query for audit log data that's been collected. Here's a basic example of filtering based on fields in the audit log and limiting the results.

apiVersion: activity.miloapis.com/v1alpha1
kind: AuditLogQuery
metadata:
  name: recent-pod-events
spec:
  filter: objectRef.resource == 'pods'
  limit: 500

Response for that query:

▶ task test-infra:kubectl -- create -o yaml -f examples/queries/pagination/02-filtered-query.yaml              
apiVersion: activity.miloapis.com/v1alpha1
kind: AuditLogQuery
metadata:
  name: recent-pod-events
spec:
  filter: objectRef.resource == 'pods'
  limit: 500
status:
  continueAfter: "2025-12-05T00:01:18.952267Z"
  message: Returned 500 events. More results available.
  phase: Completed
  results:
  - annotations:
      authorization.k8s.io/decision: allow
      authorization.k8s.io/reason: 'RBAC: allowed by RoleBinding'
      generator: audit-log-generator
      load-test: "true"
    apiVersion: audit.k8s.io/v1
    auditID: audit-1764892879684066000-996
    kind: Event
    level: RequestResponse
    objectRef:
      apiVersion: v1
      name: Pod-1764892879-159
      namespace: default
      resource: pods
    requestReceivedTimestamp: "2025-12-05T00:01:19.684052Z"
    requestURI: /api/v1/namespaces/default/pods/Pod-1764892879-159
    responseStatus:
      code: 403
      metadata: {}
      status: Failure
    sourceIPs:
    - 10.0.20.42
    stage: ResponseComplete
    stageTimestamp: "2025-12-05T00:01:19.684052Z"
    user:
      groups:
      - system:authenticated
      username: user:charlie
    userAgent: kubectl/v1.30.0 (linux/amd64) kubernetes/1234567
    verb: list

Got the Clickhouse Grafana plugin installed as a data source in the test infrastructure environment to make it easy to view the state of Clickhouse during testing.

---

Up next:

• Collect operational telemetry for system components
• Deploy to staging environment and connect to Milo's apiserver to collect audit logs
• Integrate with Milo's multi-tenant hierarchy

[drewr]

lets a user search for audit logs

To be clear, this isn't full-text search capability right? Is it a browser-side filtering?

[scotwells]

@drewr correct, no full-text search capabilities. The apiserver is exposing a server-side filter option that uses CEL for a filter expression that is converted to a Clickhouse SQL WHERE clause to filter in the database.

[scotwells]

I setup telemetry collection for NATS, Vector, and the apiserver. Created a dashboard to visualize health and performance of the audit log pipeline.

This is based on audit logs from the kind cluster environment being collected and stored in Clickhouse along with some performance tests that insert fake audit log data directly into the NATS queue.

---

Going to work on making the system support Milo's multi-tenant hierarchy next.

[scotwells]

Opened a PR on milo's apiserver to automatically include scope annotations on the audit logs that are generated by the apiserver so we don't need to handle this at the ingest layer.

All audit logs will include new annotations that indicate the scope type and scope name.

  "annotations": {
    "platform.miloapis.com/scope.type": "organization",
    "platform.miloapis.com/scope.name": "acme-corp",
  }

The platform.miloapis.com/scope.type annotation will either be set to global, organization, project, or user depending on what scope the request is executing in.

[scotwells]

Created a PR in the new activity repo that includes creating a test environment that includes Clickhouse and RustFS to support hot / cold storage.