Skip to content

Conversation

@rStelmach
Copy link

Summary

Add a new elasticstack_kibana_stream Terraform resource backed by the main kbapi Kibana client, plus a small internal condition model and Streams API helpers, to validate the first end‑to‑end integration between Terraform and Streams.
This POC focuses on group streams, base‑stream creation, and a read‑only view of ingest, without committing yet to a final public Terraform schema for all Streams features.

What this PR changes

  • Streams API helpers (internal/clients/kibana_oapi/streams.go)

    • Wrap the existing kbapi.ClientWithResponses on top of the Kibana HTTP client.
    • Provide focused helpers for:
      • EnableStreams / DisableStreams (POST /api/streams/_enable|_disable),
      • GetStreamJSON, PutStreamRaw, DeleteStream (with the "Expected undefined, received null" quirk handled),
      • GetStreamIngestJSON / PutStreamIngest,
      • GetStreamGroupJSON / PutStreamGroup.
  • New Terraform resource: elasticstack_kibana_stream (internal/kibana/streams)

    • Registered in provider/plugin_framework.go via streams.NewResource.
    • Schema + models:
      • Top‑level: id, name, space_id, description, type, create_if_missing.
      • group { members, metadata, tags } – managed via /api/streams/{name}/_group.
      • ingest { type } – computed, read‑only from /api/streams/{name}/_ingest in this POC.
    • CRUD behavior:
      • Group streams: optionally create the base stream, upsert _group, round‑trip members/metadata/tags, and delete via DELETE /api/streams/{name} with quirks handled.
      • Ingest streams: read‑only (Terraform reads _ingest but does not write it yet) .
  • Internal condition expression model (internal/kibana/streams/conditions.go)

    • Small internal tree (Condition, FieldComparison, And, Or) plus MarshalCondition helper.
    • Unit tests in conditions_test.go verify JSON output for simple and nested trees.

Testing

Kibana's side :

  • generate logs

Terraform side

  • Unit tests:
    • go test ./internal/kibana/streams/... (condition model + flatten/expand helpers).
  • Manual POC validation:
    • Adjust kibana url in new main.tf
    • Build the provider: make build.
    • cd streams then run:
      • terraform init
      • terraform apply
    • Confirm:
      • Group streams: base stream creation via create_if_missing = true, _group upsert, and round‑trip of members / metadata / tags.
      • Ingest streams: read‑only ingest behavior (ingest.type populated from _ingest, no writes to _ingest).

@rStelmach rStelmach changed the title Feat/streams poc Streams support POC Nov 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants