Merge branch 'main' of github.com:elastic/terraform-provider-elasticstack into copilot/fix-1290-2

Author: nick-benoit

.buildkite/release.yml

@@ -1,7 +1,7 @@
 steps:
   - label: Release
     agents:
-      image: "golang:1.25.1@sha256:bb979b278ffb8d31c8b07336fd187ef8fafc8766ebeaece524304483ea137e96"
+      image: "golang:1.25.1@sha256:8305f5fa8ea63c7b5bc85bd223ccc62941f852318ebfbd22f53bbd0b358c07e1"
       cpu: "16"
       memory: "24G"
       ephemeralStorage: "20G"

.github/copilot-instructions.md

@@ -1,14 +1,9 @@
 You will be tasked to fix an issue from an open-source repository. This is a Go based repository hosting a Terrform provider for the elastic stack (elasticsearch and kibana) APIs. This repo currently supports both [plugin framework](https://developer.hashicorp.com/terraform/plugin/framework/getting-started/code-walkthrough) and [sdkv2](https://developer.hashicorp.com/terraform/plugin/sdkv2) resources. Unless you're told otherwise, all new resources _must_ use the plugin framework. 
 
-
-
-
 Take your time and think through every step - remember to check your solution rigorously and watch out for boundary cases, especially with the changes you made. Your solution must be perfect. If not, continue working on it. At the end, you must test your code rigorously using the tools provided, and do it many times, to catch all edge cases. If it is not robust, iterate more and make it perfect. Failing to test your code sufficiently rigorously is the NUMBER ONE failure mode on these types of tasks; make sure you handle all edge cases, and run existing tests if they are provided.
 
-
 Please see [README.md](../README.md) and the [CONTRIBUTING.md](../CONTRIBUTING.md) docs before getting started.
 
-
 # Workflow
 
 ## High-Level Problem Solving Strategy
@@ -57,27 +52,27 @@ Carefully read the issue and think hard about a plan to solve it before coding.
 - After each change, verify correctness by running relevant tests.
 - If tests fail, analyze failures and revise your patch.
 - Write additional tests if needed to capture important behaviors or edge cases.
-- Ensure all tests pass before finalizing.
+- NEVER accept acceptance tests that have been skipped due to environment issues; always ensure the environment is correctly set up and all tests run successfully.
 
 ### 6.1 Acceptance Testing Requirements
-When running acceptance tests (`make testacc`), ensure the following:
-
+When running acceptance tests, ensure the following:
 
 - **Environment Variables** - The following environment variables are required for acceptance tests:
   - `ELASTICSEARCH_ENDPOINTS` (default: http://localhost:9200)
   - `ELASTICSEARCH_USERNAME` (default: elastic) 
   - `ELASTICSEARCH_PASSWORD` (default: password)
   - `KIBANA_ENDPOINT` (default: http://localhost:5601)
   - `TF_ACC` (must be set to "1" to enable acceptance tests)
-- **Ensure a valid environment if using `go test`** - Check if the required environment variables are set, if not use the defaults specified above. 
-- **Always finish with `make testacc`** - This will run all tests. Make sure all tests pass before considering a task complete.
-- **Pre-set Environment Variables** - Default environment variables are configured in the Makefile. If these defaults are suitable for your testing environment, `make testacc` will work directly without additional setup
-- **Docker Environment** - For isolated testing with guaranteed environment setup, use `make docker-testacc` which starts Elasticsearch and Kibana containers automatically
+- **Run targeted tests using `go test`** - Ensure the required environment variables are explicitly defined when running targeted tests. Example:
+  ```bash
+  ELASTICSEARCH_ENDPOINTS=http://localhost:9200 ELASTICSEARCH_USERNAME=elastic ELASTICSEARCH_PASSWORD=password KIBANA_ENDPOINT=http://localhost:5601 TF_ACC=1 go test -v -run TestAccResourceName ./path/to/testfile.go
+  ```
 
 ## 7. Final Verification
 - Confirm the root cause is fixed.
 - Review your solution for logic correctness and robustness.
 - Iterate until you are extremely confident the fix is complete and all tests pass.
+- Run the acceptance tests for any changed resources. Ensure acceptance tests pass without any environment-related skips. Use `make testacc` to verify this, explicitly defining the required environment variables.
 - Run `make lint` to ensure any linting errors have not surfaced with your changes. This task may automatically correct any linting errors, and regenerate documentation. Include any changes in your commit. 
 
 ## 8. Final Reflection and Additional Testing

.github/workflows/test.yml

@@ -45,7 +45,7 @@ jobs:
           terraform_wrapper: false
 
       - name: Lint
-        run: make lint
+        run: make check-lint
 
   test:
     name: Matrix Acceptance Test

CHANGELOG.md

@@ -14,11 +14,13 @@
 - Migrate `elasticstack_kibana_action_connector` to the Terraform plugin framework ([#1269](https://github.com/elastic/terraform-provider-elasticstack/pull/1269))
 - Migrate `elasticstack_elasticsearch_security_role_mapping` resource and data source to Terraform Plugin Framework ([#1279](https://github.com/elastic/terraform-provider-elasticstack/pull/1279))
 - Add support for `inactivity_timeout` in `elasticstack_fleet_agent_policy` ([#641](https://github.com/elastic/terraform-provider-elasticstack/issues/641))
+- Add support for `kafka` output types in `elasticstack_fleet_output` ([#1302](https://github.com/elastic/terraform-provider-elasticstack/pull/1302))
 - Add support for `prevent_initial_backfill` to `elasticstack_kibana_slo` ([#1071](https://github.com/elastic/terraform-provider-elasticstack/pull/1071))
 - [Refactor] Regenerate the SLO client using the current OpenAPI spec ([#1303](https://github.com/elastic/terraform-provider-elasticstack/pull/1303))
 - Add support for `data_view_id` in the `elasticstack_kibana_slo` resource ([#1305](https://github.com/elastic/terraform-provider-elasticstack/pull/1305))
 - Add support for `unenrollment_timeout` in `elasticstack_fleet_agent_policy` ([#1169](https://github.com/elastic/terraform-provider-elasticstack/issues/1169))
 - Handle default value for `allow_restricted_indices` in `elasticstack_elasticsearch_security_api_key` ([#1315](https://github.com/elastic/terraform-provider-elasticstack/pull/1315))
+- Fixed `nil` reference in kibana synthetics API client in case of response errors ([#1320](https://github.com/elastic/terraform-provider-elasticstack/pull/1320))
 
 ## [0.11.17] - 2025-07-21
 

Makefile

@@ -52,7 +52,6 @@ build-ci: ## build the terraform provider
 .PHONY: build
 build: lint build-ci ## build the terraform provider
 
-
 .PHONY: testacc
 testacc: ## Run acceptance tests
 	TF_ACC=1 go test -v ./... -count $(ACCTEST_COUNT) -parallel $(ACCTEST_PARALLELISM) $(TESTARGS) -timeout $(ACCTEST_TIMEOUT)
@@ -254,7 +253,10 @@ golangci-lint:
 
 
 .PHONY: lint
-lint: setup golangci-lint check-fmt check-docs ## Run lints to check the spelling and common go patterns
+lint: setup golangci-lint fmt docs-generate ## Run lints to check the spelling and common go patterns
+
+.PHONY: check-lint
+check-lint: setup golangci-lint check-fmt check-docs
 
 .PHONY: fmt
 fmt: ## Format code

docs/resources/fleet_output.md

@@ -1,4 +1,3 @@
-
 ---
 # generated by https://github.com/hashicorp/terraform-plugin-docs
 page_title: "elasticstack_fleet_output Resource - terraform-provider-elasticstack"
@@ -13,6 +12,8 @@ Creates a new Fleet Output.
 
 ## Example Usage
 
+### Basic output
+
 ```terraform
 provider "elasticstack" {
   kibana {}
@@ -32,6 +33,168 @@ resource "elasticstack_fleet_output" "test_output" {
 }
 ```
 
+### Basic Kafka output
+
+```terraform
+terraform {
+  required_providers {
+    elasticstack = {
+      source  = "elastic/elasticstack"
+      version = "~> 0.11"
+    }
+  }
+}
+
+provider "elasticstack" {
+  elasticsearch {}
+  kibana {}
+}
+
+# Basic Kafka Fleet Output
+resource "elasticstack_fleet_output" "kafka_basic" {
+  name                 = "Basic Kafka Output"
+  output_id            = "kafka-basic-output"
+  type                 = "kafka"
+  default_integrations = false
+  default_monitoring   = false
+
+  hosts = [
+    "kafka:9092"
+  ]
+
+  # Basic Kafka configuration
+  kafka = {
+    auth_type     = "user_pass"
+    username      = "kafka_user"
+    password      = "kafka_password"
+    topic         = "elastic-beats"
+    partition     = "hash"
+    compression   = "gzip"
+    required_acks = 1
+
+    headers = [
+      {
+        key   = "environment"
+        value = "production"
+      }
+    ]
+  }
+}
+```
+
+### Advanced Kafka output
+
+```terraform
+terraform {
+  required_providers {
+    elasticstack = {
+      source  = "elastic/elasticstack"
+      version = "~> 0.11"
+    }
+  }
+}
+
+provider "elasticstack" {
+  elasticsearch {}
+  kibana {}
+}
+
+# Advanced Kafka Fleet Output with SSL authentication
+resource "elasticstack_fleet_output" "kafka_advanced" {
+  name                 = "Advanced Kafka Output"
+  output_id            = "kafka-advanced-output"
+  type                 = "kafka"
+  default_integrations = false
+  default_monitoring   = false
+
+  hosts = [
+    "kafka1:9092",
+    "kafka2:9092",
+    "kafka3:9092"
+  ]
+
+  # Advanced Kafka configuration
+  kafka = {
+    auth_type      = "ssl"
+    topic          = "elastic-logs"
+    partition      = "round_robin"
+    compression    = "snappy"
+    required_acks  = -1
+    broker_timeout = 10
+    timeout        = 30
+    version        = "2.6.0"
+    client_id      = "elastic-beats-client"
+
+    # Custom headers for message metadata
+    headers = [
+      {
+        key   = "datacenter"
+        value = "us-west-1"
+      },
+      {
+        key   = "service"
+        value = "beats"
+      },
+      {
+        key   = "environment"
+        value = "production"
+      }
+    ]
+
+    # Hash-based partitioning
+    hash = {
+      hash   = "host.name"
+      random = false
+    }
+
+    # SASL configuration
+    sasl = {
+      mechanism = "SCRAM-SHA-256"
+    }
+  }
+
+  # SSL configuration (reusing common SSL block)
+  ssl = {
+    certificate_authorities = [
+      file("${path.module}/ca.crt")
+    ]
+    certificate = file("${path.module}/client.crt")
+    key         = file("${path.module}/client.key")
+  }
+
+  # Additional YAML configuration for advanced settings
+  config_yaml = yamlencode({
+    "ssl.verification_mode"   = "full"
+    "ssl.supported_protocols" = ["TLSv1.2", "TLSv1.3"]
+    "max.message.bytes"       = 1000000
+  })
+}
+
+# Example showing round-robin partitioning with event grouping
+resource "elasticstack_fleet_output" "kafka_round_robin" {
+  name                 = "Kafka Round Robin Output"
+  output_id            = "kafka-round-robin-output"
+  type                 = "kafka"
+  default_integrations = false
+  default_monitoring   = false
+
+  hosts = ["kafka:9092"]
+
+  kafka = {
+    auth_type   = "none"
+    topic       = "elastic-metrics"
+    partition   = "round_robin"
+    compression = "lz4"
+
+    round_robin = [
+      {
+        group_events = 100
+      }
+    ]
+  }
+}
+```
+
 <!-- schema generated by tfplugindocs -->
 ## Schema
 
@@ -48,14 +211,83 @@ resource "elasticstack_fleet_output" "test_output" {
 - `default_integrations` (Boolean) Make this output the default for agent integrations.
 - `default_monitoring` (Boolean) Make this output the default for agent monitoring.
 - `hosts` (List of String) A list of hosts.
+- `kafka` (Attributes) Kafka-specific configuration. (see [below for nested schema](#nestedatt--kafka))
 - `output_id` (String) Unique identifier of the output.
-- `ssl` (Block List) SSL configuration. (see [below for nested schema](#nestedblock--ssl))
+- `ssl` (Attributes) SSL configuration. (see [below for nested schema](#nestedatt--ssl))
 
 ### Read-Only
 
 - `id` (String) The ID of this resource.
 
-<a id="nestedblock--ssl"></a>
+<a id="nestedatt--kafka"></a>
+### Nested Schema for `kafka`
+
+Optional:
+
+- `auth_type` (String) Authentication type for Kafka output.
+- `broker_timeout` (Number) Kafka broker timeout.
+- `client_id` (String) Kafka client ID.
+- `compression` (String) Compression type for Kafka output.
+- `compression_level` (Number) Compression level for Kafka output.
+- `connection_type` (String) Connection type for Kafka output.
+- `hash` (Attributes) Hash configuration for Kafka partition. (see [below for nested schema](#nestedatt--kafka--hash))
+- `headers` (Attributes List) Headers for Kafka messages. (see [below for nested schema](#nestedatt--kafka--headers))
+- `key` (String) Key field for Kafka messages.
+- `partition` (String) Partition strategy for Kafka output.
+- `password` (String, Sensitive) Password for Kafka authentication.
+- `random` (Attributes) Random configuration for Kafka partition. (see [below for nested schema](#nestedatt--kafka--random))
+- `required_acks` (Number) Number of acknowledgments required for Kafka output.
+- `round_robin` (Attributes) Round robin configuration for Kafka partition. (see [below for nested schema](#nestedatt--kafka--round_robin))
+- `sasl` (Attributes) SASL configuration for Kafka authentication. (see [below for nested schema](#nestedatt--kafka--sasl))
+- `timeout` (Number) Timeout for Kafka output.
+- `topic` (String) Kafka topic.
+- `username` (String) Username for Kafka authentication.
+- `version` (String) Kafka version.
+
+<a id="nestedatt--kafka--hash"></a>
+### Nested Schema for `kafka.hash`
+
+Optional:
+
+- `hash` (String) Hash field.
+- `random` (Boolean) Use random hash.
+
+
+<a id="nestedatt--kafka--headers"></a>
+### Nested Schema for `kafka.headers`
+
+Required:
+
+- `key` (String) Header key.
+- `value` (String) Header value.
+
+
+<a id="nestedatt--kafka--random"></a>
+### Nested Schema for `kafka.random`
+
+Optional:
+
+- `group_events` (Number) Number of events to group.
+
+
+<a id="nestedatt--kafka--round_robin"></a>
+### Nested Schema for `kafka.round_robin`
+
+Optional:
+
+- `group_events` (Number) Number of events to group.
+
+
+<a id="nestedatt--kafka--sasl"></a>
+### Nested Schema for `kafka.sasl`
+
+Optional:
+
+- `mechanism` (String) SASL mechanism.
+
+
+
+<a id="nestedatt--ssl"></a>
 ### Nested Schema for `ssl`
 
 Required: