GCP Infrastructure manager terraform for elastic-agent (elastic#3776)

### Summary of your changes
Replaces deprecated GCP Deployment Manager with modern Infrastructure
Manager (Terraform) for deploying Elastic Agent CSPM integration.
Provides identical resources with improved tooling and user experience.

#### New Directory: deploy/infrastructure-manager/gcp-elastic-agent/
Files Added:
main.tf - Main infrastructure configuration (compute instance, network,
service account, IAM bindings)
variables.tf - Input variable definitions
outputs.tf - Deployment outputs
service_account.tf - Standalone service account deployment for agentless
mode
terraform.tfvars.example - Example configuration for main deployment
service_account.tfvars.example - Example configuration for SA-only
deployment
README.md - Comprehensive deployment guide

#### Resources Created
Identical to Deployment Manager implementation:
Compute instance (Ubuntu, n2-standard-4, 32GB disk) with Elastic Agent
pre-installed
Service account with roles/cloudasset.viewer and roles/browser
VPC network with auto-created subnets
IAM bindings (project or organization scope)
Optional SSH firewall rule

#### Compatibility
The new deployment script `infrastructure-manager/deploy.sh` is
compatible with kibana deployment command of the form:
```bash
gcloud config set project elastic-security-test && \
FLEET_URL=https://a6f784d2fb4d48bea7724fbe41ef17d3.fleet.us-central1.gcp.qa.elastic.cloud:443 \
ENROLLMENT_TOKEN=<REDUCTED> \
STACK_VERSION=9.2.3 \
./deploy.sh
```

### Related Issues
- Resolves: elastic#3132

(cherry picked from commit fdf76cc)

Author: amirbenun

deploy/infrastructure-manager/gcp-elastic-agent/README.md

@@ -0,0 +1,156 @@
+## Elastic Agent Infrastructure Manager (Terraform)
+
+Deploy Elastic Agent for CIS GCP integration using GCP Infrastructure Manager. Creates a compute instance with Elastic Agent pre-installed and configured with necessary permissions.
+
+### Prerequisites
+
+1. Elastic Stack with Fleet Server deployed
+2. GCP project with required permissions (see [Required Permissions](#required-permissions))
+3. Fleet URL and enrollment token from Kibana
+
+### Quick Deploy
+
+#### Option 1: Cloud Shell (Recommended)
+
+[![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.svg)](https://shell.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/elastic/cloudbeat.git&cloudshell_git_branch=main&cloudshell_workspace=deploy/infrastructure-manager/gcp-elastic-agent&show=terminal&ephemeral=true)
+
+```bash
+# Set required configuration
+export FLEET_URL="<YOUR_FLEET_URL>"
+export ENROLLMENT_TOKEN="<YOUR_TOKEN>"
+export STACK_VERSION="<YOUR_AGENT_VERSION>"
+
+# Optional: Set these to override defaults
+# export ORG_ID="<YOUR_ORG_ID>"  # For org-level monitoring
+# export DEPLOYMENT_NAME="elastic-agent-deployment"  # Default: elastic-agent-deployment
+# export ZONE="us-central1-a"  # Default: us-central1-a
+# export ELASTIC_ARTIFACT_SERVER="<CUSTOM_SERVER_URL>"  # Default: https://artifacts.elastic.co/downloads/beats/elastic-agent
+
+# Deploy using the deploy script
+./deploy.sh
+```
+
+#### Option 2: GCP Console
+
+1. Go to [Infrastructure Manager Console](https://console.cloud.google.com/infra-manager/deployments/create)
+2. Configure:
+   - **Source**: Git repository
+   - **Repository URL**: `https://github.com/elastic/cloudbeat.git`
+   - **Branch**: `main`
+   - **Directory**: `deploy/infrastructure-manager/gcp-elastic-agent`
+   - **Location**: `us-central1`
+3. Add input variables (see table below)
+4. Click **Create**
+
+### Input Variables
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `project_id` | Yes | - | GCP Project ID |
+| `fleet_url` | Yes | - | Fleet Server URL |
+| `enrollment_token` | Yes | - | Enrollment token (sensitive) |
+| `elastic_agent_version` | Yes | - | Agent version (e.g., `8.15.0`) |
+| `elastic_artifact_server` | No | `https://artifacts.elastic.co/downloads/beats/elastic-agent` | Artifact server URL for downloading Elastic Agent |
+| `zone` | No | `us-central1-a` | GCP zone |
+| `scope` | No | `projects` | `projects` or `organizations` |
+| `parent_id` | Yes | - | Project ID or Organization ID |
+| `startup_validation_enabled` | No | `true` | Enable validation of startup script completion |
+| `startup_timeout_seconds` | No | `600` | Maximum time to wait for startup (seconds) |
+
+### Resources Created
+
+- Compute instance (Ubuntu, n2-standard-4, 32GB disk)
+- Service account with `cloudasset.viewer` and `browser` roles
+- VPC network with auto-created subnets
+- IAM bindings (project or organization level)
+
+### Startup Validation
+
+By default, Terraform waits for the startup script to complete and validates success:
+- **Enabled**: Deployment fails if agent installation fails
+- **Timeout**: 5 minutes (configurable via `startup_timeout_seconds`)
+- **Requires**: `gcloud` CLI installed where Terraform runs
+
+**Disable validation** (for testing or debugging):
+```bash
+# Via environment variable (for deploy.sh)
+export STARTUP_VALIDATION_ENABLED=false
+./deploy.sh
+
+# Or pass to gcloud directly
+gcloud infra-manager deployments apply ${DEPLOYMENT_NAME} \
+  --location=${LOCATION} \
+  --input-values="...,startup_validation_enabled=false"
+```
+
+**Guest Attributes Written**:
+
+The startup script writes these attributes for monitoring:
+- `elastic-agent/startup-status`: `"in-progress"`, `"success"`, or `"failed"`
+- `elastic-agent/startup-error`: Error message (only when failed)
+- `elastic-agent/startup-timestamp`: Completion timestamp (UTC)
+
+Query manually:
+```bash
+gcloud compute instances get-guest-attributes ${INSTANCE_NAME} \
+  --zone ${ZONE} \
+  --query-path=elastic-agent/
+```
+
+### Management
+
+**View deployment:**
+```bash
+gcloud infra-manager deployments describe ${DEPLOYMENT_NAME} --location=${LOCATION}
+```
+
+**Delete deployment:**
+```bash
+gcloud infra-manager deployments delete ${DEPLOYMENT_NAME} --location=${LOCATION}
+```
+
+### Troubleshooting
+
+**Check deployment status:**
+```bash
+# The instance name is based on the deployment name with a random suffix
+# Format: elastic-agent-vm-<random-suffix>
+# Example: elastic-agent-vm-0bc08b82
+
+# Check startup script status via guest attributes
+gcloud compute instances get-guest-attributes elastic-agent-vm-<suffix> \
+  --zone ${ZONE} \
+  --query-path=elastic-agent/startup-status
+
+# Expected values:
+# - "in-progress": Installation is running
+# - "success": Installation completed successfully
+# - "failed": Installation failed (check logs below)
+
+# To find your instance name:
+gcloud compute instances list --filter="name~^elastic-agent-vm-"
+```
+
+**Check agent logs (without SSH):**
+```bash
+# View serial console output (includes startup script execution)
+gcloud compute instances get-serial-port-output ${INSTANCE_NAME} --zone ${ZONE}
+
+# Filter for elastic-agent specific logs
+gcloud compute instances get-serial-port-output ${INSTANCE_NAME} --zone ${ZONE} \
+  | grep elastic-agent-setup
+```
+
+**Check agent logs (with SSH):**
+```bash
+gcloud compute ssh ${INSTANCE_NAME} --zone ${ZONE}
+sudo journalctl -u google-startup-scripts.service
+```
+
+**Common Issues:**
+
+1. **404 error downloading agent**: Check `ELASTIC_ARTIFACT_SERVER` and `STACK_VERSION` are correct
+2. **Guest attributes show "failed"**: Check serial console logs for error details
+3. **Guest attributes not available**: Guest attributes are enabled by default and populate during startup
+
+**Console:** [Infrastructure Manager Deployments](https://console.cloud.google.com/infra-manager/deployments)

deploy/infrastructure-manager/gcp-elastic-agent/deploy.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+set -e
+
+# Configure GCP project
+PROJECT_ID=$(gcloud config get-value core/project)
+SERVICE_ACCOUNT="infra-manager-deployer"
+
+# Ensure prerequisites are configured
+"$(dirname "$0")/setup.sh" "${PROJECT_ID}" "${SERVICE_ACCOUNT}"
+
+# Required environment variables (no defaults - must be provided)
+# FLEET_URL, ENROLLMENT_TOKEN, STACK_VERSION
+
+# Optional environment variables (defaults are in variables.tf)
+# ORG_ID          - Set for org-level monitoring
+# ZONE            - GCP zone (default: us-central1-a)
+# DEPLOYMENT_NAME - Deployment name prefix (default: elastic-agent-deployment)
+# ELASTIC_ARTIFACT_SERVER - Artifact server URL
+
+# Generate unique suffix for resource names (8 hex characters)
+RESOURCE_SUFFIX=$(openssl rand -hex 4)
+
+# Set deployment name with suffix
+DEPLOYMENT_NAME="${DEPLOYMENT_NAME:-elastic-agent-deployment}-${RESOURCE_SUFFIX}"
+
+# Determine zone for location extraction
+# We need the zone to derive the region (location) - use default if not set
+EFFECTIVE_ZONE="${ZONE:-us-central1-a}"
+LOCATION="${EFFECTIVE_ZONE%-?}" # Extract region from zone
+
+# Build input values - only include values that are set
+# Defaults are defined in variables.tf (single source of truth)
+INPUT_VALUES="project_id=${PROJECT_ID}"
+INPUT_VALUES="${INPUT_VALUES},resource_suffix=${RESOURCE_SUFFIX}"
+
+# Required values
+INPUT_VALUES="${INPUT_VALUES},fleet_url=${FLEET_URL}"
+INPUT_VALUES="${INPUT_VALUES},enrollment_token=${ENROLLMENT_TOKEN}"
+INPUT_VALUES="${INPUT_VALUES},elastic_agent_version=${STACK_VERSION}"
+
+# Optional values - only add if explicitly set (let TF use its defaults otherwise)
+if [ -n "${ZONE}" ]; then
+    INPUT_VALUES="${INPUT_VALUES},zone=${ZONE}"
+fi
+
+if [ -n "${ELASTIC_ARTIFACT_SERVER}" ]; then
+    # Remove trailing slash if present
+    ELASTIC_ARTIFACT_SERVER="${ELASTIC_ARTIFACT_SERVER%/}"
+    INPUT_VALUES="${INPUT_VALUES},elastic_artifact_server=${ELASTIC_ARTIFACT_SERVER}"
+fi
+
+# Set scope and parent_id based on ORG_ID
+if [ -n "${ORG_ID}" ]; then
+    INPUT_VALUES="${INPUT_VALUES},scope=organizations"
+    INPUT_VALUES="${INPUT_VALUES},parent_id=${ORG_ID}"
+else
+    INPUT_VALUES="${INPUT_VALUES},scope=projects"
+    INPUT_VALUES="${INPUT_VALUES},parent_id=${PROJECT_ID}"
+fi
+
+# Deploy from local source (repo already cloned by Cloud Shell)
+echo "Starting deployment ${DEPLOYMENT_NAME}..."
+gcloud infra-manager deployments apply "${DEPLOYMENT_NAME}" \
+    --location="${LOCATION}" \
+    --service-account="projects/${PROJECT_ID}/serviceAccounts/${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com" \
+    --local-source="." \
+    --input-values="${INPUT_VALUES}"
+
+EXIT_CODE=$?
+if [ $EXIT_CODE -ne 0 ]; then
+    echo ""
+    echo "Deployment failed with exit code $EXIT_CODE"
+    echo ""
+    echo "Common failure reasons:"
+    echo "  - Wrong artifacts server for pre-release artifact (check ELASTIC_ARTIFACT_SERVER for snapshots/pre-releases)"
+    echo "  - Service account permissions missing for ${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com"
+    echo "  - Invalid input values (fleet_url, enrollment_token, etc.)"
+    echo ""
+    echo "Useful debugging commands:"
+    echo "  # View deployment status"
+    echo "  gcloud infra-manager deployments describe ${DEPLOYMENT_NAME} --location=${LOCATION}"
+    echo ""
+    echo "  # Verify service account permissions"
+    echo "  gcloud projects get-iam-policy ${PROJECT_ID} --flatten='bindings[].members' --filter='bindings.members:serviceAccount:${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com' --format='table(bindings.role)'"
+    echo ""
+    echo "  # View Cloud Build logs"
+    echo "  gsutil cat \$(gcloud infra-manager revisions describe \$(gcloud infra-manager deployments describe ${DEPLOYMENT_NAME} --location=${LOCATION} --format='value(latestRevision)') --location=${LOCATION} --format='value(logs)')/*.txt"
+    echo ""
+    echo "  # View VM startup script logs"
+    echo "  gcloud compute instances get-serial-port-output elastic-agent-vm-${RESOURCE_SUFFIX} --zone=${EFFECTIVE_ZONE} --project=${PROJECT_ID}"
+    echo ""
+    exit $EXIT_CODE
+fi
+
+echo ""
+echo "Deployment successful!"

deploy/infrastructure-manager/gcp-elastic-agent/main.tf

@@ -0,0 +1,74 @@
+terraform {
+  required_version = ">= 1.0"
+  required_providers {
+    google = {
+      source  = "hashicorp/google"
+      version = "~> 5.0"
+    }
+    null = {
+      source  = "hashicorp/null"
+      version = "~> 3.0"
+    }
+  }
+}
+
+provider "google" {
+  project = var.project_id
+}
+
+locals {
+  # Use suffix from deploy.sh to ensure all resource names stay within GCP limits and allow multiple deployments
+  resource_suffix = var.resource_suffix
+  sa_name         = "elastic-agent-sa-${local.resource_suffix}"
+  sa_email        = module.service_account.email
+  network_name    = "elastic-agent-net-${local.resource_suffix}"
+  instance_name   = "elastic-agent-vm-${local.resource_suffix}"
+}
+
+# Resource suffix for all resource names
+variable "resource_suffix" {
+  description = "Unique suffix for resource names (8 hex characters)"
+  type        = string
+}
+
+module "service_account" {
+  source = "./modules/service_account"
+
+  project_id           = var.project_id
+  service_account_name = local.sa_name
+  scope                = var.scope
+  parent_id            = var.parent_id
+}
+
+module "compute_instance" {
+  source = "./modules/compute_instance"
+
+  instance_name           = local.instance_name
+  network_name            = local.network_name
+  machine_type            = var.machine_type
+  zone                    = var.zone
+  sa_email                = local.sa_email
+  elastic_agent_version   = var.elastic_agent_version
+  elastic_artifact_server = var.elastic_artifact_server
+  fleet_url               = var.fleet_url
+  enrollment_token        = var.enrollment_token
+
+  depends_on = [
+    module.service_account
+  ]
+}
+
+module "startup_validation" {
+  source = "./modules/startup_validation"
+
+  enabled       = var.startup_validation_enabled
+  project_id    = var.project_id
+  instance_name = local.instance_name
+  instance_id   = module.compute_instance.id
+  zone          = var.zone
+  timeout       = var.startup_timeout_seconds
+
+  depends_on = [
+    module.compute_instance
+  ]
+}