feat: add V2 feed support by b3ngriffiths · Pull Request #16 · form3tech-oss/terraform-provider-chronicle

b3ngriffiths · 2025-11-17T22:56:48Z

Note

Claude Code Implementation: This PR was developed using Claude Code, which implemented new Chronicle V2 feed resources with complete CRUD operations, comprehensive test coverage including unit tests and acceptance tests for all feed types, and proper schema validation following Terraform provider best practices. All code was iteratively reviewed and refined through Claude Code's autonomous feedback loop, with particular attention to authentication handling, state management, and error scenarios, before final human review as part of this PR.

Summary

This PR implements support for Chronicle's GA-released V2 feed connectors, which use Google Cloud Storage Transfer Service for improved performance and reliability.

Release Notes: https://docs.cloud.google.com/chronicle/docs/secops/release-notes#January_30_2026

New Resources

chronicle_feed_amazon_s3_v2 - S3 bucket ingestion via STS
chronicle_feed_amazon_sqs_v2 - SQS-based push ingestion from S3
chronicle_feed_google_cloud_storage_v2 - GCS bucket ingestion via STS
chronicle_feed_google_cloud_storage_event_driven - Event-driven GCS ingestion
chronicle_feed_azure_blobstore_v2 - Azure Blob Storage ingestion via STS

Key Features

Modern Authentication: Support for federated authentication (AWS IAM roles, Azure Workload Identity, GCP service accounts)
Enhanced Control: Configurable source deletion options and lookback periods (1-180 days)
Resource Tagging: Namespace and labels support for asset correlation and organization
API Compliance: All implementations match Chronicle Feed Management API GA specifications

Quick Examples

AWS S3 V2 with IAM Role:

resource "chronicle_feed_amazon_s3_v2" "example" {
  display_name = "AWS S3 V2 Feed"
  log_type     = "AWS_CLOUDTRAIL"
  
  details {
    s3_uri                = "s3://my-logs/cloudtrail/"
    source_delete_options = "ON_SUCCESS"
    
    authentication {
      aws_iam_role_arn = "arn:aws:iam::123456789012:role/chronicle-role"
    }
  }
}

Important Notes

V2 feeds require appropriate IAM permissions for cross-cloud access
Federated authentication is recommended for production environments
See examples/resources/ for complete configuration examples
All feeds include comprehensive acceptance tests

References

Implements four new V2 feed types as standalone resources: - chronicle_feed_amazon_sqs_v2 - chronicle_feed_amazon_s3_v2 - chronicle_feed_google_cloud_storage_v2 - chronicle_feed_azure_blobstore_v2 All V2 feeds use the Google Cloud Storage Transfer Service for improved ingestion performance. Key changes: - Client layer: Added V2 configuration structs for all feed types - Resource layer: Implemented Terraform resource wrappers with schema - Validation: Added V2-specific source deletion option validation - Provider: Registered all 4 new V2 resources - Examples: Created comprehensive example configurations - Documentation: Added implementation summary and usage guide V2 feeds are completely independent of V1 feeds with no migration path. Users can run both V1 and V2 feeds simultaneously and cutover when ready. Notable V2 differences: - Only 2 source deletion options (NEVER, ON_SUCCESS) - New max_lookback_days field (default: 180 days) - Simplified configuration using Transfer Service - Improved ingestion latency and throughput See V2_FEEDS_IMPLEMENTATION.md for complete details.

Implements acceptance tests for all 4 V2 feed types: - chronicle_feed_amazon_sqs_v2_test.go - chronicle_feed_amazon_s3_v2_test.go - chronicle_feed_google_cloud_storage_v2_test.go - chronicle_feed_azure_blobstore_v2_test.go Test coverage for each feed type includes: - Basic CRUD operations with import verification - Update authentication credentials (AWS feeds) - Enable/disable feed state changes - Update log type - Update max_lookback_days - Dual authentication (SQS V2 with S3) - Shared Key vs SAS Token authentication (Azure V2) Tests follow existing patterns from V1 feeds and include: - Resource existence checks - Attribute verification - Import state verification with appropriate ignore fields - Destroy verification - Helper functions for configuration generation All tests compile successfully and are ready for acceptance testing against a live Chronicle API instance.

…alidation Implements the missing Google Cloud Storage Event-Driven feed type that was announced as GA. This feed type uses Pub/Sub notifications for push-based, event-driven ingestion with reduced latency compared to the polling-based GCS V2 feed. New Feed Type: - chronicle_feed_google_cloud_storage_event_driven - Event-driven GCS feed with Pub/Sub - Requires bucket_uri, pubsub_subscription, sourceDeletionOption, maxLookbackDays - Uses Google Security Operations service account (no explicit auth configuration) - Includes 4 acceptance tests: Basic, UpdateEnabled, UpdateSubscription, UpdateMaxLookbackDays Client Layer: - client/feed_google_cloud_storage_event_driven.go - GCS Event-Driven configuration structs - Updated client/feed.go to add GOOGLE_CLOUD_STORAGE_EVENT_DRIVEN constant and factory support Resource Layer: - chronicle/resource_feed_google_cloud_storage_event_driven.go - Terraform resource implementation - Updated chronicle/provider.go to register the new resource Validation Improvements: - Added validateMaxLookbackDays function (enforces 1-180 day range per API spec) - Applied maxLookbackDays validation to all V2 feeds: - chronicle_feed_amazon_sqs_v2 - chronicle_feed_amazon_s3_v2 - chronicle_feed_google_cloud_storage_v2 - chronicle_feed_azure_blobstore_v2 - chronicle_feed_google_cloud_storage_event_driven Tests: - chronicle/resource_feed_google_cloud_storage_event_driven_test.go - 4 acceptance tests Examples: - examples/resources/chronicle_feed_google_cloud_storage_event_driven/resource.tf - Complete example with Pub/Sub setup instructions All changes verified with successful build.

This commit addresses critical discrepancies between the V2 feed implementations and the official Chronicle Feed Management API documentation for the GA release. ## AMAZON_S3_V2 Fixes **Client Layer (client/feed_amazon_s3_v2.go):** - Fixed authentication structure to nest credentials in `access_key_secret_auth` object - Removed `region` field from authentication (not in API spec) - Added support for `aws_iam_role_arn` as alternative authentication method - Changed JSON field names to match API: `access_key_secret_auth`, `aws_iam_role_arn` **Resource Layer (chronicle/resource_feed_amazon_s3_v2.go):** - Removed `region` from authentication schema - Made `access_key_id` and `secret_access_key` optional and mutually exclusive with `aws_iam_role_arn` - Updated expand/flatten functions to handle both auth methods - Added ConflictsWith constraints to prevent using both auth methods **Examples (examples/resources/chronicle_feed_amazon_s3_v2/resource.tf):** - Removed region field from authentication - Added example using IAM role ARN for federated authentication ## AMAZON_SQS_V2 Fixes **Client Layer (client/feed_amazon_sqs_v2.go):** - **BREAKING:** Changed property key from `amazonSqsV2Settings` to `amazon_sqs_v2_settings` (snake_case) - **BREAKING:** Replaced separate fields (region, accountNumber, queueName) with single `queue` field (ARN format) - Fixed authentication structure: `access_key_secret_auth` instead of `sqsAccessKeySecretAuth` - Removed optional S3 authentication (not in API spec - same credentials used for both) - Added support for `aws_iam_role_arn` as alternative authentication method **Resource Layer (chronicle/resource_feed_amazon_sqs_v2.go):** - **BREAKING:** Replaced `region`, `account_number`, `queue_name` with single `queue` field (ARN format) - Simplified authentication to support either access keys OR IAM role ARN - Removed optional S3 authentication block (same credentials for both SQS and S3) - Updated expand/flatten functions to handle new queue ARN format **Examples (examples/resources/chronicle_feed_amazon_sqs_v2/resource.tf):** - Updated to use queue ARN format: `arn:aws:sqs:region:account_id:queue_name` - Removed separate region, account_number, queue_name fields - Simplified authentication (no separate S3 auth) - Added example using IAM role ARN for federated authentication ## AZURE_BLOBSTORE_V2 Enhancements **Client Layer (client/feed_azure_blobstore_v2.go):** - Fixed JSON field name from `sharedKey` to `accessKey` (matches API spec) - Added support for `azure_v2_workload_identity_federation` with `clientID` and `tenantID` - Created new `AzureV2WorkloadIdentityFederation` struct **Resource Layer (chronicle/resource_feed_azure_blobstore_v2.go):** - Added `workload_identity_federation` block with `client_id` and `tenant_id` fields - Made all three auth methods mutually exclusive (shared_key, sas_token, workload_identity_federation) - Updated expand/flatten functions to handle federated identity - Fixed mapping: Terraform `shared_key` → API `accessKey` **Examples (examples/resources/chronicle_feed_azure_blobstore_v2/resource.tf):** - Added Example 3 showing Workload Identity Federation usage - Updated comments to reflect three authentication options ## API Compliance All changes verified against Chronicle Feed Management API documentation: - S3 V2: Uses `access_key_secret_auth` nesting and optional `aws_iam_role_arn` - SQS V2: Uses snake_case property key, queue ARN format, unified authentication - Azure V2: Uses `accessKey` JSON field and supports workload identity federation ## Breaking Changes **AMAZON_SQS_V2:** - Property key changed to snake_case (API requirement) - Schema changed to use single `queue` ARN field instead of separate fields - Authentication simplified (no separate S3 auth) **AMAZON_S3_V2:** - Removed `region` field from authentication Users will need to update their Terraform configurations to match the new schema. All changes compile successfully and align with GA API specifications.

- Fix S3 V2 tests: Remove non-existent 'region' field from authentication - Fix SQS V2 tests: Rewrite to match new schema (queue ARN, unified auth) - Move V2 shared constants to validation.go for better organization - All tests now correctly match resource schemas

Critical fixes: - Fix CheckDestroy: Remove .test suffix from Type comparisons in all V2 tests * Was comparing 'chronicle_feed_amazon_s3_v2.test' instead of 'chronicle_feed_amazon_s3_v2' * This caused destroy checks to silently skip validation - Add authentication validation: Enforce proper credential usage * AWS feeds: Require paired credentials (access_key_id + secret_access_key) * AWS feeds: Enforce exactly one auth method (IAM role XOR access keys) * Azure V2: Enforce exactly one of three auth methods (ExactlyOneOf) * Prevents empty/invalid authentication blocks - Fix import security: Never populate secrets from API response * Removed secret field population during import (originalConf == nil) * Only non-secret fields (IAM role ARN, federated identity) populated * Secrets remain empty after import until explicitly set by user * Aligns with Terraform security best practices Quality improvements: - Verified getLogType() empty string is safe (falls back to user config) - Ran gofmt on all modified files

Provider bugs (would cause auth failures): - Fix getAPIAuthOpts: backstoryapi_access_token field name (was checking credentials) - Fix getAPIAuthOpts: ingestionapi_access_token field name (was checking credentials) - Fix getAPIAuthOpts: ForwarderAPIEnvVar call (was calling BigQueryAPIEnvVar) - Wire up feed_custom_endpoint to client (was defined but never applied) Client bugs (would cause feed operations to fail): - Fix joinBaseFeedMapAndConcreteFeedMap: Remove double toMapWithJSONTags call * concreteFeedMap is already a map, calling toMapWithJSONTags on it again would cause type errors or malformed feed structures - Add nil guard in ReadFeed for unsupported feed types * Prevents panic when newConcreteFeedConfiguration returns nil * Returns clear error: "unsupported feed source type" Test bugs (CheckDestroy never validated properly): - Fix all feed test CheckDestroy functions (15 files total) * Was comparing rs.Type against "chronicle_feed_type.test" * Should compare against "chronicle_feed_type" only * This caused destroy checks to silently skip, masking cleanup failures - Fixed in V2 feeds: S3, SQS, Azure, GCS, GCS Event-Driven - Fixed in V1 feeds: S3, SQS, Azure, GCS Bucket, Office365, Okta System Log, Okta Users, Proofpoint, Qualys VM, Thinkst Canary All fixes verified with successful build.

- godot: Add period to V2 constants comment - lll: Split long lines in event_driven resource and tests - unparam: Remove unused/constant parameters from test helpers * Remove unused t parameter from S3V2AuthUpdated check * Remove constant logType parameter, hardcode AWS_CLOUDTRAIL

Remove constant parameters from test helper functions: - feedAmazonS3V2Ref: remove name parameter (always 'test') - testAccCheckChronicleFeedAmazonS3V2: remove namespace parameter (always 'test') - feedAmazonSQSV2Ref: remove name parameter (always 'test') - testAccCheckChronicleFeedAmazonSQSV2: remove logType parameter (always 'AWS_CLOUDTRAIL') - feedAzureBlobStoreV2Ref: remove name parameter (always 'test') - testAccCheckChronicleFeedAzureBlobStoreV2WithSharedKey: remove logType and sourceDeleteOptions parameters - feedGoogleCloudStorageV2Ref: remove name parameter (always 'test') - testAccCheckChronicleFeedGoogleCloudStorageV2: remove namespace parameter (always 'test') - feedGoogleCloudStorageEventDrivenRef: remove name parameter (always 'test') - testAccCheckChronicleFeedGoogleCloudStorageEventDriven: remove logType parameter (always 'GCP_CLOUDAUDIT') All constant values are now hardcoded in the function implementations.

Remove variable declarations that are no longer needed after hardcoding constant parameters in test helper functions: - S3 V2: Remove unused namespace variable - SQS V2: Remove unused logType variable, fix function call with too many arguments - Azure Blobstore V2: Remove unused logType and sourceDeleteOptions variables - GCS Event Driven: Remove unused logType variables - GCS V2: Remove unused namespace variable All variables that were passed as constant parameters are now hardcoded in the helper function implementations.

Remove unused logType variable declaration from TestAccChronicleFeedAmazonSQSV2_UpdateMaxLookbackDays function. The logType is hardcoded as 'AWS_CLOUDTRAIL' in the helper function.

claude added 2 commits November 17, 2025 22:04

b3ngriffiths requested a review from a team as a code owner November 17, 2025 22:56

b3ngriffiths and others added 23 commits November 18, 2025 09:04

Delete V2_FEEDS_IMPLEMENTATION.md

b38dfe9

Fix gofmt formatting issues

0da1fa7

Add compiled binary to .gitignore

46ea7eb

Fix V2 feed implementations to match Chronicle GA API specifications

fd133ca

Fix final unused logType variable in SQS V2 test

c9680ca

Remove unused logType variable declaration from TestAccChronicleFeedAmazonSQSV2_UpdateMaxLookbackDays function. The logType is hardcoded as 'AWS_CLOUDTRAIL' in the helper function.

fix: resolve golangci-lint issus in v2 feeds

67f4783

fix: resolve golangci-lint issus in v2 feeds

e2ca4d5

fix: resolve golangci-lint issus in v2 feeds

edeea88

fix: resolve golangci-lint issus in v2 feeds

c99277b

chore: update README

72551d7

fix: resolve golangci-lint issus in v2 feeds

c4858f4

fix: resolve golangci-lint issus in v2 feeds

cef97fd

fix: resolve golangci-lint issus in v2 feeds

c92c91f

fix: resolve golangci-lint issus in v2 feeds

aa31a8b

fix: resolve golangci-lint issus in v2 feeds

357bf57

b3ngriffiths closed this Feb 2, 2026

b3ngriffiths deleted the claude/add-v2-feed-support-012N9tpL6hNjQmTE8KPCS9XC branch February 2, 2026 18:32

Navigation Menu

Uh oh!

feat: add V2 feed support - #16

feat: add V2 feed support#16
b3ngriffiths wants to merge 25 commits into
form3tech-oss:masterfrom
b3ngriffiths:claude/add-v2-feed-support-012N9tpL6hNjQmTE8KPCS9XC

b3ngriffiths commented Nov 17, 2025 •

edited by ben-f3

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

b3ngriffiths commented Nov 17, 2025 • edited by ben-f3 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

New Resources

Key Features

Quick Examples

Important Notes

References

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

b3ngriffiths commented Nov 17, 2025 •

edited by ben-f3

Loading