Skip to content

Conversation

@r-vasquez
Copy link
Contributor

We chose to manually inject the string instead of
generating a full comment registry for just 2
fields. If the configuration for cloud expands we
can revisit this mechanism.

Sample

# Shadow Link Configuration Template for Redpanda Cloud
# The name of the shadow link
name: ""
# Configurations for Shadow Link in Redpanda Cloud
cloud_options:
  # The ID of the source Redpanda Cloud cluster (optional)
  source_redpanda_id: ""
  # The ID of the shadow Redpanda Cloud cluster
  shadow_redpanda_id: ""

# Configuration for the internal kafka client
client_options:
  # The bootstrap servers to use
  bootstrap_servers: []
  # If provided, this is the expected ID of the source cluster.  If it does
  # not match then the connection will be rejected.  If provided, this value
  # must match the `ClusterId` field returned in the Kafka Metadata response
  # message
  source_cluster_id: ""
  # TLS settings
  tls_settings:
    # Whether or not TLS is enabled
    enabled: false
    # Certificates and keys are provided in PEM format
    tls_pem_settings:
      # The CA
      ca: ""
      # Key and Cert are optional but if one is provided, then both must be
      # The key
      key: ""
      # The cert
      cert: ""
    # If true, the SNI hostname will not be provided when TLS is used
    do_not_set_sni_hostname: false
  # Authentication settings
  authentication_configuration:
    # SASL/SCRAM configuration
    scram_configuration:
      # SCRAM username
      username: ""
      # Password
      password: ""
      # The SCRAM mechanism to use
      scram_mechanism: SCRAM_SHA_256  # SCRAM-SHA-256
    # SASL/PLAIN configuration
    plain_configuration:
      # PLAIN username
      username: ""
      # Password
      password: ""
  # Max metadata age.
  # If 0 is provided, defaults to 10 seconds
  metadata_max_age_ms: 0
  # Connection timeout.
  # If 0 is provided, defaults to 1 second
  connection_timeout_ms: 0
  # Retry base backoff.
  # If 0 is provided, defaults to 100ms
  retry_backoff_ms: 0
  # Fetch request timeout.
  # If 0 is provided, defaults to 500ms
  fetch_wait_max_ms: 0
  # Fetch min bytes.
  # If 0 is provided, defaults to 5 MiB
  fetch_min_bytes: 0
  # Fetch max bytes.
  # If 0 is provided, defaults to 20 MiB
  fetch_max_bytes: 0
  # Fetch partition max bytes.
  # If 0 is provided, defaults to 5 MiB
  fetch_partition_max_bytes: 0

# Topic metadata sync options
topic_metadata_sync_options:
  # How often to sync metadata
  # If 0 provided, defaults to 30 seconds
  interval: 30s  # duration (e.g., 30s, 1m, 1h)
  # List of filters that indicate which topics should be automatically
  # created as shadow topics on the shadow cluster.  This only controls
  # automatic creation of shadow topics and does not effect the state of the
  # mirror topic once it is created.
  # Literal filters for __consumer_offsets, _redpanda.audit_log and _schemas
  # will be rejected as well as prefix filters to match topics prefixed with
  # _redpanda or __redpanda.
  # Wildcard `*` is permitted only for literal filters and will _not_ match
  # any topics that start with _redpanda or __redpanda.  If users wish to
  # shadow topics that start with _redpanda or __redpanda, they should
  # provide a literal filter for those topics.
  auto_create_shadow_topic_filters:
      # Literal or prefix
      pattern_type: LITERAL  # Must match the filter exactly
      # Include or exclude
      filter_type: INCLUDE  # Include the items that match the filter
      # The resource name, or "*"
      # Note if "*", must be the _only_ character
      # and `pattern_type` must be `PATTERN_TYPE_LITERAL`
      name: ""
  # List of topic properties that should be synced from the source topic.
  # The following properties will always be replicated
  # - Partition count
  # - `max.message.bytes`
  # - `cleanup.policy`
  # - `timestamp.type`
  # The following properties are not allowed to be replicated and adding them
  # to this list will result in an error:
  # - `redpanda.remote.readreplica`
  # - `redpanda.remote.recovery`
  # - `redpanda.remote.allowgaps`
  # - `redpanda.virtual.cluster.id`
  # - `redpanda.leaders.preference`
  # - `redpanda.cloud_topic.enabled`
  # This list is a list of properties in addition to the default properties
  # that will be synced.  See `exclude_default`.
  synced_shadow_topic_properties: []
  # If false, then the following topic properties will be synced by default:
  # - `compression.type`
  # - `retention.bytes`
  # - `retention.ms`
  # - `delete.retention.ms`
  # - Replication Factor
  # - `min.compaction.lag.ms`
  # - `max.compaction.lag.ms`
  # If this is true, then only the properties listed in
  # `synced_shadow_topic_properties` will be synced.
  exclude_default: false
  # Enables data replication from the earliest offset
  # on the source topic/partition.
  start_at_earliest:
  # Enables data replication from the latest offset
  # on the source topic/partition.
  start_at_latest:
  # Enables data replication from the first offset on the
  # source topic/partition where the record's timestamp is
  # at or after the specified timestamp.
  start_at_timestamp: "2024-01-01T00:00:00Z"  # RFC3339 timestamp
  # Allows user to pause the topic sync task.  If paused, then
  # the task will enter the 'paused' state and not sync topics or their
  # properties from the source cluster
  paused: false

# Consumer offset sync options
consumer_offset_sync_options:
  # Sync interval
  # If 0 provided, defaults to 30 seconds
  interval: 30s  # duration (e.g., 30s, 1m, 1h)
  # Allows user to pause the consumer offset sync task.  If paused, then
  # the task will enter the 'paused' state and not sync consumer offsets from
  # the source cluster
  paused: false
  # The filters
  group_filters:
      # Literal or prefix
      pattern_type: LITERAL  # Must match the filter exactly
      # Include or exclude
      filter_type: INCLUDE  # Include the items that match the filter
      # The resource name, or "*"
      # Note if "*", must be the _only_ character
      # and `pattern_type` must be `PATTERN_TYPE_LITERAL`
      name: ""

# Security settings sync options
security_sync_options:
  # Sync interval
  # If 0 provided, defaults to 30 seconds
  interval: 30s  # duration (e.g., 30s, 1m, 1h)
  # Allows user to pause the security settings sync task.  If paused,
  # then the task will enter the 'paused' state and will not sync security
  # settings from the source cluster
  paused: false
  # ACL filters
  acl_filters:
      # The resource filter
      resource_filter:
        # The ACL resource type to match
        resource_type: ANY
        # The pattern to apply to name
        pattern_type: ANY
        # Name, if not given will default to match all items in `resource_type`.
        # Note that asterisk `*` is literal and matches resource ACLs
        # that are named `*`
        name: ""
      # The access filter
      access_filter:
        # The name of the principal, if not set will default to match
        # all principals with the specified `operation` and `permission_type`
        principal: ""
        # The ACL operation to match
        operation: ANY
        # The permission type
        permission_type: ANY
        # The host to match.  If not set, will default to match all hosts
        # with the specified `operation` and `permission_type`. Note that
        # the asterisk `*` is literal and matches hosts that are set to `*`
        host: ""

# Schema Registry sync options
schema_registry_sync_options:
  shadow_schema_registry_topic:

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v25.3.x
  • v25.2.x
  • v25.1.x

Release Notes

Features

  • rpk shadow config generate: now supports --print-template along with --for-cloud

We chose to manually inject the string instead of
generating a full comment registry for just 2
fields. If the configuration for cloud expands we
can revisit this mechanism.
Copilot AI review requested due to automatic review settings December 22, 2025 22:58
@r-vasquez r-vasquez requested review from a team and kbatuigas as code owners December 22, 2025 22:58
@r-vasquez r-vasquez requested review from c-julin and removed request for a team December 22, 2025 22:58
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enables the --print-template flag to work with --for-cloud in the rpk shadow config generate command. The implementation manually injects cloud-specific fields (name and cloud_options) into the template rather than generating them through the comment registry, as these fields are not part of the admin/v2 proto definitions.

Key changes:

  • Added comments to CloudShadowLinkOptions fields in the type definition
  • Modified template generation to conditionally include cloud-specific header and fields
  • Added cloud parameter to template writing functions to filter out cloud-incompatible options (like tls_file_settings)

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
src/go/rpk/pkg/cli/shadow/types.go Added documentation comments to CloudShadowLinkOptions struct fields
src/go/rpk/pkg/cli/shadow/config.go Updated template generation logic to support cloud mode with manual injection of cloud-specific fields and conditional filtering

sb.WriteString("# Shadow Link Configuration Template\n")
}

// Manually add name field (not in ShadowLinkConfigurations proto, but in ShadowLink)
Copy link

Copilot AI Dec 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Corrected spelling of 'Configuations' to 'Configurations' in comment.

Copilot uses AI. Check for mistakes.
@r-vasquez
Copy link
Contributor Author

Cc: @paulohtb6 @micheleRP, this was originally not supported when we released, it will be supported in the next patch release 👍

@vbotbuildovich
Copy link
Collaborator

CI test results

test results on build#78307
test_class test_method test_arguments test_kind job_url test_status passed reason test_history
DatalakeClusterRestoreTest test_restore_partition_spec {"catalog_type": "rest_hadoop", "cloud_storage_type": 1} integration https://buildkite.com/redpanda/redpanda/builds/78307#019b4862-4dcb-4e33-bf40-186ab0204c10 FLAKY 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0042, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=DatalakeClusterRestoreTest&test_method=test_restore_partition_spec
MountUnmountIcebergTest test_simple_remount {"cloud_storage_type": 1} integration https://buildkite.com/redpanda/redpanda/builds/78307#019b485b-393f-4b79-b66a-b3501803e13d FLAKY 8/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.1672, p0=0.5175, reject_threshold=0.0100. adj_baseline=0.4225, p1=0.1337, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=MountUnmountIcebergTest&test_method=test_simple_remount

@r-vasquez r-vasquez merged commit a1f92c7 into redpanda-data:dev Dec 30, 2025
27 checks passed
@vbotbuildovich
Copy link
Collaborator

/backport v25.3.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants