Native support for GCP Managed Kafka authentication

[amiraminb]

Feature Description
We request native support for authenticating with GCP Managed Kafka clusters directly from the Logstash Kafka input plugin. GCP Managed Kafka uses Google's IAM and OAuth2 for authentication, which requires obtaining a short-lived OAuth2 access token to use as the bearer token for SASL/OAUTHBEARER. Currently, the plugin does not have a straightforward way to handle this token acquisition process.

Use Case and Current Pain Points
We are using Logstash to consume messages from a GCP Managed Kafka service. Our Logstash instance runs within a Google Kubernetes Engine (GKE) pod with a dedicated service account.
Attempting to handle authentication by providing a custom Java class to fetch the token failed due to significant dependency conflicts between the required Google Auth libraries (google-auth-library-oauth2-http, urllib3, etc.) and the versions of libraries bundled with Logstash. This made the direct integration approach unworkable.

Proposed Solution
We propose that the Kafka input plugin be enhanced to natively support the GCP authentication flow. This would ideally allow the plugin to use the ambient credentials (ADC) available in a GCP environment (like GKE or Compute Engine) to automatically obtain and refresh the necessary OAuth2 tokens.
This would dramatically simplify the configuration. A user could simply enable a GCP-specific authentication mode, and the plugin would handle the token exchange internally. This would make the integration much more robust, secure, and easier to maintain for the growing number of users leveraging GCP's managed services.