Vault integration for Apache Pinot

[Akanksha-kedia]

This commit implements comprehensive Vault integration for secure token management across all Pinot components (Controller, Broker, Server, Minion).

Key Features:

• Vault-based authentication with automatic token refresh
• AuthProviderFactory for centralized auth provider creation
• Comprehensive error handling and logging
• Thread-safe token caching with configurable TTL
• Support for multiple Vault authentication methods (AppRole, Token)

Components Added:

• VaultAuth: Core Vault authentication logic
• VaultConfig: Configuration management for Vault settings
• VaultResponse: Response parsing and validation
• VaultStartupManager: Singleton initialization manager
• VaultTokenAuthProvider: Auth provider implementation
• VaultTokenCache: Thread-safe token caching
• VaultUtil: Utility methods for Vault operations
• AuthProviderFactory: Factory pattern for auth providers

Enhancements:

• Updated AuthProviderUtils with Vault support
• Enhanced StaticTokenAuthProvider with null checks
• Integrated Vault initialization in all component starters
• Added HttpSegmentFetcher auth support

Testing:

• AuthProviderFactoryTest: 390 lines of comprehensive tests
• VaultTokenAuthProviderTest: 214 lines of provider tests
• VaultUtilTest: 69 lines of utility tests

Security:

• Fixed security vulnerabilities
• Removed debug/development tags
• Proper secret handling and token lifecycle management

PR Title
Distribute Service Tokens Across Pinot Components with Static and Vault‑Based Auth Providers

✅ PR Description
This change introduces a clear and explicit service‑token distribution model for Apache Pinot components (Controller, Broker, Server, Minion) and documents it visually and configurationally.

🔐 Service Token Distribution
All Pinot components now support service‑level authentication tokens that are independent of user authentication mechanisms (Basic, ZK, LDAP).

For simplicity and clarity:

Admin credentials are reused as service tokens
In production deployments, service credentials must be separated from admin users
✅ Supported Auth Provider Modes (Final)
The following parameter is introduced and treated as authoritative plain text:

auth.provider.type = static | null | vault

This parameter is not optional, not experimental, and not open for discussion.

🧩 Behavior by Mode
1️⃣ auth.provider.type=static
Service tokens are configured manually in component config files
No external dependency
Example:

Enable the controller to fetch segments using a service token

controller.segment.fetcher.auth.token=Basic YWRtaW46dmVyeXNlY3JldA

Basic + base64encode(admin:verysecret)
✅ Tokens must NOT be surrounded by quotes
✅ Restart affected components for changes to take effect

2️⃣ auth.provider.type=vault
Controller, Broker, Server, and Minion:
Authenticate to Vault at startup
Fetch credentials (username/password)
Generate a Basic Auth service token
Cache the token in memory
No Vault calls at runtime

Example configuration:

Vault Configuration (Values Removed)

START GENAI

pinot.controller.vault.enabled=true
pinot.controller.vault.base-url=<VAULT_BASE_URL>
pinot.controller.vault.path=<VAULT_SECRET_PATH>
pinot.controller.vault.ca-cert=<VAULT_CA_CERT_PATH>
pinot.controller.vault.cert=<VAULT_CLIENT_CERT_PATH>
pinot.controller.vault.cert-key=<VAULT_CLIENT_CERT_KEY>

[Akanksha-kedia]

auth.provider.type=null
No service authentication headers are added
Intended only for local development or unsecured environments

[Akanksha-kedia]

@xiangfu0 @Jackie-Jiang please review and give input for this. and wat more we can do around this

[codecov-commenter]

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
8884	2	8882	13

View the full list of 2 ❄️ flaky test(s)

org.apache.pinot.core.operator.transform.function.GreaterThanTransformFunctionTest::testBinaryOperatorTransformFunction

Flake rate in main: 89.80% (Passed 10 times, Failed 88 times)

Stack Traces | 0.614s run time

expected [false] but found [true]

org.apache.pinot.core.operator.transform.function.GreaterThanTransformFunctionTest::testBinaryOperatorTransformFunctionNoDict

Flake rate in main: 89.80% (Passed 10 times, Failed 88 times)

Stack Traces | 0.024s run time

expected [false] but found [true]

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}