[Feature] StreamPark Support Cloud Serverless Flink #4314
Description
Search before asking
- I had searched in the feature and found no similar feature requirement.
Description
Background
Apache StreamPark is a powerful open-source tool for managing the entire lifecycle of Apache Flink applications. Currently, StreamPark supports deploying Flink jobs to platforms like YARN and Kubernetes. However, with the growing adoption of cloud-native and serverless technologies, many organizations are leveraging fully managed serverless Flink services provided by cloud vendors such as AWS, Alibaba Cloud, and Volcengine
User Scenarios
Multi-Cloud Strategy: Many enterprises operate in multi-cloud environments to avoid vendor lock-in, meet regional compliance requirements, or leverage specific cloud features. StreamPark, as a cloud-agnostic tool, can provide a unified management experience for streaming jobs across different cloud providers.
Cost and Maintenance Reduction: Serverless Flink services eliminate the need for infrastructure management, auto-scale based on workload, and reduce operational overhead. Enabling StreamPark to deploy to these services would allow users to focus on development and business logic rather than cluster maintenance.
Global and Regional Compliance: Different cloud providers have varying region availability and compliance certifications. By supporting multiple cloud serverless Flink services, StreamPark can help users deploy jobs to the most suitable region or cloud provider based on their business needs.
Proposed Solution
Extend StreamPark’s deployment options to include support for cloud serverless Flink services. This would involve:
- Adding new deployment targets for AWS, Alibaba Cloud, Volcengine, and potentially other cloud providers.
- Implementing provider-specific configurations (e.g., region, IAM roles, service endpoints) in the StreamPark job configuration.
- Ensuring compatibility with the REST APIs or SDKs provided by these cloud services for job submission and management.
Benefits
- Unified Management: StreamPark users can manage Flink jobs across on-premises, Kubernetes, and multiple cloud environments from a single interface.
- Reduced Operational Complexity: Leverage serverless Flink offerings to minimize infrastructure management and scaling concerns.
- Flexibility and Choice: Users can choose the best cloud provider or region for their specific use case without being tied to a single vendor.
We believe this feature would significantly enhance StreamPark’s value proposition for organizations adopting cloud-native and multi-cloud strategies.
Usage Scenario
No response
Related issues
No response
Are you willing to submit a PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct