How to install a centralized deployment topology

[mahmoudxyz2]

Hello,

I have tried to check docs, google and AI but no good answer.

I am new to DevOps,

I want to set a centralized deployment topology to combine Vector as agent(daemon) and aggregator but with step by step guide because I am still not expert and my all background is in the software dev.

This is what I want, I will pull image from registry(maybe bash scrip or API call for go app) for a web app and set a deployment, the daemon will get the app logs/matrices/health and send them to aggregator.

Then aggregator send them to two outputs, one S3 and the other WebSocket.

Security is one other issue for me because each web app for each deployment may be for different user.

I want to do something similar to this. Although, Go and GRPC is new me(am java and spring), but I will try.

Here are our requirements:

1. Isolated Deployments: Each user must be able to deploy their application in an
   isolated environment, either within their own cluster or one provided by us. This
   ensures that each deployment operates independently, avoiding any crosscontamination of resources.
2. Real-Time Log Monitoring: Users need the capability to view their logs in real-time.
   This enables them to monitor their application's behavior as it happens, making it
   easier to troubleshoot issues immediately.
3. Persistent Log Storage: Users should have access to historical logs, which means we
   need to store logs in a time-persistent manner, such as using S3 or similar storage
   solutions. This allows users to review past events, audit actions, and analyze trends
   over time.
4. Metrics and Alerts: Users require access to various metrics related to their
   deployments for multiple purposes, such as verifying whether the application is
   running smoothly, monitoring resource utilization, and receiving alerts when issues
   arise (e.g., when an application is down or experiencing performance degradation).
5. Security and Isolation: Each deployment must be fully separated and isolated from
   others, with security being a top priority. This ensures that user data and operations
   are secure, and that no unauthorized access or data leakage occurs between different
   user environments.