Question on Vector PubSub Source throughput capability

[atibdialpad]

Hi @bruceg @jszwedko
I have setup a Vector Deployment in GKE (Agg Mode) which consumes message from a Google PubSub Subscription.
The pipeline looks like this : PubSub Src -> Log Processing Transform Layer -> Loki

When I look at the utilisation metric for the transform stage (first stage after the source) I see it remains very low (~0.02). On the Google Subscription stats I see that the number of unacknowledged message ~ (200K - 300K). I want to understand if there is a bottleneck somewhere in the pubsub->vector_source part and how to identify / fix it.

This is different from #12990 though where the number of unack'ed mesages kept on growing. In this case, message are being processed correctly by Vector. I am trying to explain the constant ~200K-300K unacked messages.

Deployment Details :
100 Vector Pods part of a GKE Deployment
Each pod requests: 3-3.5 vCPU and 6-7GB of memory
Node type: c2-standard-8 Google Instances

Config:

[sources.fst_cloud_log_source]
  type = "gcp_pubsub"
  ack_deadline_secs = 60
  retry_delay_secs = 1 
  project = "fstelephony"
  subscription = "log-ingest-vector"                                                                                                           
  acknowledgements.enabled = true

[bruceg]

Hi, @atibdialpad What version of Vector are you running this on? Is it still the nightly I recommended on #12990 or have you moved to the 0.23 release?

[atibdialpad]

Hi @bruceg
Output from a vector pod :

root@vector-fc4cd9549-zt8qm:/# vector --version
vector 0.24.0 (x86_64-unknown-linux-gnu 7cc3a80 2022-07-14)

[atibdialpad]

Right now I have topology A setup. Since Vector Pod Utilization as well as CPU utilization is low in general, I am thinking of moving to topology B
https://discord.com/channels/742820443487993987/746070591097798688/999189641150337096

What do you think ?

[bruceg]

As described there, topology B will write all events to two sinks instead of just one. As such, it will increase the load on the system. If your current CPU utilization is low, I would not expect that change to cause overloading, but YMMV.