Excessive network usage

[juanavelez]

Several of us in our team installed Kafka UI locally on our laptops. We have it configured to connect to all the AWS MSK clusters in our 3 environments (about 6/7 per env and we have 3 envs).

We all leave the docker process running. We just discovered that a few of us went over our ISP data cap by a long margin when none of us used to go near the limit. The culprit is Kafka UI and at least in my case, it has used 1.5TB of data in 7 days. This is confirmed by docker statistics for Network Data Received.

It seems the excessive network usage is a consequence of pulling all the 20+ clusters we have configured constantly. Is there a way to only pull data when working on a specific cluster?

[molshansky-zipwhip]

The other question is: Why is it pulling so much data and what is it pulling exactly? This happens from the moment the app starts and continues despite zero interactions with the running application. We are using stock properties, no modifications to any defaults.

Literally all we need is to open it, see the state of whatever we're browsing (i.e. list of topics and num of messages), and on rare occasion make changes (i.e., alter number of partitions).

The data usage is pretty extreme and seems unnecessary, given how I think most people would be using the app. Thus far it's hard to tell what data it's pulling and why the volume of data is so extreme. I'd expect this to be a largely passive application.

[germanosin]

Kafka UI includes a scheduler that periodically collects statistics and topic data for each cluster to ensure a smooth and responsive user experience. However, if there are more than 20 clusters and the default configuration (kafka.update-metrics-rate-millis) is set to 30 seconds, this process can generate a significant amount of traffic—potentially several gigabytes per day.

If desired, you can increase the value of this parameter to reduce the update frequency, which will in turn lower traffic consumption.

[juanavelez]

Thank you for the reply. Is the kafka.update-metrics-rate-millis per cluster or global?

[molshansky-zipwhip]

Our of curiosity, where in the codebase is it polling for statistics and topic data? I started wading through it but there's a lot to digest. I'm thinking about looking at it a little more closely to see if there are any parameters that we could make configurable so it pulls less data (i.e., if there are statistics it's pulling that are of low/no value, maybe they can be omitted).

One option is that we don't poll for data until you explicitly connect to a cluster. That would mean we add a Connect/Disconnect option for each one in the UI. Could easily be bolded and/or color coded to indicate that a connection is open. This is similar to how tools like DataGrip work, where you could have multiple databases, but you only connect to the ones you need to use. And if we go so far as to only have one active cluster at a time, well, you instantly and massively reduce the amount of data you're polling for.

Another option is that we add the ability to pause the polling if there's no interaction with the UI for a configurable amount of time. In my case, since it's browser based and running in a docker container, I simply close the tab and forget the app is running. Meanwhile it's still active in the background, absolutely decimating my ISP data cap.

[Haarolean]

Thank you for the reply. Is the kafka.update-metrics-rate-millis per cluster or global?

@juanavelez that settings controls how often we update the metrics, it affects the scheduler, so one per instance

[Haarolean]

Our of curiosity, where in the codebase is it polling for statistics and topic data?

@molshansky-zipwhip feel free to take a look at ClustersStatisticsScheduler, and StatisticsService classes