Options for graph scaling for container average CPU and MEM usage

[ambarbour]

Describe the feature you would like to see

I have difficulty visualizing the container CPU and MEM usage with graphs in the latest UI. It can be hard to distinguish between even a 10x difference, particularly with many containers.

We have a server with 128GB memory that often runs about 25 containers, all with typical memory usage around 1-2GB. I used to be able to quickly see if any container was using excessive memory, but now it is hard to see the difference between 2GB and 20GB without close inspection.

Describe how you would like to see this feature implemented

I am guessing that the container CPU and MEM graphs are currently showing percentages of the total available on the host server. It would be nice to have options on how to scale those graphs, such as:

• as is, a percentage of total available
• a percentage of the total server used, not available - effectively just increases the scale
• a percentage of the container maximum usage - useful for container relative comparison
• any other way to increase relative visibility..?

To be clear, this only applies to the container graphs. I think the host graph at the top and the container numbers are all fine as is. It would just be great to be able to change the container graph scaling so that anomalies quickly grab your attention.

Describe any alternatives you've considered

No response

[amir20]

I think what you are asking is instead of making the bar chart's max 100% it should be max(data). So essentially it auto adjusts. Which is I think it is what it was doing before with D3?

Can you provide screenshots? I want to understand what it looks like with 128GB.

[ambarbour]

Thanks for the quick reply!

Yes I agree with your suggestion, which is more or less my 3rd option and the one I would choose if I could. So if you just want to pick one calculation then I think that one is a winner.

Here is how it used to look, v8.14 (only showing first 10 containers..):

And here is how it looks with the new UI, v9.0:

[amir20]

Wait, I misunderstood. I thought you meant the container page, but you are referring to the home page.

However, the same chart is used on the container chart.

I question whether normalizing would be the right solution. If normalized, all bars would be at 100%, which is not your intention.

Can you think more mathematically about what should happen?

The one change in v9 is that CPU is now normalized. Previously, it was a value between 0 and (cores x 100%). Now, it ranges from 0 to 100%.

Based on your screenshots, it works as designed because now total RAM usage in bytes is displayed.

[ambarbour]

My apologies for not being clear, I should have said I was talking about the home page only.

All I really want is to have a clearly visible indicator that a container is using more memory than others. The numbers are fine, just not the graph scale. For example, in my first screenshot it is clear that the containers with more green are using more memory than the ones with barely any green. In the second screenshot it is not clear, I have to sort and/or look at the numbers.

Mathematically I realize that the main challenge is that the graph now represents a timeseries, not just a current value.

Let me think about this some more and I will comment further.

[amir20]

The space just doesn't allow for it to have a better option. I think for a single container it makes sense to normalize. But there needs to be a relative value to normalize.

One option is to take the biggest value amongst ALL containers and use that as that max. I am not sure if it makes sense to do that because to someone it might feel like it's using a lot when it's just the biggest value amongst a bunch of small numbers.

[amir20]

Moving to discussion.

[ambarbour]

I've been thinking about this and I definitely see your point that it is probably not possible to show the cpu/mem % usage as a timeseries with the visual impact we are looking for.

How about this: Can we have an option to view those metrics as current values, not timeseries? Similar to how it was pre-v9? Again this would apply ONLY to the container list view, NOT the container details view.

FWIW - without this option we will just continue to use v8, which is also totally fine. The main value of the container list view for us is to quickly see if a container is using excessive resources, right now.

[amir20]

That makes sense. A few of thoughts though:

1. I want to make sure it's clear that CPU % is now normalized based on total CPUs. Based on your old screenshot I don't think it would matter.
2. I think an elegant solution would be to have a button on top of the menu to switch between bar vs time series.
3. I would advise in not using v8. It has security holes that have been patched in v9.

I am currently working on alerting on v10. So I think this addition would be part of that.

[ambarbour]

1 - sounds fine
2 - agreed, even better than an option
3 - good to know, thanks

And once again, thank you for your prompt and thorough responses. Your commitment to this project is excellent.

[FrancYescO]

just to add my 2 cent, i've also noticed that this introduced "normalization" caused the graphs to be a little useless, before was very clear when a container spikes for cpu or ram usage, now the graph is pretty flat both on home and container page (24cpu and 256ram host).

noticed also that in the hover block when you are on a container, the load was the "correct" one like before

[amir20]

Correct is a bit subjective. 😜

What do you think should be the right way?

1. I don't think it makes sense to show 87% when it's really 24 CPUs. It's not using 87% of cores. It's using one. Trying to be consistent here with htop and top. Also have gotten a few people asking what 150% means.
2. The homepage could be changed to 0 -- 100% progress bar which is what @ambarbour suggested.
3. Even with that, I don't think it will solve your problem because it would still be a very tiny number.

I prefer to provide one solution that works for all. 🤷🏼

[FrancYescO]

how was working before? the previous UX at least for me was good

i think the correct way is something like (but maybe you can say that is already working in this way):

• cpu show the 0-100 value based on how many cores (limit) the container can use
• ram shows relative value, so the min of the graph is the actual value the max is the limit of the ram that the container can use

but something fully relative to the container, at leat in the container view, was still something good to have (i think this was more similar on how was working before) so something like:

• cpu: min 0 max the actual max value reached after the click of the container view
• ram: min the minium value reached by container while max the maximum value reached by container (maybe with a +-10%)

ps. this offsetted-relative-view is basically what also portainer does in the statistic page, the max scale is just a +10% to the value reached from the start of the "current viewing period"

[amir20]

I think we need a better solution @FrancYescO. I prototyped this and it doesn't look right.

Before

After

It makes it look like there is a lot of CPU when there actually just 0.1% jump.

I am not a statistician but perhaps its normalize the data based off a better forumula.

[FrancYescO]

i don't think it looks wrong, if you see the graph I've posted of portainer basically the same thing is happening (a big spike with just 10%) what help to understand is the max scale on the y of the graph... maybe print it in the top right of the graph?

[FrancYescO]

You will end up to have a filled of bars graph also if the container is on a stable 0.1% usage: is just what we are asking to see when putting it in a relative manner, obviously without showing somehow the max scale (that is dynamic and actually hidden for the user) the only reference is the actual usage and the bars can look "too big for what the container is doing"

[amir20]

I made some improvements. max(current_max * 1.2, some_minimum_floor). Let me send a PR.

[amir20]

Ok made some improvement on the container page only. #4464. Not the homepage. You can test it out amir20/dozzle:pr-4464 @FrancYescO .

[FrancYescO]

i can confirm that now at least there is some bar moment also on low load, still not really realative but is something

for ram this is more evident: i.e. i have a static use o 1GB/128GB, the graph is empty, in this case i'm expecting bars to occupy something like 80% of the height and go down only is there some spike in the usage: in other word i'm expecting to see an empty bars graph only when there was a spike in resource usage and than it goes down

[amir20]

In this case, 1GB/128GB should cover something right? Can you share a screenshot? But also I should point out that that's less than 1%. I am not sure if there is good option. The new code should have something though because I normalize to min(10%).

[FrancYescO]

it looks you are touching the wrong thing, you should not floor to a minium the value of the bar, but find a way for the bars to occupy the 80% of the graph base on the first value receive, than the previous bars can you down only if the next values are bigger: what i'm trying to say is that the bar value should not change in time neither should be calculated, only the value of the max Y (top right/left of the graph) should change and be (max registerd value * 1,2)

ps. also noticed that the graph are not instantly resetted when switching to another container, until the next load came you will end to have for few secs, statistic of the previous container on the newly clicked one

[amir20]

I am doing

dozzle/assets/components/BarChart.vue

Lines 44 to 47 in ee4bad5

	const maxValue = computed(() => {
	const dataMax = Math.max(0, ...downsampledData.value);
	return Math.max(dataMax * 1.2, maxCeiling);
	});

Where maxCeiling is 10. Then dataPoint / maxValue.

I am having a hard time understanding why you think it doesn't look good. I think the screenshots are pretty good. 0.1% shouldn't show much. And 0.1/10 is a very small number. But 3.6/10 is about 30% which is what you see.

On the memory, I think it works as expected. 6.9 / 10 should almost take the whole chart. That looks good. and 2.2 uses less which it has a smaller number.

but find a way for the bars to occupy the 80% of the graph base on the first value receive

I don't think that makes sense. Because then all bars will take 80%. More memory should take more space.

ps. also noticed that the graph are not instantly resetted when switching to another container, until the next load came you will end to have for few secs, statistic of the previous container on the newly clicked one

This is an existing bug.

I think your screenshots look pretty good! A lot better than before. 🤔

[FrancYescO]

I think your screenshots look pretty good! A lot better than before. 🤔

this no doubt

Because then all bars will take 80%.

yes, all bar will take 80% but only if there is no change in resource usage while looking

More memory should take more space

this is the pitfall, you are trying a way to merge the "relative to the container", and "absolute to host".
when i'm seeing a container view, mainly i don't care about the host i just wan't to have a clear view of any change of the the resource usage during the timeframe i'm looking at it.

i.e this is a container that started from, 700MB, reached 1,2GB and than fastly cleared 300MB going back to a stable 900MB: the graph is pretty flat instead the red line is what i was expecting too see

[amir20]

I see. You are right if changes 300MB it should be shown on the UI.

I don't think that's the case though. Dozzle only streams the last 5 minutes. So maybe we are discussing different things? Perhaps for you it shows it has changed in memory, but in the last 5 minutes it is flat so not much to report on?

Edit: Can you move your mouse over the bar to confirm that it actually has changed?

[FrancYescO]

yes, obviously the changes i've wrote happened in the 5minutes and the hovering mouse was showing the correct values

if you pay attention on the first 3 bars, and last 3 you can see that are a like 1px smaller

[amir20]

That seems like a bug then. 🤔 It should definitely show it.

[amir20]

It seems to be working fine for me.

I did the following commands

/ # dd if=/dev/zero of=/dev/shm/fill bs=1M count=400
400+0 records in
400+0 records out
419430400 bytes (400.0MB) copied, 0.117254 seconds, 3.3GB/s
/ # dd if=/dev/zero of=/dev/shm/fill bs=1M count=800
800+0 records in
800+0 records out
838860800 bytes (800.0MB) copied, 0.288501 seconds, 2.7GB/s
/ # dd if=/dev/zero of=/dev/shm/fill bs=1M count=300
300+0 records in
300+0 records out
314572800 bytes (300.0MB) copied, 0.139052 seconds, 2.1GB/s
/ #

Which produced 400MB then a 10 seconds later 800MB. I dropped to 300MB.

Now I don't have 98GB of RAM. 😅 But I would expect it to be similar for you with just smaller jumps. Even with just 150MB jump I saw the bar move up.

On the homepage it looks pretty nice too.

[amir20]

Going to merge for now. I am not sure how to emulate 98GB of ram. I wonder if there is a bug with that much memory.