Updating surfaces in "real time" for persistance

[eniv]

Hi,
I'm using WebglSquare as a surface for a dynamic xy scatter scatter plot as suggested in #72.
I'm trying to create a persistence function where points from previous frames will not be cleared when a new frame is drawn.

The most naive way of doing that is holding a FIFO (first in first out) type, fixed size, buffer in CPU RAM which will hold the contents of all the points from the current frame and previous frames. This buffer can then be transferred to webgl-plot for rendering after all the surfaces are removed (Adding removeSurfaces, equivalent to removeXXXXLines to webgl-plot would be nice).
Using this technique I can preserve about ~4000 points before the frame rate starts dropping.

Can you recommend a better way of doing this? It seems redundant to buffer the same data to the GPU over and over again until it is removed from the FIFO.

[danchitnis]

Hello, thanks for getting in touch. I see your point, persistence makes sense for scatter graphs. As you mentioned, the problem is with overloading. There needs to be a kind of rolling buffer so that it can retain the last 4000 points. The problem is Javascript doesn't have a native rolling buffer, and it is hard to make one because it doesn't have access to low-level memory as C/C++ does. I will do some tests to come up will a workable solution. The GPU method is better. Each object needs an id. I haven't tried removing objects in OpenGL. Again need to do some tests.

[danchitnis]

@eniv Okay, the WebGL code was more challenging than I expected. But I managed to do it. At the moment, it is single-size, single-color squares. I plotted 1M squares with 100k square updates per frame at 60fps. I think that is beyond your 4000 sq limit! I still need to do more work to integrate it in the webglplot and create API etc. The rolling line graph will also get a new code. To be clear, it has a MaxSquare capacity, and when you add a new square, it overwrites the oldest one when it reaches capacity.

[eniv]

To be clear, it has a MaxSquare capacity, and when you add a new square, it overwrites the oldest one when it reaches capacity.

Exactly, just like a shift register.

This is awesome!
I'm looking forward to trying it out.

[danchitnis]

Here is an example that I made. The new version is at webgl-plot@next. Feel free to play around. Notice that in this example, the maxSquare should be a multiple of newDataSize, otherwise some data will be lost. The API is at a very early stage, so it may change.

[eniv]

Excellent!
I modified the example for maxSquare of 1e6 squares and newDataSize of 1e5 squares and everything runs smoothly. This looks very very promising.

[eniv]

Hi,
I experimented with this more today.
First of all, thank you!

These are a couple of items which I've noticed so far:

1. It will be really nice if there was a clear function in WebglScatterAcc for clearing the buffer. It is useful for switching persistence OFF.
2. WebglScatterAcc requires its own canvas and therefore can't be easily integrated with other shapes (line, polar, ...). This means that adding a grid for example, will become more complex. Sharing a single canvas under a single context as was previously done in WebglPlot was advantageous and allowed more sophisticated plots

-Eyal

[danchitnis]

Thanks. I had already made an update. There were a few bugs in the previous version, also added color now. You can see the old example updated, and also this new example. On my desktop (i7 9th gen with RTX A4000), I could go up to 6M maxSquares and 100k newData at 60 fps. There are two bottlenecks. The maxSquare is determined by GPU cores (not memory), and the new data is determined by the CPU thread. The data rate of the newData is equal to 528 Mbps, and the GPU core utilization is 50% which more than some of the games out there! It would be interesting to see how the python and C++ versions of this perform (they are in the works!)

[danchitnis]

It will be really nice if there was a clear function in WebglScatterAcc for clearing the buffer. It is useful for switching persistence OFF.

Yes, I can see a way to do that temporarily. But the best is to set the maxSquare equal to the newData

WebglScatterAcc requires its own canvas and therefore can't be easily integrated with other shapes (line, polar, ...). This means that adding a grid for example, will become more complex. Sharing a single canvas under a single context as was previously done in WebglPlot was advantageous and allowed more sophisticated plots

yes, this is the next priority. I am working on it. All of this will be part of the next version so it will be a re-write of the entire library.

[danchitnis]

Updated for the "auxiliary" lines. Example here.

[rhard]

Hi @danchitnis, I tried the scatter plot from your latest example here: https://codesandbox.io/s/webglscatteracc2-psutjy?file=/src/index.ts

I have two issues with it:

1. There is a white rectangle in the middle of the plot, visible in the first ~1 sec of rendering. You can also see the issue in your example:

2. The second issue is more interesting: as soon the initial rectangle goes away, my plot slows down to some 0.5 frames per second and remains slow. I don't see any performance degradation in general, and my drawing callback still executed fast. In the first second, the plot updates normally (~15 fps).

Maybe these two issues have the same origin?

Here is my drawing code. I use Tauri + SvelteKit frameworks. The version of webgl-plot is 1.0.2.

<script lang="ts">
	import { WebglPlot, WebglScatterAcc, WebglAux, ColorRGBA } from 'webgl-plot';
	import { onMount } from 'svelte';
	import { listen, emit } from '@tauri-apps/api/event';

	const horizontal_resolution = 15;
	const vertical_resolution = 11;

	let canvas: HTMLCanvasElement;
	let wglp: WebglPlot;
	let aux: WebglAux;
	let sqAcc: WebglScatterAcc;

	const maxSquare = 165;
	let screenRatio: number;
	let sqSize = 0.06;
	let h_scale: number;
	let w_scale: number;
	let h_offset: number;
	let w_offset: number;

	 const pos = new Float32Array(horizontal_resolution * vertical_resolution * 2);
	 const colors = new Uint8Array(horizontal_resolution * vertical_resolution * 3);

	onMount(async () => {
		canvas = document.getElementById('my-canvas') as HTMLCanvasElement;
		const devicePixelRatio = window.devicePixelRatio || 1;
		canvas.width = canvas.clientWidth * devicePixelRatio;
		canvas.height = canvas.clientHeight * devicePixelRatio;
		screenRatio = canvas.width / canvas.height;

		wglp = new WebglPlot(canvas);
		wglp.gScaleX = screenRatio;
		wglp.gScaleY = 1;
		h_scale = 2 - sqSize * 2;
		w_scale = 1/screenRatio * (2 - sqSize*2*screenRatio);
		h_offset = (1 - sqSize);
		w_offset = (1/screenRatio - sqSize);
		sqAcc = new WebglScatterAcc(wglp, 1650);
		sqAcc.setSquareSize(sqSize);
		sqAcc.setColor(new ColorRGBA(255, 255, 0, 255));
		sqAcc.setScale(1/screenRatio, screenRatio);

		const unlisten = await listen('frame', (event) => {
			let current_data_idx = 0;
			let countX = 0;
			let countY = 0;

			for (let vi = 0; vi < vertical_resolution; vi++) {
				for (let hi = 0; hi < horizontal_resolution; hi++) {
					countX = hi / (horizontal_resolution - 1) * h_scale - h_offset;
    				        countY = vi / (vertical_resolution - 1) * w_scale - w_offset;
					pos[current_data_idx * 2] = countX;
					pos[current_data_idx * 2 + 1] = countY;
					colors[current_data_idx * 3] = (event.payload as Array<number>)[current_data_idx]/10;
    				        colors[current_data_idx * 3 + 1] = (event.payload as Array<number>)[current_data_idx]/10;
    				        colors[current_data_idx * 3 + 2] = (event.payload as Array<number>)[current_data_idx]/10;
					current_data_idx++;
				}
			}
			sqAcc.addSquare(pos, colors);
			wglp.clear();
 			sqAcc.draw();
		});
	});
</script>

<div class="h-full flex items-center">
	<canvas class="border-8 h-600 border-black w-818" id="my-canvas" />
</div>

[danchitnis]

@rhard Thanks for the feedback

There is a white rectangle in the middle of the plot, visible in the first ~1 sec of rendering. You can also see the issue in your example:

Correct, I need to investigate this.

The second issue is more interesting: as soon the initial rectangle disappears, my plot slows down to some 0.5 frames per second and remains slow. I don't see any performance degradation in general, and my drawing callback still executed fast. In the first second, the plot updates normally (~15 fps).

This doesn't happen on my side. If you are already running at 15 fps, that means you are severely resource-limited. Reduce the buffer size in the line below until you get the frame rate of your monitor, e.g. 60 fps. Monitor your resources to see if you are not out of RAM and if GPU is below 50%. It is expected that you run this on a high-end CPU and mid-range GPU.

const sqAcc = new WebglScatterAcc(wglp, 10000);

[rhard]

@danchitnis Thank you for your prompt response.

By 15 fps, I mean this is how fast my component receives the new data inside the event listener here, and I redraw the rectangles:

const unlisten = await listen('frame', (event) => {}

I also call wglp.clear() and sqAcc.draw() functions inside this handler. This is different from your examples, where you use requestAnimationFrame loop. But I also tried to make the same loop as yours and just add the rectangles inside my event handler, but the results were pretty the same.

How can I output the real rendering FPS? I don't feel real fps is degraded, and it looks like just the new data for drawing became lost or something. Furthermore, the slowdown happens exactly at the moment when the white rectangle in the middle disappears.

[rhard]

I've also tried to reduce/increase the buffer with different numbers. I even tried to make it to be x1 with just 165 pixels needed for one single frame. This doesn't make any visible difference.

[rhard]

@danchitnis I've created an example to mimic my use case here: example

It woks OK, but it doesn't have the white rectangle at the beginning. Something strange.

[danchitnis]

By 15 fps, I mean this is how fast my component receives the new d

It is a best practice that you run the plotting function in Javascript's render loop, otherwise may have unexpected results. To check the frame rate in Chrome, follow the instructions here and then Rendering -> Frame Rendering Stats

I am not familiar Svelte framework, but these usually modify the states and rendering loops, so it could be the issue is there.