Backends: GPU Accelerated Plot Rendering

[epezent]

Continues discussions related to GPU backend rendering started in #254 and #219. Follow progress on backends branch.

@marcizhu, @bear24rw

[marcizhu]

@epezent you mentioned on #254 that you did some tests with GPU acceleration for line plots (with some incredible results so far). How's it going? You need any help with that? I'm rather curious on how you managed to implement it

[epezent]

The gist is passing the user plot data to to the GPU unmodified in plot coordinates, using the plot size and bounds to compute the projection matrix for the vertex shader, line tessellation and mitering in a geometry shader, and then a basic frag shader. Results so far a good, but there is aliasing that I'm not happy with. I'm intrigued by the SDF method described here. I'm still very much in exploration mode.

[epezent]

Another issue is that my current solution requires a draw call per line, which obviously isn't ideal, but it's something that can be solved.

[marcizhu]

Huh, interesting. I never thought that sending the vertices straight to the GPU would have such a big impact on performance. I guess the CPU struggles with all those matrix multiplications...

[epezent]

@marcizhu, @bear24rw I did a bit of rework on heatmaps. Mainly switching to using ImPool<HeatmapData>, adding ShowBackendMetrics for debug purposes, and a few other things. I expect to make several more refactors and generalizations once I add line rendering.

One thing I'm wondering is if there's a way we can combine SetHeatmapData, RenderHeatmap, and SetLogAxis into a single function, especially if they have to be called in a particular order anyway. I'm also not sure we really need all the overloads for each data type. I think something like this would be easier to maintain:

void RenderHeatmap(int itemID,
                    void* data,
                    ImGuiDataType data_type,
                    int rows,
                    int cols,
                    float scale_min,
                    float scale_max,                      
                    const ImPlotPoint& bounds_min,
                    const ImPlotPoint& bounds_max,
                    bool logx,
                    bool logy,
                    bool reverse_y,        
                    ImPlotColormap cmap,
                    ImDrawList& DrawList);

I avoid ImGuiDataType in the main API and prefer templates for compile time optimizations, but the advantage of using that here (if there is any) will pale in comparison to the GPU gains. So to keep the requirements on backends low, I suggest we just have one function. Internally we just need to map data_type to GL_FLOAT, GL_INT, etc using a switch case or similar. Since RenderHeatmap is called from a template function in implot_items.cpp, we'll need a template specialization to figure out what to provide for data_type (I've gone ahead and added this to implot_items.cpp, even if we decide against it, as it seems useful):

template <typename T>
struct ImGuiDataTypeGetter { static const ImGuiDataType Value; };
template <> const ImGuiDataType ImGuiDataTypeGetter<float>::Value  = ImGuiDataType_Float;
template <> const ImGuiDataType ImGuiDataTypeGetter<double>::Value = ImGuiDataType_Double;
template <> const ImGuiDataType ImGuiDataTypeGetter<ImU8>::Value   = ImGuiDataType_U8;
template <> const ImGuiDataType ImGuiDataTypeGetter<ImS8>::Value   = ImGuiDataType_S8;
template <> const ImGuiDataType ImGuiDataTypeGetter<ImU16>::Value  = ImGuiDataType_U16;
template <> const ImGuiDataType ImGuiDataTypeGetter<ImS16>::Value  = ImGuiDataType_S16;
template <> const ImGuiDataType ImGuiDataTypeGetter<ImU32>::Value  = ImGuiDataType_U32;
template <> const ImGuiDataType ImGuiDataTypeGetter<ImS32>::Value  = ImGuiDataType_S32;
template <> const ImGuiDataType ImGuiDataTypeGetter<ImU64>::Value  = ImGuiDataType_U64;
template <> const ImGuiDataType ImGuiDataTypeGetter<ImS64>::Value  = ImGuiDataType_S64;
...
ImGuiDataType data_type = ImGuiDataTypeGetter<T>::Value;

If you guys agree, @marcizhu can you make the change for us?

[marcizhu]

@marcizhu, @bear24rw I did a bit of rework on heatmaps. Mainly switching to using ImPool<HeatmapData>, adding ShowBackendMetrics for debug purposes, and a few other things. I expect to make several more refactors and generalizations once I add line rendering.

Yeah I saw it and it's awesome! This is something I considered adding in my PR but in the end I forgot about it. Good thing you added ShowBackendMetrics()!

If you guys agree, @marcizhu can you make the change for us?

I think it can be done in a single function (indeed it should have been implemented as a single function; but since I added features one by one I implemented each step in a dedicated function:man_facepalming:). I can make the change tomorrow and open a PR if you guys want.

[epezent]

Sounds good to me!

[leeoniya]

this might be useful: https://blog.mapbox.com/drawing-antialiased-lines-with-opengl-8766f34192dc

i think the resulting shader code is available under the mapbox-gl repo.

or i guess this fork prior to their non-oss license switch: https://github.com/maplibre/maplibre-gl-js

[epezent]

Yep, that along with these have been good resources:
https://vicrucann.github.io/tutorials/osg-shader-3dlines/
https://www.codeproject.com/Articles/199525/Drawing-nearly-perfect-2D-line-segments-in-OpenGL
https://mattdesl.svbtle.com/drawing-lines-is-hard
https://atyuwen.github.io/posts/antialiased-line/
http://jcgt.org/published/0002/02/08/paper.pdf

[leeoniya]

i would be super interested if something came out of this effort that can be used via webgl or webgl2 canvas contexts. it would make it possible for me to explore porting uPlot to a webgl backend without additional overhead from using something like Pixi.js or Two.js

[epezent]

I have some very early results to share for GPU line rendering. My implementation is super naive at the moment, but I'm quite happy with the results so far.

First, we basically get AA for free. Note the difference between ImPlot default lines, and GPU lines (note MSAA is disabled)

Default

GPU

Second, we get pretty good performance gains for high volumes of points (1M tested here). Here are results recorded on my laptop, using the integrated (Intel i7 9750H, UHD Graphics 630) and discrete (RTX 2080 Max-Q) GPUs.

Lines	Points Ea.	Default (Intel)	GPU (Intel)	Default (RTX 2080)	GPU (RTX 2080)
1	1,000,000	27 FPS	137 FPS	24 FPS	430 FPS
10	100,000	27 FPS	135 FPS	24 FPS	164 FPS
100	10,000	27 FPS	127 FPS	24 FPS	127 FPS
1,000	1,000	27 FPS	85 FPS	24 FPS	123 FPS
10,000	100	27 FPS	15 FPS	23 FPS	16 FPS

GPU performance is currently bound by the number of lines because I'm making a call to glDrawArrays for every line. Batching calls is the obvious next step, and it should be possible to regain most of the performance seen in the 1x1,000,000 case (though unfortunately I don't see a straightforward way to batch across multiple plots because of the render order). I am able to plot a 10,000,000 point line at a constant 60 FPS on my discrete GPU, which is pretty silly but also awesome! There are several other small optimizations yet to be made, and I don't think I've found the best possible shader solution yet either.

[marcizhu]

This results are quite impressive! And getting AA for free is a nice bonus 😄
I think batching all lines together is gonna be a difficult challenge, but it might be worth the effort. The only thing I can think of is somehow storing the pointers to the array of points on each PlotLine() call and then submit all lines at once when EndPlot() is called. Not the best solution, I know...
Once this line acceleration is implemented, we can try doing the same for shaded plots and I think after that we will have GPU acceleration for all plots that need it!

[epezent]

Once this line acceleration is implemented, we can try doing the same for shaded plots and I think after that we will have GPU acceleration for all plots that need it!

Also scatter/point plots, which we can render as particles.

[epezent]

I now have all lines drawn to a plot batched into a single draw call. Now, 1M points render in about the same amount of time (~150 FPS) regardless of the number of lines. Not quite as impressive as the 430 FPS I posted above. That's because this comes at the expense of having to include line color in the vertex data (struct PlotVertex { float x, y; ImU32 col }) instead of updating each line color via a shader uniform. Unfortunately, that's a 50% increase in the amount of data that needs to be loaded to the GPU each frame, so it seems I'm CPU bound for now. I'm wondering if you guys have any other ideas? Fortunately, I'm still able to use GL_LINE_STRIP by leveraging glPrimitiveRestartIndex to split the strip between different lines, versus using GL_LINES which would require redundant points.

Below are my current shader programs. I think I can get a little more out of the vertex shader by lumping the plot-to-pixel transform into the projection matrix on the CPU side. The geometry shader only draws 1-px thick lines for now, but I have a more general variant that can do thick lines for about the same cost. I explored mitering lines here, but it didn't seem necessary or worth the expense.

line.vert

#version 330 core
layout (location = 0) in vec2 Point;
layout (location = 1) in vec4 Color;
uniform mat4 ProjMtx;
uniform vec2 M;
uniform vec2 PltMin;
uniform vec2 PixMin;
out vec4 VertColor;

void main()
{
    VertColor = Color;
    vec2 pix = PixMin + M * (Point - PltMin);
    gl_Position = ProjMtx * vec4(pix, 0.0, 1.0);
}

line.geom

#version 330 core
layout (lines) in;
layout (triangle_strip, max_vertices = 6) out;
uniform vec2 Scale;
in vec4 VertColor[];
out vec4 GeomColor;  

void main() 
{   
    vec2 p1 = gl_in[0].gl_Position.xy / Scale;
    vec2 p2 = gl_in[1].gl_Position.xy / Scale;
    vec2 v = p2 - p1;
    vec2 n = normalize(v);

    GeomColor = vec4(VertColor[0].rgb, 0);
    gl_Position = vec4(vec2(p1.x + n.y, p1.y - n.x) * Scale, 0, 1);
    EmitVertex();
    GeomColor = vec4(VertColor[1].rgb, 0);
    gl_Position = vec4(vec2(p2.x + n.y, p2.y - n.x) * Scale, 0, 1);
    EmitVertex();

    GeomColor = VertColor[0];
    gl_Position = vec4(p1 * Scale, 0, 1);
    EmitVertex();
    GeomColor = VertColor[1];
    gl_Position = vec4(p2 * Scale, 0, 1);
    EmitVertex();

    GeomColor = vec4(VertColor[0].rgb, 0);;
    gl_Position = vec4(vec2(p1.x - n.y, p1.y + n.x) * Scale, 0, 1);
    EmitVertex();
    GeomColor = vec4(VertColor[1].rgb, 0);;
    gl_Position = vec4(vec2(p2.x - n.y, p2.y + n.x) * Scale, 0, 1);
    EmitVertex();

    EndPrimitive();
}  

line.frag

#version 330 core
out vec4 FragColor;
in vec4 GeomColor;

void main()
{
    FragColor = GeomColor;
}

[icyveins7]

Sorry if this is a dumb question, but I see the heatmap GPU acceleration implementation in backends, but not @epezent 's line plots. Was this merged in at any point? It would be great to use, even if not perfect by any means (that speedup would be of exceptional use to me).

[samangh]

@epezent, first of all thank you for your amazing work on implot!

I am curious about the status of the backend branch, especially with regards to line plots? I usually end up working with quite large datasets, so this would be very useful!

[samangh]

@epezent, is there any chance that you could push the changes you made for the GPU-accelerated line plots to the backends branch?

I would be happy to help with further work on this (although sadly I don't have any OpenGL experience).

[keltonhalbert]

I'd love to contribute or assist in any way I can in regards to shader accelerated plots.

[traceon]

Hi, do you have any plans to sync backends with current master? Looks like it is somewhat behind currently.

[JamesWrigley]

Gentle bump, just wondering what it would take to merge backends into master? I'm particularly interested in the GPU-accelerated heatmap.

[brenocq]

I'm currently working on the backend implementation for ImPlot3D (brenocq/implot3d#172), hopefully after I'm done with this, I can look into ImPlot's backend

[JamesWrigley]

Awesome, thanks ❤️

[samangh]

I'm currently working on the backend implementation for ImPlot3D (brenocq/implot3d#172), hopefully after I'm done with this, I can look into ImPlot's backend

Great! Thank you @brenocq for stepping up as a new maintainer! It's good see that this very useful library is under active development again ❤️