How to render 400k+ points vector symbol layer properly?

[Duoquote]

As I stated in question, I have just over 400k+ points to render, I am fetching them from postgresql as mvt and rendering with this;

    const map = new maplibregl.Map({
      container: mapContainer,
      style: HOST + "/api/v1/maps/styles/",
      center: [35.5532715, 38.9798405],
      zoom: 5,
      pixelRatio: 1,
      canvasContextAttributes: { antialias: true },
      transformRequest: (url, resourceType) => {
        if (["SpriteJSON", "SpriteImage"].includes(resourceType) || url.startsWith(HOST)) {
          return {
            url,
            headers: {
              "x-csrftoken": axios.defaults.headers.common["X-CSRFToken"],
              "x-session-id": axios.defaults.headers.common["X-Session-ID"],
            },
            credentials: "include"
          };
        }
      },
    });

    map.scrollZoom.setZoomRate(10);
    map.scrollZoom.setWheelZoomRate(10);
    map.dragRotate.disable();
    map.keyboard.disable();
    map.touchZoomRotate.disableRotation();

    map.on("load", () => {
      map.on("zoomend", () => {
        const zoom = map.getZoom();
        if (zoom !== Math.round(zoom)) {
          map.setZoom(Math.round(zoom));
        }
      });

      map.addSource("test", {
        type: "vector",
        tiles: [`${HOST}/api/v1/external/map/{z}/{x}/{y}.pbf`],
      });
      setMap(map);

      axios({
        method: "get",
        url: "/api/v1/external/sprites/generate/",
      }).then(() => {
        map.addSprite("pois", `${HOST}/api/v1/external/sprites/sprite`);
        map.addLayer({
          "id": "test-layer",
          "type": "symbol",
          "source": "test",
          "source-layer": "points",
          "layout": {
            "icon-image": ["get", "icon"],
            "icon-overlap": "always",
          }
        });
      })
    });

I have tried various combinations like, limiting resulting points in tiles or even result set but it always renders a little problematic, for example the following screenshot is hard limited in backend per tile;

I have also tried sending map bounding box inside transformRequest to backend but it introduces another issue, the bounding box retrieval is sometimes delayed providing last state (understandable for sure, I hoped it fetches tiles after the bounding box is changed).

I have tried rendering it all at once but it results in this issue.

What's the better way to render these points? It doesn't have to be using MVT/tile solution or an easy solution, I have control over backend too I can implement whatever fits best, it also have to be performant.

[HarelM]

The short answer is: don't render 400k points on the screen.
You might want to use heatmap layer or any other type of filtering to show data that a user can understand, and not a blur of points.
You can do some server side or client side clustering maybe and show the number of points in an area and as you zoom in show more details...

[Duoquote]

I asked it wrong I think sorry, my intention wasn't rendering 400k points at the same time, I would like to render less if possible but as I tried limiting the output, it doesn't create a meaningful renders. We could render for example at zoom level 8 10k points representing there are some points at some location, but for some reason (I guess tiles are cached or something) it shows up like last cached tile or something.

[HarelM]

10k is still too much, you should be aiming for 10s of points at any zoom level...

[Duoquote]

I agree with that but it's not really that demanding and clients like the look, and it shows there are some density for some locations (I know we could use heatmap for this but this is gonna be our marker layer, I will also have a heatmap layer alongside of symbol layer).

[acalcutt]

My suggestion is to use something like https://github.com/felt/tippecanoe to create a vector tileset which limits the number of points on the screen at lower zooms. tippecanoe can process your data so high zoom levels have all the details, but lower zoom levels limit the number of points

For example, in my wifidb project, I export all my points to geojson with this process, then use tippecanoe to process it into something usable with maplibre with this process. That gets used in this map.

Like mentioned above, without doing that, maplibre doesn't really handle large amounts of points being displayed at once. In my project I set the max for dynamic points to either 25k/50k limit, and that is a bit slow (though that partially may be the sql query speed in my case). For example, this signal map page for a access point with millions of points ( https://wifidb.net/wifidb/opt/map.php?func=exp_ap_sig&labeled=0&id=1041816&from=0&inc=50000&sig_label=none ) you can see at the bottoms of the page I added links for different ranges of points to limit to a max of 50k

[acalcutt]

In my opinion, the screenshot looks a bit dense when zoomed out and wouldn't hurt to cut it down some at that level.

[Duoquote]

I kept it fixed per tile, at 1-10 zoom levels I am just showing some points as representative. As the limit is fixed, when we zoom enough all the points are present.

The limit is actually an error on my side, its a webgl related issue at 32k points, but there is really no point to render 32k points in a tile anyways.

EDIT:

It's quite performant with 3k points in a tile so I guess I will keep it that way, important thing for me is to display it dense if there are that much of points.

[acalcutt]

in tippecanoe they have an option that I use '--drop-densest-as-needed'

If the tiles come out too big, the --drop-densest-as-needed option will make Tippecanoe try dropping what should be the least visible features at each zoom level. (If it drops too many features, use -x to leave out some feature attributes that you didn't really need.)

Which makes the dropped points less visible.

There are also other dropping options.

[Duoquote]

Thanks for the suggestions but it would be a bigger overhead with tippiecanoe for what it is currently I have, and as the postgres already provides mvt tiles I don't think I should add another library in the project that does the same. The filtering is also happens on server side and interactively on the ui, when some filter is applied I would have to create the layer all over again. The layer is not going to be static. And the project already is a few gigabytes over 4+ years, there is not much possibility to fit tippecanoe here. But thanks anyways!

[acalcutt]

ya, in your case it probably would not make sense, but just mentioning it because one of your issues seems to be not getting to choose what is getting dropped.