We can use `--without-raster` flag compiling PostGIS 3 and remove GDAL requirement: https://postgis.net/docs/manual-dev/postgis_installation.html#install_requirements