Terrain and physics collider have difference from the same heightfield

[slapin]

When generating colliders for Terrain for both Bullet and Jolt Physics the resulting collider have
large enough differences from how geometry looks. I guess that is very long standing problem.
That never happens on flat terrain but the steeper it is the more the difference. It generally happens on some steeper slopes while even ground is usually 1-1 match. The steeper slopes get elevated on geometry side which means
whatever moves on surface will get under surface. On my observation it never exceeds 2m difference on my layout,
but it is still quite annoying. What can I do about it? I use 1024x1024 heightmap with large "endless" world so my terrain is quite smooth + I add some noise values to make it a bit more interesting and supply the generated heightfield to both terrain and to physics (I have to flip it a bit for physics, but that is not a problem).

Setup code:

#define TERRAIN_SIZE 65
#define TERRAIN_WORLD_SIZE 500.0f
#define ENDLESS_TERRAIN_FILE_PREFIX Ogre::String("EndlessWorldTerrain")
#define ENDLESS_TERRAIN_FILE_SUFFIX Ogre::String("dat")

#define ENDLESS_PAGING

// max range for a int16
#define ENDLESS_PAGE_MIN_X (-0x7FFF)
#define ENDLESS_PAGE_MIN_Y (-0x7FFF)
#define ENDLESS_PAGE_MAX_X 0x7FFF
#define ENDLESS_PAGE_MAX_Y 0x7FFF
...
				std::cout << "Terrain setup\n";
				if (!priv.mDummyPageProvider)
					priv.mDummyPageProvider =
						new DummyPageProvider(
							/* eng.mWorld
								->getBtWorld() */);
				terrain.mTerrainGlobals =
					OGRE_NEW Ogre::TerrainGlobalOptions();

				OgreAssert(terrain.mTerrainGlobals,
					   "Failed to allocate global options");

				terrain.mTerrainGroup =
					OGRE_NEW Ogre::TerrainGroup(
						eng.mScnMgr,
						Ogre::Terrain::ALIGN_X_Z,
						TERRAIN_SIZE,
						TERRAIN_WORLD_SIZE);
				terrain.mTerrainGroup->setFilenameConvention(
					ENDLESS_TERRAIN_FILE_PREFIX,
					ENDLESS_TERRAIN_FILE_SUFFIX);
				terrain.mTerrainGroup->setOrigin(
					terrain.mTerrainPos);
				// Configure global
				terrain.mTerrainGlobals->setMaxPixelError(1);
				// testing composite map
				// mTerrainGlobals->setCompositeMapDistance(30);
				terrain.mTerrainGlobals->setCompositeMapDistance(
					300);
				//mTerrainGlobals->setUseRayBoxDistanceCalculation(true);
				terrain.mTerrainGlobals
					->getDefaultMaterialGenerator()
					->setLightmapEnabled(false);

				terrain.mTerrainGlobals->setCompositeMapAmbient(
					eng.mScnMgr->getAmbientLight());
				terrain.mTerrainGlobals->setCompositeMapDiffuse(
					sun.mSun->getDiffuseColour());
				terrain.mTerrainGlobals->setLightMapDirection(
					sun.mSun->getDerivedDirection());

				// Configure default import settings for if we use imported image
				Ogre::Terrain::ImportData &defaultimp =
					terrain.mTerrainGroup
						->getDefaultImportSettings();
				defaultimp.terrainSize = TERRAIN_SIZE;
				defaultimp.worldSize = TERRAIN_WORLD_SIZE;
				defaultimp.inputScale = 1.0f;
				// defaultimp.minBatchSize = 33;
				defaultimp.minBatchSize = 5;
				defaultimp.maxBatchSize = 65;
				Ogre::Image combined;
				combined.loadTwoImagesAsRGBA(
					"Ground23_col.jpg", "Ground23_spec.png",
					"General");
				Ogre::TextureManager::getSingleton().loadImage(
					"Ground23_diffspec", "General",
					combined);
				defaultimp.layerList.resize(1);

				defaultimp.layerList[0].worldSize = 60;
				defaultimp.layerList[0].textureNames.push_back(
					"Ground23_diffspec");
				// Paging setup
				terrain.mPageManager =
					OGRE_NEW Ogre::PageManager();
				// Since we're not loading any pages from .page files, we need a way just
				// to say we've loaded them without them actually being loaded
				terrain.mPageManager->setPageProvider(
					priv.mDummyPageProvider);
				terrain.mPageManager->addCamera(camera.mCamera);
				terrain.mPageManager->setDebugDisplayLevel(0);
				terrain.mTerrainPaging =
					OGRE_NEW Ogre::TerrainPaging(
						terrain.mPageManager);
				terrain.mPagedWorld =
					terrain.mPageManager->createWorld();

				terrain.mTerrainPagedWorldSection =
					terrain.mTerrainPaging
						->createWorldSection(
							terrain.mPagedWorld,
							terrain.mTerrainGroup,
							300, 500,
							ENDLESS_PAGE_MIN_X,
							ENDLESS_PAGE_MIN_Y,
							ENDLESS_PAGE_MAX_X,
							ENDLESS_PAGE_MAX_Y);
				terrain.definer = OGRE_NEW FlatTerrainDefiner(
					eng.mScnMgr /*, eng.mWorld */);

				terrain.mTerrainPagedWorldSection->setDefiner(
					terrain.definer);

				terrain.mTerrainGroup->freeTemporaryResources();
				std::cout << "Terrain setup done\n";
				ECS::get().set<PlacementObjects>({});
				terrain.mTerrainGroup->loadAllTerrains(true);

definer:

class FlatTerrainDefiner
	: public Ogre::TerrainPagedWorldSection::TerrainDefiner,
	  public Ogre::FrameListener {
	Ogre::SceneManager *mScnMgr;
	// Ogre::Bullet::DynamicsWorld *mWorld;
	struct gen_collider {
		Ogre::TerrainGroup *group;
		long x;
		long y;
	};
	std::deque<struct gen_collider> collider_queue;
	std::deque<struct gen_collider> colliderRemove_queue;

public:
	FlatTerrainDefiner(Ogre::SceneManager *
				   scm /*, Ogre::Bullet::DynamicsWorld *world */)
		: Ogre::TerrainPagedWorldSection::TerrainDefiner()
		, Ogre::FrameListener()
		, mScnMgr(scm)
	// , mWorld(world)
	{
		Ogre::Root::getSingleton().addFrameListener(this);
	}

private:
	std::mutex mtx;

public:
	void createTerrainChunk(Ogre::TerrainGroup *terrainGroup, long x,
				long y)
	{
		Ogre::Terrain *terrain = terrainGroup->getTerrain(x, y);
		float minH = terrain->getMinHeight();
		float maxH = terrain->getMaxHeight();
		uint16_t terrainSize = terrainGroup->getTerrainSize();
		OgreAssert(terrain && terrain->getHeightData() &&
				   terrain->isLoaded(),
			   "invalid terrain supplied");
		uint16_t size = terrain->getSize();
		float *heightData = terrain->getHeightData();
		Ogre::SceneNode *node = terrain->_getRootSceneNode();
		float worldSize = terrain->getWorldSize();
		float scaled = worldSize / (size - 1);
		Ogre::Vector3 bodyPosition = terrain->getPosition();
		// bodyPosition.y += (maxH + minH) / 2.0f;
		bodyPosition.x += worldSize / 2.0f;
		bodyPosition.z += worldSize / 2.0f;
		Ogre::Vector3 offset =
			node->_getDerivedPosition() - bodyPosition;
		flecs::entity e = PhysicsModule::createTerrainChunkBody(
			node, heightData, offset,
			Ogre::Vector3(scaled, 1, scaled), size);
		node->getUserObjectBindings().setUserAny("_collider", e);
	}
	void define(Ogre::TerrainGroup *terrainGroup, long x, long y) override
	{
		std::lock_guard<std::mutex> guard(mtx);
		uint16_t terrainSize = terrainGroup->getTerrainSize();
		float *heightMap = OGRE_ALLOC_T(float, terrainSize *terrainSize,
						MEMCATEGORY_GEOMETRY);
		//	float *heightMapCollider = OGRE_ALLOC_T(
		//		float, terrainSize *terrainSize, MEMCATEGORY_GEOMETRY);
		//		Ogre::Vector2 worldOrigin =
		//			Ogre::Vector2(img.getWidth(), img.getHeight()) * 0.5f;
		float chunk = 128.0f;
		Ogre::Vector2 revisedValuePoint;
		Ogre::Vector3 worldPos;
		terrainGroup->convertTerrainSlotToWorldPosition(x, y,
								&worldPos);
		for (int i = 0; i < terrainSize; i++)
			for (int j = 0; j < terrainSize; j++) {
				long world_x = (long)(worldPos.x + j -
						      (terrainSize - 1) / 2);
				long world_y = (long)(worldPos.z + i -
						      (terrainSize - 1) / 2);
				float height = 0.0f;
				height +=
					HeightData::get_singleton()->get_height(
						terrainGroup, world_x, world_y);

				// height = -2.0f;
				heightMap[i * terrainSize + j] = height;
				//				heightMapCollider[(terrainSize - i - 1) *
				//							  terrainSize +
				//						  j] = height;
			}
		terrainGroup->defineTerrain(x, y, heightMap);
		Ogre::LogManager::getSingleton().logError(
			"defined terrain at " +
			Ogre::StringConverter::toString(x) + " " +
			Ogre::StringConverter::toString(y));
		//		collider_queue.push_back(
		//			{ terrainGroup, x, y, heightMapCollider });
		delete[] heightMap;
		collider_queue.push_back({ terrainGroup, x, y });
	}
	bool frameStarted(const Ogre::FrameEvent &evt) override
	{
		(void)evt;
		update();
		return true;
	}
	void update()
	{
		std::lock_guard<std::mutex> guard(mtx);
		static bool created = false;
		while (!collider_queue.empty()) {
			Ogre::TerrainGroup *group =
				collider_queue.front().group;
			if (group->isDerivedDataUpdateInProgress())
				break;
			long x = collider_queue.front().x;
			long y = collider_queue.front().y;
            // std::cout << x << " " << y << " "
            //	  << collider_queue.size() << std::endl;
			Ogre::Terrain *terrain = group->getTerrain(x, y);
			Ogre::Vector3 worldPos;
			group->convertTerrainSlotToWorldPosition(x, y,
								 &worldPos);
#if 0
            std::cout << "terrain: " << terrain;
            if (terrain)
                std::cout
                    << terrain->getHeightData() << " "
                    << terrain->isLoaded() << " "
                    << terrain->isDerivedDataUpdateInProgress()
                    << std::endl;
#endif
			if (terrain && terrain->getHeightData() &&
			    terrain->isLoaded() &&
			    !terrain->isDerivedDataUpdateInProgress()) {
				Ogre::LogManager::getSingleton().logError(
					"can create collider for " +
					Ogre::StringConverter::toString(x) +
					" " +
					Ogre::StringConverter::toString(y));
				// float minH = terrain->getMinHeight();
				// float maxH = terrain->getMaxHeight();
				int size = terrain->getSize();
				float worldSize = terrain->getWorldSize();
				{
					createTerrainChunk(group, x, y);
				}
#if 0
				if (true) {
					btRigidBody *body =
						mWorld->addTerrainRigidBody(
							group, x, y, 2,
							0x7ffffffd & (~16));
					OgreAssert(
						body,
						"Could not create RigidBody");
					Ogre::LogManager::getSingleton().logError(
						"created rigid body " +
						Ogre::StringConverter::toString(
							Ogre::Bullet::convert(
								body->getWorldTransform()
									.getOrigin())));
					Ogre::LogManager::getSingleton().logError(
						"minHeight " +
						Ogre::StringConverter::toString(
							minH));
					Ogre::LogManager::getSingleton().logError(
						"maxHeight " +
						Ogre::StringConverter::toString(
							maxH));
					Ogre::LogManager::getSingleton().logError(
						"size " +
						Ogre::StringConverter::toString(
							size));
					Ogre::LogManager::getSingleton().logError(
						"world size " +
						Ogre::StringConverter::toString(
							worldSize));
					Ogre::LogManager::getSingleton().logError(
						"created collider for " +
						Ogre::StringConverter::toString(
							x) +
						" " +
						Ogre::StringConverter::toString(
							y));
					created = true;
				}
#endif
				// FIXME: create entities and components instead
#if 0
				Ogre::SceneNode *items =
					terrain->_getRootSceneNode()
						->createChildSceneNode();
				for (int i = 0; i < ECS::get<PlacementObjects>()
							    .altar_items.size();
				     i++) {
					const struct PlacementObjects::item &item =
						ECS::get<PlacementObjects>()
							.altar_items[i];
					Ogre::Entity *ent =
						group->getSceneManager()
							->createEntity(
								item.entity);
					Ogre::SceneNode *what =
						items->createChildSceneNode();
					what->attachObject(ent);
					what->setOrientation(item.rotation);
					what->setPosition(item.position);
				}
#endif
				/* Spawn items */
				StaticGeometryModule::addGeometryForSlot(x, y);
				collider_queue.pop_front();

			} else {
				/* Terrain data not ready maybe move to next terrain */
				gen_collider m = collider_queue.front();
				collider_queue.pop_front();
				collider_queue.push_back(m);
				break; // allow system to move on
			}
		}
		if (collider_queue.empty() &&
		    !ECS::get<Terrain>().mTerrainReady) {
			ECS::get_mut<Terrain>().mTerrainReady = true;
			ECS::modified<Terrain>();
		}
	}
};
class DummyPageProvider : public Ogre::PageProvider {
public:
	DummyPageProvider(/* btDynamicsWorld *world */)
		: Ogre::PageProvider()
	{
	}
	bool prepareProceduralPage(Ogre::Page *page,
				   Ogre::PagedWorldSection *section)
	{
		return true;
	}
	bool loadProceduralPage(Ogre::Page *page,
				Ogre::PagedWorldSection *section)
	{
		return true;
	}
	bool unloadProceduralPage(Ogre::Page *page,
				  Ogre::PagedWorldSection *section)
	{
		long x, y;
		ECS::get<Terrain>().mTerrainGroup->unpackIndex(page->CHUNK_ID,
							       &x, &y);
		StaticGeometryModule::removeGeometryForSlot(x, y);
		return true;
	}
	bool unprepareProceduralPage(Ogre::Page *page,
				     Ogre::PagedWorldSection *section)
	{
		return true;
	}
};

[slapin]

These images kind of demonstrate the problem

See the half-buried character at the center of the image? That is there.
And leaving steep edge to more flatter surface completely remove the problem.
It looks like Terrain kind of invents some details which are not present in heightfield,
as Jolt (white lines) indicates. And the steeper the edge the bigger the difference between plain heightfield and the terrain.

[paroj]

did you already use https://pybullet.org/Bullet/BulletFull/classbtHeightfieldTerrainShape.html#a1e1500e53130eaabb2d3701168d381c2 ?

[slapin]

With Bullet it gives better result. However I need it to work with Jolt too (I am becoming fan of Jolt because of how easy it is to use)... However on low resolution terrain neither is perfect...

[slapin]

If interested, my Jolt Physics wrapper code:
https://github.com/slapin/jolt-ogre-wrapper

[slapin]

I don't see a support for zigzag pattern in Jolt Physics heightfield collision...
Time to bug Jolt Physics author about it...

[slapin]

Created
jrouwe/JoltPhysics#1904

[slapin]

Lets try plan B and try to disable zigzag pattern on the terrain as Jolt author would prefer not to do it. jrouwe/JoltPhysics#1904 (comment)

[slapin]

I mean add option to disable it.

[slapin]

I guess there are 2 options here for Jolt Physics:

1. Implement custom shape for Ogre terrain. https://jrouwe.github.io/JoltPhysics/#creating-custom-shapes
2. Add option to disable zigzag pattern.