It seems to cause serious lag in play mode if the "Asset Quick Access" window is open with a GameObject referenced from the scene, but the GameObject is later removed from the scene without being removed from the "Asset Quick Access" window.
In our case, using Unity 2022.3.22 with Terrains, this issue appears to cause significant lag. Disabling the terrain seems to help, but some lag remains. Perhaps certain game elements cause the window to redraw more frequently, and the tool trying to find the missing GameObject reference contributes to the lag?