Overview

This repository collects summaries of over 300 recent studies on 3D scene generation, along with the downstream applications, and will be continuously updated.

If you have suggestions for new resources, improvements to methodologies, or corrections for broken links, please don't hesitate to open an issue or submit a pull request. Contributions of all kinds are welcome and greatly appreciated.

Table of Contents

• Overview
  • Table of Contents
• Methods: A Hierarchical Taxonomy
  • Procedural Generation
    • Rule-based Generation
    • Optimization-based Generation
    • LLM-based Generation
  • Neural-3D Generation
    • Scene Parameters
    • Scene Graph
    • Semantic Layout
    • Implicit Layout
  • Image-based Generation
    • Holistic Generation
    • Iterative Generation
  • Video-based Generation
    • Two-stage Generation
    • One-stage Generation
• Datasets
  • Indoor Datasets
  • Natural Datasets
  • Urban Datasets
• Applications and Tasks
  • 3D Scene Editing
  • Human-Scene Interaction
  • Embodied AI
  • Robotics
  • Autonomous Driving

Methods: A Hierarchical Taxonomy

Procedural Generation

Rule-based Generation

Year	Venue	Acronym	Paper	Project	Repo@GitHub
1988	SIGGRAPH		Terrain simulation using a model of stream erosion
1989	SIGGRAPH		The synthesis and rendering of eroded fractal terrains
1993	Graphics Interface		A fractal model of mountains and rivers
1998	SIGGRAPH		Realistic modeling and rendering of plant ecosystems
2001	SIGGRAPH	CityEngine	Procedural modeling of cities
2005	VRST		Modeling Landscapes with Ridges and Rivers
2006	TOG		Procedural modeling of buildings
2007	GDTW	Citygen	Citygen: An Interactive System for Procedural City Generation
2007	I3D		Example-based model synthesis
2007	TVCG		Terrain Synthesis from Digital Elevation Models
2008	CGF		Real-Time Rendering and Editing of Vector-based Terrains
2008	TOG		Continuous model synthesis
2008	TOG		Interactive Procedural Street Modeling
2009	CGF		Arches: a Framework for Modeling Complex Terrains
2009	CGF		Interactive Geometric Simulation of 4D Cities
2009	TOG		Interactive design of urban spaces using geometrical and behavioral modeling
2010	CGF		Procedural Generation of Roads
2011	CGF		Interactive Modeling of City Layouts using Layers of Procedural Content
2011	SI3D		Urban Ecosystem Design
2011	TOG		Metropolis procedural modeling
2012	CGF		Procedural Generation of Parcels in Urban Modeling
2012	TOG		Inverse design of urban procedural models
2013	TOG		Terrain Generation Using Procedural Models Based on Hydrology
2013	TOG	Urban Pattern	Urban Pattern: Layout Design by Hierarchical Domain Splitting
2015	TOG	WorldBrush	WorldBrush: Interactive Example-Based Synthesis of Procedural Virtual Worlds
2016	CGF		Example-Driven Procedural Urban Roads
2016	3DV		Proceduralization for Editing 3D Architectural Models
2016	TOG		Interactive Sketching of Urban Procedural Models
2017	TOG		Authoring landscapes by combining ecosystem and terrain erosion simulation
2017	TOG		Fast Weather Simulation for Inverse Procedural Design of 3D Urban Models
2017	TOG		Interactive Example-Based Terrain Authoring with Conditional Generative Adversarial Networks
2019	TOG		Synthetic Silviculture: Multi-scale Modeling of Plant Ecosystems
2021	TOG		Authoring Consistent Landscapes with Flora and Fauna
2022	TOG	Ecoclimates	Ecoclimates: Climate-Response Modeling of Vegetation
2022	TOG		Procedural Urban Forestry
2023	CVPR	Infinigen	Infinite Photorealistic Worlds using Procedural Generation
2023	TOG		Forming Terrains by Glacial Erosion
2023	TOG		Large-scale terrain authoring through interactive erosion simulation
2023	TOG		Authoring and Simulating Meandering Rivers
2025	CVPRW	Proc-GS	Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians
2025	CEUS	VoxCity	VoxCity: A seamless framework for open geospatial data integration, grid-based semantic 3D city model generation, and urban environment simulation

Optimization-based Generation

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2002	Graphics Interface		Constraint-based Automatic Placement for Scene Composition
2010	TOG		Computer-Generated Residential Building Layouts
2011	SIGGRAPH		Interactive Furniture Layout Using Interior Design Guidelines
2011	SIGGRAPH	Make it home	Make it home: automatic optimization of furniture arrangement
2012	TOG		Example-based synthesis of 3D object arrangements
2015	TVCG	Clutterpalette	The Clutterpalette: An Interactive Tool for Detailing Indoor Scenes
2018	CGF		MIQP-based Layout Design for Building Interiors
2018	CVPR		Human-centric Indoor Scene Synthesis Using Stochastic Grammar
2018	VR		Automatic Furniture Arrangement Using Greedy Cost Minimization
2021	MM	MageAdd	MageAdd: Real-Time Interaction Simulation for Scene Synthesis
2021	TVCG		Fast 3D Indoor Scene Synthesis by Learning Spatial Relation Priors of Objects
2021	arXiv	LUMINOUS	LUMINOUS: Indoor Scene Generation for Embodied AI Challenges
2022	NeurIPS	ProcTHOR	ProcTHOR: Large-Scale Embodied AI Using Procedural Generation
2024	CVPR	Infinigen Indoors	Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation

LLM-based Generation

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2023	NeurIPS	LayoutGPT	LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
2024	CVPR	GraphDreamer	GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs
2024	ECCV	AnyHome	AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes
2024	ECCV	SceneTeller	SceneTeller: Language-to-3D Scene Generation
2024	ICML	SceneCraft	SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code
2024	MM		Controllable Procedural Generation of Landscapes
2024	SIGGRAPH Asia	DIScene	DIScene: Object Decoupling and Interaction Modeling for Complex Scene Generation
2024	arXiv		Open-Universe Indoor Scene Generation using LLM Program Synthesis and Uncurated Object Databases
2024	arXiv	I-Design	I-Design: Personalized LLM Interior Designer
2024	arXiv	LLplace	LLplace: The 3D Indoor Scene Layout Generation and Editing via Large Language Model
2024	arXiv	CityCraft	CityCraft: A Real Crafter for 3D City Generation
2024	arXiv	CityX	CityX: Controllable Procedural Content Generation for Unbounded 3D Cities
2024	arXiv	GraphCanvas3D	Graph Canvas for Controllable 3D Scene Generation
2024	arXiv	UrbanWorld	UrbanWorld: An Urban World Model for 3D City Generation
2025	3DV	3D-GPT	3D-GPT: Procedural 3D Modeling with Large Language Models
2025	AAAI	SceneX	SceneX: Procedural Controllable Large-scale Scene Generation
2025	AAAI		Hierarchically-Structured Open-Vocabulary Indoor Scene Synthesis with Pre-trained Large Language Model
2025	CVPR		Global-Local Tree Search in VLMs for 3D Indoor Scene Generation
2025	CVPR	LayoutVLM	LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models
2025	CVPR	The Scene Language	The Scene Language: Representing Scenes with Programs, Words, and Embeddings
2025	ACL Findings	UnrealLLM	UnrealLLM: Towards Highly Controllable and Interactable 3D Scene Generation by LLM-powered Procedural Content Generation
2025	UIST	EchoLadder	EchoLadder: Progressive AI-Assisted Design of Immersive VR Scenes
2025	arXiv	WorldCraft	WorldCraft: Photo-Realistic 3D World Creation and Customization via LLM Agents
2025	arXiv	Cube	Cube: A Roblox View of 3D Intelligence
2025	arXiv	Scenethesis	Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation
2025	arXiv		Agentic 3D Scene Generation with Spatially Contextualized VLMs
2025	arXiv	RoomCraft	RoomCraft: Controllable and Complete 3D Indoor Scene Generation
2025	arXiv	ReSpace	ReSpace: Text-Driven 3D Scene Synthesis and Editing with Preference Alignment
2025	arXiv	DirectLayout	Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning
2025	arXiv	HLG	HLG: Comprehensive 3D Room Construction via Hierarchical Layout Generation
2025	arXiv	HOLODECK 2.0	HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editing
2025	arXiv	LatticeWorld	LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generation
2025	arXiv	CausalStruct	Causal Reasoning Elicits Controllable 3D Scene Generation
2025	arXiv	3D-Generalist	3D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worlds
2025	NeurIPS	SceneWeaver	SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent
2025	NeurIPS	MesaTask	MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning
2025	Nature Computational Science		Urban planning in the era of large language models
2025	arXiv	DisCo-Layout	DisCo-Layout: Disentangling and Coordinating Semantic and Physical Refinement in a Multi-Agent Framework for 3D Indoor Layout Synthesis
2025	arXiv	Reason-3D	Text-to-Scene with Large Reasoning Models
2025	arXiv	WorldGen	WorldGen: From Text to Traversable and Interactive 3D Worlds
2025	arXiv	MarketGen	MarketGen: A Scalable Simulation Platform with Auto-Generated Embodied Supermarket Environments
2025	arXiv	MajutsuCity	MajutsuCity: Language-driven Aesthetic-adaptive City Generation with Controllable 3D Assets and Layouts
2025	arXiv	Yo'City	Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansion
2025	arXiv	RAISECity	RAISECity: A Multimodal Agent Framework for Reality-Aligned 3D World Generation at City-Scale
2026	arXiv	SceneFoundry	SceneFoundry: Generating Interactive Infinite 3D Worlds
2026	AAAI	LandCraft	LandCraft: Designing the Structured 3D Landscapes via Text Guidance

Neural-3D Generation

Scene Parameters

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2018	SIGGRAPH	DeepSynth	Deep Convolutional Priors for Indoor Scene Synthesis
2019	CVPR	FastSynth	Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models
2020	SIGGRAPH		Deep Generative Modeling for Scene Synthesis via Hybrid Representations
2021	3DV	SceneFormer	SceneFormer: Indoor Scene Generation with Transformers
2021	ICCV	Sync2Gen	Scene Synthesis via Uncertainty-Driven Attribute Synchronization
2021	NeurIPS	ATISS	ATISS: Autoregressive Transformers for Indoor Scene Synthesis
2022	ECCV	Pose2Room	Pose2Room: Understanding 3D Scenes from Human Activities
2022	SIGGRAPH Asia	SUMMON	Scene Synthesis from Human Motion
2023	CVPR		Learning 3D Scene Priors with 2D Supervision
2023	CVPR	MIME	MIME: Human-Aware 3D Scene Generation
2023	SIGGRAPH	COFS	COFS: COntrollable Furniture layout Synthesis
2023	NeurIPS		Language-driven Scene Synthesis using Multi-conditional Diffusion Model
2024	3DV	RoomDesigner	RoomDesigner: Encoding Anchor-latents for Style-consistent and Shape-compatible Indoor Scene Generation
2024	CVPR	DiffuScene	DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis
2024	CVPR	SceneWiz3D	SceneWiz3D: Towards Text-guided 3D Scene Composition
2024	CVPR	PhyScene	PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI
2024	ECCV	DreamScene	DreamScene: 3D Gaussian-Based Text-to-3D Scene Generation via Formation Pattern Sampling
2024	ICML	GALA3D	GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting
2024	ICML		Disentangled 3D Scene Generation with Layout Learning
2024	MM	RelScene	RelScene: A Benchmark and baseline for Spatial Relations in text-driven 3D Scene Generation
2024	NeurIPS	DeBaRA	DeBaRA: Denoising-Based 3D Room Arrangement Generation
2024	SIGGRAPH	INFERACT	Physics-based Scene Layout Generation From Human Motion
2024	arXiv	Lay-A-Scene	Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priors
2025	3DV	Ctrl-Room	Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraints
2025	CVPR	SceneFactor	SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation
2025	CVPR	CASAGPT	CASAGPT: Cuboid Arrangement and Scene Assembly for Interior Design
2025	CoRL		Steerable Scene Generation with Post Training and Inference-Time Search
2025	NeurIPS	FactoredScenes	From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries

Scene Graph

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2014	EMNLP		Learning Spatial Knowledge for Text to 3D Scene Generation
2016	CGF		Learning 3D Scene Synthesis from Annotated RGB-D Images
2017	TOG		Adaptive synthesis of indoor scenes via activity-associated object relation graphs
2018	TOG		Language-Driven Synthesis of 3D Scenes from Scene Databases
2019	ICCV	Meta-Sim	Meta-Sim: Learning to Generate Synthetic Datasets
2019	SIGGRAPH	GRAINS	GRAINS: Generative Recursive Autoencoders for INdoor Scenes
2019	SIGGRAPH	PlanIT	PlanIT: Planning and Instantiating Indoor Scenes with Relation Graph and Spatial Prior Networks
2020	CVPR	3D-SLN	End-to-End Optimization of Scene Layout
2020	ECCV	Meta-Sim 2	Meta-Sim 2 Unsupervised Learning of Scene Structure for Synthetic Data Generation
2021	ICCV	Graph-to-3D	Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs
2023	NeurIPS	CommonScenes	CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion
2023	TPAMI	SceneHGN	SceneHGN: Hierarchical Graph Networks for 3D Indoor Scene Generation With Fine-Grained Geometry
2024	ECCV	SEK	External Knowledge Enhanced 3D Scene Generation from Sketch
2024	ECCV	Forest2Seq	Forest2Seq: Revitalizing Order Prior for Sequential Indoor Scene Synthesis
2024	ECCV	EchoScene	EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion
2024	ICLR	InstructScene	InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior
2025	AAAI	MMGDreamer	MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generation
2025	CVPR	FreeScene	FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Prompts
2025	ICCV		Controllable 3D Outdoor Scene Generation via Scene Graphs
2025	arXiv	HiScene	HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation
2025	arXiv	Imaginarium	Imaginarium: Vision-guided High-Quality 3D Scene Layout Generation

Semantic Layout

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2021	ICCV	SGSDI	Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images
2021	ICCV	GANcraft	GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds
2023	CVPR	DisCoScene	DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis
2023	ICCV	InfiniCity	InfiniCity: Infinite-Scale City Synthesis
2023	ICCV	CC3D	CC3D: Layout-Conditioned Generation of Compositional 3D Scenes
2023	ICCV	Set-the-Scene	Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenes
2023	ICCV	UrbanGIRAFFE	UrbanGIRAFFE: Representing Urban Scenes as Compositional Generative Neural Feature Fields
2023	TPAMI	SceneDreamer	SceneDreamer: Unbounded 3D Scene Generation From 2D Image Collections
2023	arXiv	CompoNeRF	CompoNeRF: Text-guided Multi-object Compositional NeRF with Editable 3D Scene Layout
2024	3DV	Comp3D	Compositional 3D Scene Generation using Locally Conditioned Diffusion
2024	CVPR	CityDreamer	CityDreamer: Compositional Generative Model of Unbounded 3D Cities
2024	CVPR	BerfScene	BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation
2024	NeurIPS	SceneCraft	SceneCraft: Layout-Guided 3D Scene Generation
2024	SIGGRAPH	BlockFusion	BlockFusion: Expandable 3D Scene Generation Using Latent Tri-plane Extrapolation
2024	SIGGRAPH Asia	Frankenstein	Frankenstein: Generating Semantic-Compositional 3D Scenes in One Tri-Plane
2024	arXiv	Urban Architect	Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior
2025	CVPR	GaussianCity	Generative Gaussian Splatting for Unbounded 3D City Generation
2025	ICLR	Layout-your-3D	Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint
2025	arXiv	Layout2Scene	Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors
2025	arXiv	PrITTI	PrITTI: Primitive-based Generation of Controllable and Editable 3D Semantic Scenes
2025	TPAMI	CityDreamer4D	CityDreamer4D: Compositional Generative Model of Unbounded 4D Cities
2025	TPAMI	UrbanGen	UrbanGen: Urban Generation with Compositional and Controllable Neural Fields
2025	ICCV	Sat2City	Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusion
2025	arXiv	X-Scene	X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability
2025	arXiv	EarthCrafter	EarthCrafter: Scalable 3D Earth Generation via Dual-Sparse Latent Diffusion
2025	arXiv	SPATIALGEN	SPATIALGEN: Layout-guided 3D Indoor Scene Generation

Implicit Layout

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2021	CVPR	GIRAFFE	GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields
2021	ICCV	GSN	Unconstrained Scene Generation With Locally Conditioned Radiance Fields
2021	ICML	NeRF-VAE	NeRF-VAE: A geometry aware 3d scene generative model
2022	NeurIPS	GAUDI	GAUDI: A Neural Architect for Immersive 3D Scene Generation
2023	CVPR	Persistent Nature	Persistent Nature: A generative model of unbounded 3D worlds
2023	CVPR	NeuralField-LDM	NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models
2023	arXiv		Diffusion Probabilistic Models for Scene-Scale 3D Categorical Data
2024	CVPR	DiffInDScene	DiffInDScene: Diffusion-based High-Quality 3D Indoor Scene Generation
2024	CVPR	XCube	XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies
2024	CVPR	SemCity	SemCity: Semantic Scene Generation with Triplane Diffusion
2024	ECCV	PDD	Pyramid Diffusion for Fine 3D Large Scene Generation
2024	NeurIPS	Director3D	Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text
2025	CVPR	LT3SD	LT3SD: Latent Trees for 3D Scene Diffusion
2025	CVPR	SplatFlow	SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesis
2025	CVPR	Prometheus	Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation
2025	ICLR	DynamicCity	DynamicCity: Large-Scale Occupancy Generation from Dynamic Scenes
2025	ICCV	NuiScene	NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenes
2025	arXiv	VideoRFSplat	VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting Generation with Flexible Pose and Multi-View Joint Modeling
2025	arXiv	LSD-3D	LSD-3D: Large-Scale 3D Driving Scene Generation with Geometry Grounding
2025	arXiv	UniUGG	UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
2025	arXiv	FlashWorld	FlashWorld: High-quality 3D Scene Generation within Seconds
2025	arXiv	Diff4Splat	Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models
2025	arXiv	Terra	Terra: Explorable Native 3D World Model with Point Latents
2025	arXiv	TRELLISWorld	TRELLISWorld: Training-Free World Generation from Object Generators
2026	AAAI	WorldGrow	WorldGrow: Generating Infinite 3D World

Image-based Generation

Holistic Generation

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2019	ICIP		360-Degree Image Completion by Two-Stage Conditional Gans
2020	CVPR	Sat2Ground	Geometry-Aware Satellite-to-Ground Image Synthesis for Urban Areas
2020	WACV		360 Panorama Synthesis from a Sparse Set of Images with Unknown Field of View
2021	AAAI	SIG-SS	Spherical Image Generation from a Single Image by Considering Scene Symmetry
2021	CVPR	EnvMapNet	HDR Environment Map Estimation for Real-Time Augmented Reality
2021	ICCV	Sat2vid	Sat2vid: Street-view panoramic video synthesis from a single satellite image
2022	3DV	ImmerseGAN	Guided Co-Modulated GAN for 360° Field of View Extrapolation
2022	CVPR	OmniDreamer	Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation
2022	ECCV	BIPS	BIPS: Bi-modal Indoor Panorama Synthesis via Residual Depth-aided Adversarial Learning
2022	SIGGRAPH Asia	Text2Light	Text2Light: Zero-Shot Text-Driven HDR Panorama Generation
2022	TMM	PanoGAN	Cross-View Panorama Image Synthesis
2022	TPAMI	Sat2Str	Geometry-Guided Street-View Panorama Synthesis from Satellite Imagery
2023	CVPR	DiffCollage	DiffCollage: Parallel Generation of Large Content with Diffusion Models
2023	ICCV	Sat2Density	Sat2Density: Faithful Density Learning from Satellite-Ground Image Pairs
2023	MM	PanoDiff	360-Degree Panorama Generation from Few Unregistered NFoV Images
2023	NeurIPS	MVDiffusion	MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion
2023	TPAMI		Spherical Image Generation From a Few Normal-Field-of-View Images by Considering Scene Symmetry
2023	arXiv	LDM3D	LDM3D: Latent Diffusion Model for 3D
2023	arXiv	Diffusion360	Diffusion360: Seamless 360 Degree Panoramic Image Generation based on Diffusion Models
2024	ICLR	PanoDiffusion	PanoDiffusion: 360-degree Panorama Outpainting via Diffusion
2024	CVPR	ControlRoom3D	ControlRoom3D 🤖Room Generation using Semantic Proxy Rooms
2024	CVPR	Sat2Scene	Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion
2024	CVPR	PanFusion	Taming stable diffusion for text to 360◦ panorama image generation
2024	ECCV	DreamScene360	DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting
2024	ECCV		Geospecific View Generation - Geometry-Context Aware High-resolution Ground View Inference from Satellite Views
2024	IJCAI	FastScene	FastScene: Text-Driven Fast Indoor 3D Scene Generation via Panoramic Gaussian Splatting
2024	NeurIPS	DiffPano	DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion
2024	TPAMI	PERF	PERF: Panoramic Neural Radiance Field from a Single Panorama
2024	TVCG	Dream360	Dream360: Diverse and Immersive Outdoor Virtual Scene Creation via Transformer-Based 360° Image Outpainting
2024	WACV	StitchDiffusion	Customizing 360-Degree Panoramas through Text-to-Image Diffusion Models
2024	arXiv	HoloDreamer	HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions
2024	arXiv	SceneDreamer360	SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting
2025	ICLR	CubeDiff	CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation
2025	SIGGRAPH	LayerPano3D	LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation
2025	arXiv	EmbodiedGen	EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligence
2025	arXiv	ImmerseGen	ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxies
2025	arXiv	HunyuanWorld 1.0	HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixels
2025	arXiv	Matrix-3D	Matrix-3D: Omnidirectional Explorable 3D World Generation
2025	ICCV		A Recipe for Generating 3D Worlds From a Single Image
2025	ICCV	DreamCube	DreamCube: 3D Panorama Generation via Multi-plane Synchronization
2025	arXiv	OmniX	OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes

Iterative Generation

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2019	TOG		3D Ken Burns Effect from a Single Image
2020	CVPR	SynSin	SynSin: End-to-end view synthesis from a single image
2020	CVPR	3D Photo	3D Photography Using Context-Aware Layered Depth Inpainting
2020	CVPR		Single-View View Synthesis with Multiplane Images
2020	NeurIPS	GVS	Generative View Synthesis: From Single-view Semantics to Novel-view Images
2021	ICCV	Worldsheet	Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image
2021	ICCV	InfiniteNature	Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image
2021	ICCV	GFVS	Geometry-free view synthesis: Transformers and no 3d priors
2021	ICCV	Pathdreamer	Pathdreamer: A World Model for Indoor Navigation
2021	ICCV	PixelSynth	PixelSynth: Generating a 3D-Consistent Experience from a Single Image
2022	CVPR	LOTR	Look outside the room: Synthesizing a consistent long-term 3d scene video from a single image
2022	ECCV	InfiniteNature-Zero	InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images
2022	NeurIPS	SGAM	SGAM: Building a Virtual 3D World through Simultaneous Generation and Mapping
2023	AAAI	SE3DS	Simple and Effective Synthesis of Indoor 3D Scenes
2023	CVPR	3D Cinemagraphy	3D Cinemagraphy from a Single Image
2023	CVPR		Consistent View Synthesis with Pose-Guided Diffusion Models
2023	ICCV	DiffDreamer	DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Models
2023	ICCV	Text2Room	Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models
2023	ICCV		Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models
2023	MM	Make-It-4D	Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Image
2023	NeurIPS	SceneScape	SceneScape: Text-Driven Consistent Scene Generation
2023	NeurIPS	PanoGen	PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation
2023	arXiv	Text2Immersion	Text2Immersion: Generative Immersive Scene with 3D Gaussians
2024	AAAI	AOG-Net	Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation
2024	CVPR	WonderJourney	WonderJourney: Going from Anywhere to Everywhere
2024	CVPR	3D-SceneDreamer	3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation
2024	ECCV	PanoFree	PanoFree: Tuning-Free Holistic Multi-view Image Generation with Cross-view Self-Guidance
2024	MM	iControl3D	iControl3D: An Interactive System for Controllable 3D Scene Generation
2024	NeurIPS	ODIN	From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos
2024	NeurIPS	CAT3D	CAT3D: Create Anything in 3D with Multi-View Diffusion Models
2024	TVCG	Text2NeRF	Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields
2024	arXiv	OPa-Ma	OPa-Ma: Text Guided Mamba for 360-degree Image Out-painting
2024	arXiv	Scene123	Scene123: One Prompt to 3D Scene Generation via Video-Assisted and Consistency-Enhanced MAE
2025	TVCG	LucidDreamer	LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes
2025	3DV	RealmDreamer	RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion
2025	3DV	Invisible Stitch	Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting
2025	AAAI	BloomScene	BloomScene: Lightweight Structured 3D Gaussian Splatting for Crossmodal Scene Generation
2025	CVPR	WonderWorld	WonderWorld: Interactive 3D Scene Generation from a Single Image
2025	CVPR	ArtiScene	ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary
2025	ICLR	3D-MOM	Optimizing 4D Gaussians for Dynamic Scene Video from Single Landscape Images
2025	arXiv	WonderTurbo	WonderTurbo: Generating Interactive 3D World in 0.72 Seconds
2025	ICCV	Bolt3D	Bolt3D: Generating 3D Scenes in Seconds
2025	arXiv	SynCity	SynCity: Training-Free Generation of 3D Worlds
2025	arXiv	MeSS	MeSS: City Mesh-Guided Outdoor Scene Generation with Cross-View Consistent Diffusion
2025	arXiv	CausNVS	CausNVS: Autoregressive Multi-view Diffusion for Flexible 3D Novel View Synthesis
2025	arXiv	WonderZoom	WonderZoom: Multi-Scale 3D World Generation
2025	ICCV	ScenePainter	ScenePainter: Semantically Consistent Perpetual 3D Scene Generation with Concept Relation Alignment

Video-based Generation

Two-stage Generation

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2024	SIGGRAPH	Streetscapes	Streetscapes Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion
2024	NeurIPS	4Real	4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models
2024	arXiv	VividDream	VividDream: Generating 3D Scene with Ambient Dynamics
2024	arXiv	PaintScene4D	PaintScene4D: Consistent 4D Scene Generation from Text Prompts
2025	ICLR	GenXD	GenXD: Generating Any 3D and 4D Scenes
2025	CVPR	StarGen	StarGen: A Spatiotemporal Autoregression Framework with Video Diffusion Model for Scalable and Controllable Scene Generation
2025	TMM	DreamJourney	DreamJourney: Perpetual View Generation with Video Diffusion Models
2025	ICCV	DimensionX	DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion
2025	ICCV	Free4D	Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency

One-stage Generation

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2023	arXiv	GAIA-1	GAIA-1: A Generative World Model for Autonomous Driving
2023	arXiv	ADriver-I	ADriver-I: A General World Model for Autonomous Driving
2024	ICLR	MagicDrive	MagicDrive: Street View Generation with Diverse 3D Geometry Control
2024	CVPR	Panacea	Panacea: Panoramic and Controllable Video Generation for Autonomous Driving
2024	CVPR	Drive-WM	Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
2024	CVPR	360DVD	360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model
2024	ECCV	DriveDreamer	DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
2024	ECCV	DrivingDiffusion	DrivingDiffusion: Layout-Guided Multi-View Driving Scenarios Video Generation with Latent Diffusion Model
2024	ECCV	WoVoGen	WoVoGen: World Volume-Aware Diffusion for Controllable Multi-camera Driving Scene Generation
2024	NeurIPS	Vista	Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
2024	NeurIPS	DIAMOND	Diffusion for World Modeling: Visual Details Matter in Atari
2024	arXiv	MagicDrive3D	MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes
2024	arXiv	Delphi	Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation
2024	arXiv	BEVWorld	BEVWorld: A Multimodal World Model for Autonomous Driving via Unified BEV Latent Space
2024	arXiv	DriveArena	DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving
2024	arXiv	DiVE	DiVE: DiT-based Video Generation with Enhanced Control
2024	arXiv	DreamForge	DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes
2024	arXiv	SyntheOcc	SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs
2024	arXiv	HoloDrive	HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving
2024	arXiv	CogDriving	Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attention
2024	arXiv	Imagine360	Imagine360: Immersive 360 Video Generation from Perspective Anchor
2024	arXiv	DrivingWorld	DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
2024	arXiv	ViewCrafter	ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis
2024	arXiv	ViewExtrapolator	Novel View Extrapolation with Video Diffusion Priors
2025	AAAI	DriveDreamer-2	DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generation
2025	ICLR	4K4DGen	4K4DGen: Panoramic 4D Generation at 4K Resolution
2025	ICLR	GameGen-X	GameGen-X: Interactive Open-world Game Video Generation
2025	ICLR	GameNGen	Diffusion Models Are Real-Time Game Engines
2025	ICLR	Genex	Generative World Explorer
2025	ICLR	GLAD	Glad: A Streaming Scene Generator for Autonomous Driving
2025	CVPR	DrivingSphere	DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation
2025	CVPR	StreetCrafter	StreetCrafter: Street View Synthesiswith Controllable Video Diffusion Models
2025	CVPR	DriveScape	DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation
2025	CVPR	UniScene	UniScene: Unified Occupancy-centric Driving Scene Generation
2025	CVPR	GEM	GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control
2025	CVPR	UMGen	Generating Multimodal Driving Scenes via Next-Scene Prediction
2025	CVPR	CAT4D	CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models
2025	CVPR	Wonderland	Wonderland: Navigating 3D Scenes from a Single Image
2025	CVPR	VideoScene	VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step
2025	CVPR	Scene Splatter	Scene Splatter: Momentum 3D Scene Generation from Single Image with Video Diffusion Model
2025	CVPR	DynamicScaler	DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenes
2025	ICML	AdaWorld	AdaWorld: Learning Adaptable World Models with Latent Actions
2025	Nature	WHAM	World and Human Action Models towards gameplay ideation
2025	arXiv	DreamDrive	DreamDrive: Generative 4D Scene Modeling from Street View Images
2025	arXiv	MaskGWM	MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction
2025	arXiv	UniFuture	Seeing the Future, Perceiving the Future: A Unified Driving World Model for Future Generation and Perception
2025	arXiv	SimWorld	SimWorld: A Unified Benchmark for Simulator-Conditioned Scene Generation via World Model
2025	arXiv	DiST-4D	DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation
2025	arXiv	GAIA-2	GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving
2025	arXiv	SteerX	SteerX: Creating Any Camera-Free 3D and 4D Scenes with Geometric Steering
2025	arXiv	WonderVerse	WonderVerse: Extendable 3D Scene Generation with Video Generative Models
2025	arXiv	FlexWorld	FlexWorld: Progressively Expanding 3D Scenes for Flexiable-View Synthesis
2025	arXiv	GaussVideoDreamer	GaussVideoDreamer: 3D Scene Generation with Video Diffusion and Inconsistency-Aware Gaussian Splatting
2025	arXiv	WORLDMEM	WORLDMEM: Long-term Consistent World Simulation with Memory
2025	arXiv	HoloTime	HoloTime: Taming Video Diffusion Models for Panoramic 4D Scene Generation
2025	arXiv	MineWorld	MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft
2025	arXiv	GameFactory	GameFactory: Creating New Games with Generative Interactive Videos
2025	arXiv	CoGen	CoGen: 3D Consistent Video Generation via Adaptive Conditioning for Autonomous Driving
2025	arXiv	Dreamland	Dreamland: Controllable World Creation with Simulator and Generative Models
2025	arXiv	Voyager	Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation
2025	arXiv	Matrix-Game	Matrix-Game: Interactive World Foundation Model
2025	arXiv	Matrix-Game 2.0	Matrix-Game 2.0: An Open-Source, Real-Time, and Streaming Interactive World Model
2025	arXiv	Hunyuan-GameCraft	Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition
2025	arXiv	CoCo4D	CoCo4D: Comprehensive and Complex 4D Scene Generation
2025	arXiv	WonderFree	WonderFree: Enhancing Novel View Quality and Cross-View Consistency for 3D Scene Exploration
2025	arXiv	4DVD	4DVD: Cascaded Dense-view Video Diffusion Model for High-quality 4D Content Generation
2025	arXiv	IDCNet	IDCNet: Guided Video Diffusion for Metric-Consistent RGBD Scene Generation with Precise Camera Control
2025	arXiv	4DNeX	4DNeX: Feed-Forward 4D Generative Modeling Made Easy
2025	ICCV	WonderPlay	WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions
2025	ICCV	MagicDrive-V2	MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control
2025	ICCV	DynamicVoyager	Voyaging into Unbounded Dynamic Scenes from a Single View
2025	ICCV	InfiniCube	InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Models
2025	ICCV	VMem	VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory
2025	SIGGRAPH Asia	VideoFrom3D	VideoFrom3D: 3D Scene Video Generation via Complementary Image and Video Diffusion Models
2025	SIGGRAPH Asia	WorldExplorer	WorldExplorer: Towards Generating Fully Navigable 3D Scenes
2025	arXiv	WorldForge	WorldForge: Unlocking Emergent 3D/4D Generation in Video Diffusion Model via Training-Free Guidance
2025	arXiv		From Virtual Games to Real-World Play
2025	arXiv	FantasyWorld	FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction
2025	arXiv	EvoWorld	EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memory
2025	arXiv	Captain Safari	Captain Safari: A World Engine
2025	arXiv	MagicWorld	MagicWorld: Interactive Geometry-driven Video World Exploration
2025	arXiv	One4D	One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control

Datasets

Indoor Datasets

Year	Type	Source	Acronym	Paper	Project
2012	Indoor, Nature	Real	SUN360	Recognizing scene viewpoint using panoramic place representation
2012	Indoor	Real	NYUv2	Indoor Segmentation and Support Inference From RGBD Images
2015	Indoor	Real	SunRGBD	Sun RGB-D: A RGB-D scene understanding benchmark suite
2016	Indoor	Real	SceneNN	SceneNN: A Scene Meshes Dataset with aNNotations
2017	Indoor	Real	2D-3D-S	Joint 2D-3D-Semantic Data for Indoor Scene Understanding
2017	Indoor	Real	Matterport3D	Matterport3D: Learning from RGB-D Data in Indoor Environments
2017	Indoor	Real	ScanNet	ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
2017	Indoor	Real	Laval Indoor	Learning to Predict Indoor Illumination from a Single Image
2018	Indoor, Urban	Real	RealEstate10K	Stereo Magnification: Learning View Synthesis using Multiplane Images
2019	Indoor	Real	Replica	The Replica Dataset: A Digital Replica of Indoor Spaces
2020	Indoor	Real	3DSSG	Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions
2021	Indoor	Real	HM3D	Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
2023	Indoor	Real	ScanNet++	ScanNet++: A high-fidelity dataset of 3D indoor scenes
2023	Indoor, Nature, Urban	Real	DL3DV-10K	DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision
2012	Indoor	Synthetic	SceneSynth	Example-based synthesis of 3D object arrangements
2017	Indoor	Synthetic	SUNCG	Semantic Scene Completion from a Single Depth Image
2020	Indoor	Synthetic	Structured3D	Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling
2020	Indoor	Synthetic	HyperSim	HyperSim: A photorealistic synthetic dataset for holistic indoor scene understanding
2021	Indoor	Synthetic	3D-FRONT	3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics
2021	Indoor	Synthetic	3D-Future	3D-FUTURE: 3D Furniture shape with TextURE
2023	Indoor	Synthetic	SG-FRONT	CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion
2025	Indoor	Synthetic	SE(3) Scene	Steerable Scene Generation with Post Training and Inference-Time Search
2025	Indoor	Synthetic	SYNBUILD-3D	SYNBUILD-3D: A large, multi-modal, and semantically rich synthetic dataset of 3D building models at Level of Detail 4
2025	Indoor	Synthetic	InternScenes	InternScenes: A Large-scale Simulatable Indoor Scene Dataset with Realistic Layouts
2025	Indoor	Synthetic	SPATIALGEN	SPATIALGEN: Layout-guided 3D Indoor Scene Generation
2025	Indoor	Synthetic	MesaTask	MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning

Natural Datasets

Year	Type	Source	Acronym	Paper	Project
2017	Nature	Real	Laval Outdoor	Deep Sky Modeling for Single Image Outdoor Lighting Estimation
2019	Nature	Real	LHQ	Aligning latent and image spaces to connect the unconnectable
2021	Nature	Real	ACID	Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image

Urban Datasets

Year	Type	Source	Acronym	Paper	Project
2012	Urban	Real	KITTI	Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite
2016	Urban	Real	Cityscapes	The Cityscapes dataset for semantic urban scene understanding
2019	Urban	Real	SemanticKITTI	SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences
2020	Urban	Real	Waymo	Scalability in Perception for Autonomous Driving: Waymo Open Dataset
2020	Urban	Real	nuScenes	nuScenes: A multimodal dataset for autonomous driving
2023	Urban	Real	KITTI-360	KITTI-360: A novel dataset and benchmarks for urban scene understanding in 2D and 3D.
2020	Urban	Real	HoliCity	HoliCity: A city-scale data platform for learning holistic 3D structures
2022	Urban	Real	OmniCity	OmniCity: Omnipotent City Understanding with Multi-level and Multi-view Images
2024	Urban	Real	OSM	CityDreamer: Compositional Generative Model of Unbounded 3D Cities
2024	Urban	Real	GoogleEarth	CityDreamer: Compositional Generative Model of Unbounded 3D Cities
2017	Urban	Synthetic	CARLA	CARLA: An Open Urban Driving Simulator
2022	Urban	Synthetic	CarlaSC	MotionSC: Data Set and Network for Real-Time Semantic Mapping in Dynamic Environments
2020	Urban	Synthetic	Virtual-KITTI-2	Virtual KITTI 2
2025	Urban	Synthetic	CityTopia	CityDreamer4D: Compositional Generative Model of Unbounded 4D Cities

Applications and Tasks

3D Scene Editing

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2022	CVPR	StyleMesh	StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions
2023	CVPR	DisCoScene	DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis
2023	CVPR	LEGO-Net	LEGO-Net: Learning Regular Rearrangements of Objects in Rooms
2023	CVPR	Lift3D	Lift3D: Synthesize 3D Training Data by Lifting 2D GAN to 3D Generative Radiance Field
2023	CVPR	Text2Scene	Text2Scene: Text-driven Indoor Scene Stylization with Part-aware Details
2023	ICRA	CabiNet	CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generation
2023	MM	RoomDreamer	RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texture
2024	CVPR	SceneTex	SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priors
2024	CVPR	ControlRoom3D	ControlRoom3D 🤖Room Generation using Semantic Proxy Rooms
2024	ECCV	StyleCity	StyleCity: Large-Scale 3D Urban Scenes Stylization
2024	ECCV	RoomTex	RoomTex: Texturing Compositional Indoor Scenes via Iterative Inpainting
2024	ECCV	3D-GOI	3D-GOI: 3D GAN Omni-Inversion for Multifaceted and Multi-object Editing
2024	MM	SceneExpander	SceneExpander: Real-Time Scene Synthesis for Interactive Floor Plan Editing
2024	NeurIPS	Neural Assets	Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models
2024	NeurIPS	DeBaRA	DeBaRA: Denoising-Based 3D Room Arrangement Generation
2024	SIGGRAPH Asia	InstanceTex	InstanceTex: Instance-level Controllable Texture Synthesis for 3D Scenes via Diffusion Priors
2024	TVCG	SceneDirector	SceneDirector: Interactive Scene Synthesis by Simultaneously Editing Multiple Objects in Real-Time
2024	VR	DreamSpace	DreamSpace: Dreaming Your Room Space with Text-Driven Panoramic Texture Propagation
2025	3DV	Ctrl-Room	Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraints
2025	CVPR	RoomPainter	RoomPainter: View-Integrated Diffusion for Consistent Indoor Scene Texturing
2025	SIGGRAPH	ReStyle3D	ReStyle3D: Scene-Level Appearance Transfer with Semantic Correspondences
2025	NeurIPS	Styl3R	Styl3R: Instant 3D Stylized Reconstruction for Arbitrary Scenes and Styles

Human-Scene Interaction

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2022	CVPR		Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis
2022	ECCV	COINS	Compositional Human-Scene Interaction Synthesis with Semantic Control
2023	CVPR	SceneDiffuser	Diffusion-based Generation, Optimization, and Planning in 3D Scenes
2023	ICCV	DIMOS	Synthesizing Diverse Human Motions in 3D Indoor Scenes
2023	SIGGRAPH	InterPhys	Synthesizing Physical Character-Scene Interactions
2024	3DV	InterScene	Synthesizing Physically Plausible Human Motions in 3D Scenes
2024	CVPR	GenZI	GenZI: Zero-Shot 3D Human-Scene Interaction Generation
2024	ICLR	UniHSI	UniHSI: Unified Human-Scene Interaction via Prompted Chain-of-Contacts
2025	ICCV	SIMS	SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generation
2025	CVPR	TokenHSI	TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization
2025	arXiv	FantasyHSI	FantasyHSI: Video-Generation-Centric 4D Human Synthesis In Any Scene through A Graph-based Multi-Agent Framework

Embodied AI

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2022	NeurIPS	ProcTHOR	ProcTHOR: Large-Scale Embodied AI Using Procedural Generation
2024	CVPR	Holodeck	Holodeck: Language Guided Generation of 3D Embodied AI Environments
2024	CVPR	PhyScene	PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI
2024	NeurIPS	Architect	Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting
2024	arXiv	GRUtopia	GRUtopia: Dream General Robots in a City at Scale
2024	arXiv	EmbodiedCity	EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment
2024	arXiv	InfiniteWorld	InfiniteWorld: A Unified Scalable Simulation Framework for General Visual-Language Robot Interaction
2025	ICLR	MetaUrban	MetaUrban: An Embodied AI Simulation Platform for Urban Micromobility
2025	arXiv	LEGO-Eval	LEGO-Eval: Towards Fine-grained Evaluation on Synthesizing 3D Embodied Environments with Tool Augmentation
2025	arXiv	MarketGen	MarketGen: A Scalable Simulation Platform with Auto-Generated Embodied Supermarket Environments
2025	arXiv	TabletopGen	TabletopGen: Instance-Level Interactive 3D Tabletop Scene Generation from Text or Single Image

Robotics

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2023	NeurIPS	UniPi	Learning Universal Policies via Text-Guided Video Generation
2023	NeurIPS	HiP	Compositional Foundation Models for Hierarchical Planning
2024	CoRL	Imagination Policy	Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policies
2024	CoRL	Eurekaverse	Eurekaverse: Environment Curriculum Generation via Large Language Models
2024	ICLR	GR-1	Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
2024	ICML	RoboGen	RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation
2024	ICML	VLP	Using Left and Right Brains Together: Towards Vision and Language Planning
2024	IROS	ActNeRF	Uncertainty-aware Active Learning of NeRF-based Object Models for Robot Manipulators using Visual and Re-orientation Actions
2024	NeurIPS	CLOVER	Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation
2024	arXiv	GR-2	GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation
2025	ICLR	SlowFast-VGen	SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation
2025	ICML	Video Prediction Policy	Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
2025	arXiv	VideoWorld	VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
2025	arXiv	Cosmos-Transfer1	Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control
2025	arXiv	TesserAct	TesserAct: Learning 4D Embodied World Models

Autonomous Driving

Year	Venue	Acronym	Paper	Project	Repo@GitHub
2023	arXiv	GAIA-1	GAIA-1: A Generative World Model for Autonomous Driving
2023	arXiv	Cam4DOcc	Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications
2024	CVPR	Drive-WM	Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving
2024	ECCV	DriveDreamer	DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving
2024	ECCV	OccWorld	OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving
2024	ECCV	WoVoGen	WoVoGen: World Volume-Aware Diffusion for Controllable Multi-camera Driving Scene Generation
2024	ICLR	MagicDrive	MagicDrive: Street View Generation with Diverse 3D Geometry Control
2024	NeurIPS	Vista	Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability
2024	arXiv	OccSora	OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving
2024	arXiv	Delphi	Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generation
2024	arXiv	DriveArena	DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving
2024	arXiv	DiVE	DiVE: DiT-based Video Generation with Enhanced Control
2024	arXiv	DreamForge	DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes
2024	arXiv	DrivingWorld	DrivingWorld: Constructing World Model for Autonomous Driving via Video GPT
2025	AAAI	Drive-OccWorld	Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving
2025	CVPR	DrivingSphere	DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation
2025	ICLR	GLAD	Glad: A Streaming Scene Generator for Autonomous Driving
2025	arXiv	DreamDrive	DreamDrive: Generative 4D Scene Modeling from Street View Images
2025	arXiv	Cosmos-Transfer1	Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control