-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
271 lines (264 loc) · 17.7 KB
/
index.html
File metadata and controls
271 lines (264 loc) · 17.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="description" content="Project page for Portals: Persistent, Editable 4D Spatial World Models on Edge Devices.">
<meta property="og:title" content="Portals: Persistent, Editable 4D Spatial World Models on Edge Devices">
<meta property="og:description" content="CVPR 2026 4D World Models workshop paper on deployed, persistent, editable 4D worlds for edge devices.">
<meta property="og:url" content="https://imclab.github.io/portals-cvpr2026/">
<meta property="og:image" content="static/images/portals-banner.png">
<meta property="og:image:width" content="1414">
<meta property="og:image:height" content="760">
<meta name="twitter:title" content="Portals: Persistent, Editable 4D Spatial World Models on Edge Devices">
<meta name="twitter:description" content="CVPR 2026 workshop paper on deployed persistent 4D spatial worlds.">
<meta name="twitter:image" content="static/images/portals-banner.png">
<meta name="twitter:card" content="summary_large_image">
<meta name="keywords" content="4D world models, Gaussian splatting, augmented reality, edge devices, persistent spatial computing">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Portals: Persistent, Editable 4D Spatial World Models on Edge Devices</title>
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet">
<link rel="stylesheet" href="static/css/bulma.min.css">
<link rel="stylesheet" href="static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="static/css/bulma-slider.min.css">
<link rel="stylesheet" href="static/css/fontawesome.all.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="static/css/index.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script defer src="static/js/fontawesome.all.min.js"></script>
<script src="static/js/bulma-carousel.min.js"></script>
<script src="static/js/bulma-slider.min.js"></script>
<script src="static/js/index.js"></script>
</head>
<body>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">Portals: Persistent, Editable 4D Spatial World Models on Edge Devices</h1>
<div class="is-size-5 publication-authors">
<span class="author-block">James A. Tunick<sup>1,2</sup>,</span>
<span class="author-block">Ryan Brant<sup>1</sup>,</span>
<span class="author-block">Jacob D. Pennock<sup>1</sup>,</span>
<span class="author-block">Justin Kasowski<sup>1</sup></span>
</div>
<div class="is-size-5 publication-authors">
<span class="author-block"><sup>1</sup>H3M, Inc.</span>
<span class="author-block"><sup>2</sup>The IMC Lab, New York</span>
</div>
<div class="is-size-5 publication-authors">
<span class="author-block">CVPR 2026 Workshop on 4D World Models: Bridging Generation and Reconstruction</span>
</div>
<div class="column has-text-centered">
<div class="publication-links">
<span class="link-block">
<a href="static/pdfs/portals_cvpr2026_workshop_paper.pdf" target="_blank" class="external-link button is-normal is-rounded is-dark">
<span class="icon"><i class="fas fa-file-pdf"></i></span>
<span>Paper</span>
</a>
</span>
<span class="link-block">
<a href="mailto:JTunick@TheIMCLab.com" class="external-link button is-normal is-rounded is-dark">
<span class="icon"><i class="fas fa-envelope"></i></span>
<span>Contact</span>
</a>
</span>
<span class="link-block">
<span class="external-link button is-normal is-rounded is-dark is-disabled">
<span class="icon"><i class="fas fa-globe"></i></span>
<span>Web Demo Coming Soon</span>
</span>
</span>
<span class="link-block">
<span class="external-link button is-normal is-rounded is-dark is-disabled">
<span class="icon"><i class="fab fa-github"></i></span>
<span>Code Coming Soon</span>
</span>
</span>
<span class="link-block">
<span class="external-link button is-normal is-rounded is-dark is-disabled">
<span class="icon"><i class="fas fa-video"></i></span>
<span>Video Coming Soon</span>
</span>
</span>
</div>
<div class="project-meta-links">
<a href="https://h3m.ai" target="_blank" rel="noopener noreferrer">H3M.ai</a>
<span class="meta-divider">•</span>
<a href="https://www.theimclab.com" target="_blank" rel="noopener noreferrer">The IMC Lab</a>
<span class="meta-divider">•</span>
<a href="https://github.com/imclab" target="_blank" rel="noopener noreferrer">imclab GitHub</a>
<span class="meta-divider">•</span>
<a href="https://github.com/JT5D" target="_blank" rel="noopener noreferrer">JT5D GitHub</a>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="hero teaser">
<div class="container is-max-desktop">
<div class="hero-body">
<img src="static/images/portals-banner.png" alt="Portals system architecture overview" class="teaser-image">
<h2 class="subtitle has-text-centered">
Persistent, editable 4D spatial world models for phones, smart glasses, and mixed-reality headsets, built with edge-device rendering constraints as a first-class requirement.
</h2>
</div>
</div>
</section>
<section class="section hero is-light">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
We present <strong>Portals</strong>, a deployed systems architecture that bridges 4D world-model research and persistent spatial experiences on phones, smart glasses, and augmented-reality headsets. We argue that the central systems problem in this spatial-computing shift is not only scene representation quality, but whether dynamic spatial worlds can run in real time on constrained hardware, persist across revisits, and remain editable by non-expert creators. Built on 3D Gaussian Splatting, 4D-GS, and Generalizable Human Gaussians, Portals has been deployed across mobile AR, Apple Vision Pro, and web clients for reconstructed environments, volumetric humans, and holographic spatial media. We contribute: (1) an edge-device runtime built around LOD-adaptive Gaussian Splatting and a shared spatial-media compute substrate that fuses depth, stencil, audio, and ML-pose channels through a single dispatch, driving 370+ source-agnostic VFX effects at 60 fps on iPhone 14 Pro (2.7-4.1x speedup); (2) a persistent geospatial scene-state architecture with layered world metadata, reloadable scene payloads, and anchor-guided re-alignment across sessions; (3) a creator-facing composition pipeline that bridges reconstruction and generation through VFX composition, voice-driven semantic actions, and no-code authoring; and (4) benchmark axes for evaluating 4D world models under deployment constraints such as mobile rendering efficiency, scene persistence, and editable world state. Prior clinical deployment of volumetric AR at Memorial Sloan Kettering established the real-time rendering primitives underlying this work.
</p>
</div>
</div>
</div>
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column is-four-fifths">
<h2 class="title is-3 has-text-centered">Visual Highlights</h2>
<div class="visual-highlights">
<figure class="visual-hero">
<img src="static/images/runtime-real-app-grid-v3.jpg" alt="Real in-app runtime captures from Portals on iPhone" class="paper-figure">
<figcaption class="figure-caption">Real runtime stills spanning authoring, environment VFX, holograms, and responsive effects driven by body segmentation, hand input, audio-reactive behaviors, and face-aware overlays.</figcaption>
</figure>
<div class="visual-grid visual-grid-secondary">
<figure class="visual-card">
<img src="static/images/avp-imc-contact.jpg" alt="Portals deployments across mobile and immersive hardware" class="paper-figure">
<figcaption class="figure-caption">Representative deployment surfaces spanning mobile AR, spatial capture, and immersive display.</figcaption>
</figure>
<figure class="visual-card">
<img src="static/images/portals-ai-contact.jpg" alt="Portals AI-assisted spatial authoring workflow" class="paper-figure">
<figcaption class="figure-caption">AI-assisted spatial composition and scene control inside the shipped creator workflow.</figcaption>
</figure>
<figure class="visual-card">
<img src="static/images/webgl-splat-contact.jpg" alt="Browser-based splat and web delivery view from Portals" class="paper-figure">
<figcaption class="figure-caption">Browser-facing WebGL and WebGPU delivery path for persistent spatial media on the web.</figcaption>
</figure>
</div>
</div>
</div>
</div>
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column is-four-fifths">
<h2 class="title is-3 has-text-centered">Device Captures</h2>
<p class="has-text-centered" style="margin-bottom:1.5rem;color:#666;">All imagery captured on physical devices — iPhone 15 Pro, iPad Pro, and Apple Vision Pro.</p>
<div class="visual-grid visual-grid-secondary">
<figure class="visual-card">
<img src="static/images/hero_composer.jpg" alt="Voice composer with AR VFX and voice command overlay" class="paper-figure">
<figcaption class="figure-caption">Voice-driven AR composer with real-time VFX. Voice command "add a pink sphere" parsed locally in <1ms.</figcaption>
</figure>
<figure class="visual-card">
<img src="static/images/avp_vision_pro.jpg" alt="Apple Vision Pro running Portals at IMC with NYC skyline" class="paper-figure">
<figcaption class="figure-caption">Apple Vision Pro rendering interactive 3D worlds with passthrough at The IMC Lab, NYC.</figcaption>
</figure>
<figure class="visual-card">
<img src="static/images/mobile_composer.png" alt="Portals mobile composer UI" class="paper-figure">
<figcaption class="figure-caption">Full AR authoring UI on iPhone 15 Pro with voice input, VFX palette, and scene manipulation.</figcaption>
</figure>
</div>
<div class="visual-grid visual-grid-secondary" style="margin-top:1rem;">
<figure class="visual-card">
<img src="static/images/xr_stage.jpg" alt="iPad running real-time voxelized body tracking on XR LED stage" class="paper-figure">
<figcaption class="figure-caption">iPad at 60fps with voxelized body tracking on an LED wall XR production stage.</figcaption>
</figure>
<figure class="visual-card">
<img src="static/images/shader_gallery.jpg" alt="XR shader effects gallery" class="paper-figure">
<figcaption class="figure-caption">Compute shaders on-device: galaxy simulations, particle systems, cellular automata, anatomical models.</figcaption>
</figure>
<figure class="visual-card">
<img src="static/images/hologram_env.jpg" alt="Hologram mode with generative environment VFX" class="paper-figure">
<figcaption class="figure-caption">Hologram mode with procedural environment VFX and real-time voice parametric control.</figcaption>
</figure>
</div>
<div class="visual-grid visual-grid-secondary" style="margin-top:1rem;">
<figure class="visual-card">
<img src="static/images/neural_avatar.jpg" alt="Neural style transfer avatar" class="paper-figure">
<figcaption class="figure-caption">Real-time neural style transfer applied to volumetric avatar capture.</figcaption>
</figure>
<figure class="visual-card">
<img src="static/images/depth_installation.jpg" alt="Depth-to-VFX pipeline in gallery installation" class="paper-figure">
<figcaption class="figure-caption">Sparse depth maps driving interactive projections in a gallery installation.</figcaption>
</figure>
</div>
</div>
</div>
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column is-four-fifths">
<h2 class="title is-3 has-text-centered">Paper Figures</h2>
<div class="content has-text-centered">
<img src="static/images/architecture.png" alt="Portals system architecture figure from the paper" class="paper-figure">
<p class="figure-caption">System architecture showing the persistent world stack, shared spatial-media substrate, and deployment surfaces across mobile, headset, and web clients.</p>
<img src="static/images/content-pipeline.png" alt="Portals content pipeline from the paper" class="paper-figure">
<p class="figure-caption">Content pipeline across capture, reconstruction, editing, and deployment. In the shipped system, asset ingestion centers on phone capture, photogrammetry, and text or reference-image generation, while voice, direct manipulation, and no-code controls act as authoring surfaces over the same persistent scene graph.</p>
<img src="static/images/model-diagrams.png" alt="Portals model and representation diagrams from the paper" class="paper-figure figure-stack">
<p class="figure-caption">Model and representation diagrams summarizing how environments, volumetric humans, and spatial media fit into the same editable 4D world framework.</p>
<img src="static/images/lod-performance.png" alt="Portals level-of-detail and performance figure from the paper" class="paper-figure figure-stack">
<p class="figure-caption">Level-of-detail and performance figure highlighting mobile-first runtime constraints and the efficiency tradeoffs required for edge deployment.</p>
<img src="static/images/systems-comparison.png" alt="Systems comparison figure from the paper" class="paper-figure figure-stack">
<p class="figure-caption">Comparison of Portals against prior systems along persistence, editability, and deployment-relevant world-model axes.</p>
</div>
</div>
</div>
</div>
</section>
<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column is-four-fifths">
<h2 class="title is-3 has-text-centered">Status</h2>
<div class="content has-text-centered">
<p>The paper PDF linked here matches the final submission build. It documents a mobile-first Unity URP and VFX Graph runtime for persistent editable 4D worlds spanning reconstructed environments, volumetric humans, and holographic spatial media, with a WebGPU-first browser path for web delivery. The interactive web demo, code release, and video will be published after workshop timing is finalized.</p>
</div>
</div>
</div>
</div>
</section>
<footer class="footer">
<div class="container">
<div class="content has-text-centered">
<a class="icon-link" href="static/pdfs/portals_cvpr2026_workshop_paper.pdf" target="_blank">
<i class="fas fa-file-pdf"></i>
</a>
</div>
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>
This page reuses the same academic project-page template family used across workshop speaker sites, adapted from the Nerfies/Bulma project page pattern.
</p>
<p class="footer-links">
<a href="mailto:JTunick@TheIMCLab.com">JTunick@TheIMCLab.com</a>
<span class="meta-divider">•</span>
<a href="https://h3m.ai" target="_blank" rel="noopener noreferrer">H3M.ai</a>
<span class="meta-divider">•</span>
<a href="https://www.theimclab.com" target="_blank" rel="noopener noreferrer">The IMC Lab</a>
<span class="meta-divider">•</span>
<a href="https://github.com/imclab" target="_blank" rel="noopener noreferrer">imclab GitHub</a>
<span class="meta-divider">•</span>
<a href="https://github.com/JT5D" target="_blank" rel="noopener noreferrer">JT5D GitHub</a>
</p>
</div>
</div>
</div>
</div>
</footer>
</body>
</html>