The first person I placed into the cleaned Garden scene towered over it: the person asset was about 3.64 authored units tall, the garden’s final extent was about 17.9, and I had never told SplatStage how long a meter was. The app had followed my inputs. The result looked ridiculous because the scene was visually coherent and metrically unknown.
That result made the gap concrete for me. A Gaussian splat can look uncannily like a place and still know almost nothing about that place.
It may not know which way is up. The machine in the image may be color and opacity rather than an identified asset. The floor may look solid but offer no collision surface. Nothing in the representation says whether a door is open now, which sensor belongs to it, or what should happen when a robot reaches it.
Gaussian splatting had done its job. I was asking the result to make a second promise: reconstruct appearance first, then become part of a digital twin.
Captured appearance
Authored composition, unresolved scale
ReconStudio gave me a repeatable path from photographs or video to a Gaussian splat and an OpenUSD asset. The next question was what it would take to move that reconstruction into a scene where captured context and engineered assets could coexist. I built SplatStage to investigate that boundary.
My starting point is the industrial side of digital twins: asset models, operational data and the decisions the system needs to support. For years, I treated reconstruction and scene authoring as a specialist handoff. Building ReconStudio and SplatStage forced me to understand how choices about coordinate frames, scale, USD references and collision proxies shape everything that comes afterward.
These notes are for people making the same crossing—from the operational and data side of a twin into reconstruction and 3D authoring. The question I keep returning to is how each 3D choice changes what the downstream twin can honestly claim. I have kept the lessons that changed an architectural decision or narrowed that claim; the smaller implementation discoveries stay in the build record.
This series has three parts. This article defines the mental model. Part 2 covers the application and its architecture. Part 3 tests the workflow on a 5.8-million-Gaussian scene and separates what worked from what remains unfinished.
One shared foundation, then two branches
I find it more useful to ask what a scene is ready for than to debate whether it deserves the word twin. Reconstruction and visual readiness form a shared path. After that, operational readiness and simulation readiness are separate branches:
| Level | The scene can answer | Evidence it needs |
|---|---|---|
| Reconstruction | “What might this place look like from another nearby viewpoint?” | registered cameras, Gaussian positions and shapes, opacity and view-dependent color |
| Visual-ready foundation | “Can I place and inspect this capture with engineered assets in a coherent scene?” | useful bounds, recorded orientation, measured scale or an explicit unknown, replaceable OpenUSD references, a dependency-complete package |
| Operational twin | “What is this asset, and what is happening to it now?” | stable identity, semantics, telemetry, timestamps, relationships and operating rules |
| Simulation-ready scene | “How will bodies, mechanisms and sensors behave?” | collision geometry, physical properties, joints, materials, sensor models and validation in the target runtime |
The first two rows build on each other. The final two do not have to arrive in order. A collider does not make an asset operational, and a live temperature reading does not tell a physics engine how a pallet should contact the floor. A system that needs both branches has to connect them explicitly.
The distinction also prevents scope inflation. In this project, I can show that a splat was cleaned, oriented, exported and composed with other OpenUSD assets. I cannot use that result to claim a live or physically validated twin.
What a splat preserves
The original 3D Gaussian Splatting work represents a scene with many anisotropic Gaussians optimized from camera images. Each Gaussian carries a position, a shape and orientation, opacity, and coefficients used to reconstruct view-dependent appearance.
This representation works well for novel-view rendering. It is also why a splat can contain a wispy tree, a reflection, the soft edge of foliage and a complicated room without first converting all of them into clean surfaces.
It does not follow that the same representation contains the facts needed for engineering work.
| A splat often contains | It does not automatically contain |
|---|---|
| relative spatial arrangement | verified real-world dimensions |
| captured appearance | object identity or part numbers |
| Gaussian extent and orientation | watertight surfaces |
| opacity and view-dependent color | collision behavior |
| artifacts left by reconstruction | intent about what should remain |
| a coordinate frame created by the solve | a trusted facility coordinate system |
OpenUSD can carry the Gaussian data in a ParticleField3DGaussianSplat prim. That makes the representation composable and inspectable in an OpenUSD stage. The schema does not conjure semantics, metric scale or physics that the source never measured.
I keep the splat as the appearance layer and use other layers for the facts the cameras never measured.
Cleanup, fidelity and intent are different edits
Once the reconstruction exists, “editing the splat” sounds like one problem. It is at least three.
Cleanup removes evidence that the reconstruction probably invented
Gaussian reconstructions often include distant floaters, large translucent clouds or elongated needles. Those can come from weakly observed regions, reflections, sky or unstable estimates. Statistical and geometric signals can help find them: robust bounds, opacity, Gaussian scale and shape ratios.
This is where the representation is unusually editable. The operation can examine the Gaussian fields directly and produce another valid splat without turning it into a mesh.
But geometric evidence is not certainty. A thin cable and an elongated artifact can look similar to a shape filter. A transparent surface and low-confidence haze can both have low opacity. The safer pattern is preview, count, inspect and then commit to a new version.
Fidelity edits decide how much of the reconstruction to keep
Decimation, spherical-harmonic reduction and opacity pruning trade storage or rendering cost against visual quality. These are not cleanup in the same sense. A smaller representation may be the right product output even when every removed Gaussian was valid.
The decision needs a use case and a comparison, not just an outlier rule. The ReconStudio experiments had already shown that Gaussian count, output size and measured image quality do not move together in a simple way.
Intentional edits require meaning the splat does not have
“Remove the vase” is different from “remove Gaussians outside this robust bound.” A splat does not normally label a coherent group as vase. In SplatStage I can select a cylinder around the object and remove the Gaussians inside it. The geometry of that instruction is exact. Its semantics come from the person making the selection.
The result can also expose a hole, blur or grey smear because the cameras did not record what was behind the removed object. Deletion is not inpainting. It removes captured evidence; it does not reconstruct the hidden scene.
That is why I made SplatStage store the selection, operation and before/after evidence separately instead of treating a convincing viewport as the audit trail.
A visual twin needs a coordinate claim
Composition forces a question a standalone viewer can avoid: where does this capture belong relative to everything else?
A camera reconstruction establishes its own coordinate system. It may be internally coherent without being aligned to gravity or scaled in meters. To place a known asset, the stage needs at least:
- an up direction derived from selected ground points or another reliable reference;
- an origin or anchor that makes placement repeatable;
- a scale derived from a measured distance, or an explicit statement that scale remains unverified;
- a record of the transform applied to the capture.
The word “derived” matters. Declaring metersPerUnit = 1 in an OpenUSD stage says how to interpret a unit; it is not proof that the reconstruction was measured in meters. If I skip the known-distance calibration, the honest result is a scene whose units are asserted but whose real-world scale remains unverified.
SplatStage supports a two-point distance calibration and ground-plane fitting. In the Garden run covered in Part 3, I measured the up direction but skipped the known-distance step while completing the structural path. I left the resulting scale unresolved rather than correcting it by eye, because a plausible transform would still not establish a measured scale.
Why the splat and the engineered asset should remain separate
The tempting alternative is to merge everything into one baked result. I chose the opposite structure.
The Gaussian reconstruction stays a captured-background layer. Equipment, people, beacons or CAD-derived assets remain ordinary OpenUSD references. A small wrapper stage places both under a common world:
/World
/Capture → cleaned Gaussian ParticleField
/Assets → replaceable engineered USD references
/Physics → reserved for later physical layersThis separation has practical consequences:
- The capture can be cleaned again without rebuilding the engineered assets.
- A stand-in can be replaced by a production asset without rewriting the splat.
- Semantics and live connections can attach to asset prims without pretending the captured pixels already knew them.
- Collision geometry can come from CAD or authored proxies rather than from the visual Gaussian field.
- Reviewers can inspect which layer supports which claim.
OpenUSD references carry that composition. A reference adds another layer or asset to the stage while allowing the source to remain independently replaceable; the OpenUSD glossary describes the behavior in more detail.
I keep them separate so a reviewer can see which claims come from the captured scene and which come from engineered assets.
The boundary I call visual-ready
For this project, I use “visual-ready twin foundation” for a scene that meets six conditions:
- The useful captured region has been separated from obvious reconstruction debris.
- The stage records how its up direction was established.
- Metric scale is either measured or explicitly marked as unverified.
- The Gaussian capture and engineered assets are separate, replaceable OpenUSD layers.
- The final viewport reads the composed USD stage rather than silently falling back to the working PLY.
- The package contains its referenced dependencies under relative paths, and its contents can be inspected without the authoring application.
That is already useful. It creates a shared visual canvas for planning, review and the next round of authoring. It is also a much narrower promise than an operational twin.
The third condition is deliberately weak. Marking scale unknown prevents a silent metric claim and still allows structural composition. It does not make CAD placement dimensionally correct. Any use that depends on accurate distances has to treat scale calibration as a release gate.
SplatStage contains an early path for authoring collider proxies, but the Garden workflow did not exercise it. The exported stage has no colliders, physical materials, joints, sensors, operational identities or live data. I have not yet validated the packaged stage in the external simulation runtimes I care about. The composed visual layer can support an operational twin next; collider, physics and runtime work remain a separate simulation-readiness build.
I use visual-ready to name that boundary. At that point I can clean, orient, compose and inspect the captured layer, but I still cannot use it for live operations or physical simulation.
In Part 2, I build the workflow behind that promise: immutable splat versions, selection predicates, a direct PLY-to-ParticleField exporter, four explicit viewing modes and a wrapper stage that keeps the capture separate from engineered assets.
Disclaimer: The views and opinions expressed in this account are those of my own and do not represent those of my employer, NVIDIA.