Gaussian Splats#

A ParticleField is a USD prim containing Gaussians (also called splats). Each Gaussian is an oriented, translucent ellipsoid with a position, rotation, scale, opacity, and radiance. Use a ParticleField to render captured appearance or procedurally authored Gaussians.

The same captured cactus rendered with RTPT, PT, and Minimal

Left to right: RTPT, PT, and Minimal, after 32 frames with the same captured ParticleField and camera. The modes differ in lighting, background, and image reconstruction.#

Note

With the default emissive material, Gaussians store captured lighting in their radiance. Scene lights affect them through shadows rather than direct illumination: changing a light’s intensity or color does not directly brighten or tint the Gaussians, but scene objects can cast shadows onto them.

Without a bound material, ParticleField3DGaussianSplat uses ParticleFieldEmissive. To adjust brightness on an explicitly bound instance of this material, set inputs:emission_intensity on its shader. The default is 1.0; lower values dim the Gaussians and higher values brighten them.

Converting 3DGS PLY Files to USD#

Use NVIDIA’s usd-convert-gsplat to convert a 3D Gaussian Splat (3DGS) .ply file to USD with a ParticleField3DGaussianSplat prim. The input should contain 3DGS data, including scales, rotations, opacity, and spherical harmonics, rather than only point positions or mesh geometry.

From your uv project directory, add the converter and OpenUSD Python bindings. See Getting Started in Python for the Python setup:

uv add usd-convert-gsplat usd-core

Convert your 3DGS file to a binary USD file, replacing the example paths with your own:

uv run python -m usd_convert_gsplat -i "scene.ply" -o "scene.usdc" --rotate-x 180

The optional --rotate-x 180 applies a 180-degree rotation around the X axis. Adjust or omit it to match your source data’s orientation to your scene; this rotation is not required for every dataset.

To render the converted asset, compose it with a camera and render-output configuration. See the Inline Composition section of Loading USD for a scene-wrapper example, and Output Configuration for the RenderProduct and RenderVar setup. See Python: Gaussian Splats for this example’s scene requirements and run commands.

Creating a ParticleField#

Author a ParticleField using the ParticleField3DGaussianSplat prim type. Its schema supplies the ParticleField APIs; you do not need to apply them individually. Each per-Gaussian array must describe the same number of Gaussians.

The following equivalent examples create a ParticleField containing one orange Gaussian. The quaternion is written with its real component first, and the scales are linear values in scene units. The example uses degree-zero spherical harmonics (SH0), so the color does not depend on viewing direction.

def ParticleField3DGaussianSplat "Splat" {
    point3f[] positions = [(0, 0, 0)]
    quatf[] orientations = [(1, 0, 0, 0)]
    float3[] scales = [(1, 0.5, 0.25)]
    float[] opacities = [0.9]
    uniform int radiance:sphericalHarmonicsDegree = 0
    # SH0 = (linear RGB - 0.5) / 0.2820947918; this is orange RGB (1, .3, 0).
    float3[] radiance:sphericalHarmonicsCoefficients = [(1.772454, -0.708982, -1.772454)]
}

splat = stage.DefinePrim("/Splat", "ParticleField3DGaussianSplat")
splat.CreateAttribute("positions", Sdf.ValueTypeNames.Point3fArray, custom=False).Set([(0, 0, 0)])
splat.CreateAttribute("orientations", Sdf.ValueTypeNames.QuatfArray, custom=False).Set([Gf.Quatf(1)])
splat.CreateAttribute("scales", Sdf.ValueTypeNames.Float3Array, custom=False).Set([(1, 0.5, 0.25)])
splat.CreateAttribute("opacities", Sdf.ValueTypeNames.FloatArray, custom=False).Set([0.9])
splat.CreateAttribute("radiance:sphericalHarmonicsDegree", Sdf.ValueTypeNames.Int,
                      custom=False, variability=Sdf.VariabilityUniform).Set(0)
splat.CreateAttribute("radiance:sphericalHarmonicsCoefficients", Sdf.ValueTypeNames.Float3Array,
                      custom=False).Set([(1.772454, -0.708982, -1.772454)])
Gaussian attributes#

USD declaration

Meaning

point3f[] positions

Gaussian centers in local scene units.

quatf[] orientations

Normalized Gaussian rotations; identity is (1, 0, 0, 0).

float3[] scales

Positive linear scale along each Gaussian’s local axis.

float[] opacities

Peak opacity for each Gaussian; use values from 0 to 1.

int radiance:sphericalHarmonicsDegree = 3

Schema default; author 0 for the single-coefficient example shown here.

float3[] radiance:sphericalHarmonicsCoefficients

RGB spherical-harmonic coefficients, grouped by Gaussian. Degree 0 uses one float3 per Gaussian.

For degree zero, convert linear RGB to a coefficient with SH0 = (RGB - 0.5) / 0.2820947918. Do not put display-encoded RGB values directly into the coefficient array. This example uses degree zero, with one RGB coefficient per Gaussian.

Render Modes#

Select the mode on the RenderProduct using omni:rtx:rendermode.

Mode

Token

Example behavior

Real-Time Path Tracing

RealTimePathTracing

Uses stochastic transparency by default; optional non-stochastic primary-ray compositing is described below.

Path Tracing

PathTracing

Renders the same field with path-tracing accumulation and reconstruction.

Minimal

MinimalRendering

Use Full MDL and DLSS Ray Reconstruction as configured below. Lighting and background differ from RTPT/PT.

For Minimal, set both int omni:rtx:minimal:mode = 3 (Full MDL) and int omni:rtx:minimal:dlss:mode = 2 (DLSS Ray Reconstruction). Full MDL renders the Gaussians; Ray Reconstruction denoises their stochastic transparency. Author the DLSS attribute explicitly; it is not declared in the generated RTX settings schema.

For accumulated Gaussian depth and albedo settings, see Real-Time Path Tracing.

Non-Stochastic Rendering#

RTPT can composite emissive Gaussians in a ParticleField on primary camera rays without randomly accepting or rejecting each Gaussian for transparency. Enable omni:rtx:rtpt:gaussian:nonStochastic:enabled on the RenderProduct; its default is false. This reduces transparency noise without waiting for temporal accumulation, at a performance cost compared with stochastic rendering. It does not make lighting, shadows, or the entire image noise-free.

Close-up of the cactus with non-stochastic rendering disabled and enabled

RTPT first frame through Camera_close: disabled (left), enabled (right). Both use the same scene and disable adaptive disocclusion sampling; only the non-stochastic attribute differs. Matching crops enlarged 3× without smoothing.#

The following overrides enable this mode on an existing /Render/Camera RenderProduct and disable incompatible adaptive disocclusion sampling:

over "Render" {
    over "Camera" {
        rel camera = </World/Camera_close>
        token omni:rtx:rendermode = "RealTimePathTracing"
        bool omni:rtx:rtpt:gaussian:nonStochastic:enabled = true
        # Adaptive disocclusion sampling would force stochastic rendering.
        bool omni:rtx:rtpt:adaptiveSampling:disocclusion:enabled = false
    }
}

product = UsdRender.Product.Get(stage, "/Render/Camera")
product.GetCameraRel().SetTargets(["/World/Camera_close"])
prim = product.GetPrim()
prim.CreateAttribute("omni:rtx:rendermode", Sdf.ValueTypeNames.Token).Set("RealTimePathTracing")
prim.CreateAttribute("omni:rtx:rtpt:gaussian:nonStochastic:enabled", Sdf.ValueTypeNames.Bool).Set(True)
# Adaptive disocclusion sampling would force stochastic rendering.
prim.CreateAttribute(
    "omni:rtx:rtpt:adaptiveSampling:disocclusion:enabled", Sdf.ValueTypeNames.Bool
).Set(False)

Non-stochastic mode is only supported in Real-Time Path Tracing. The non-stochastic path is for ParticleFieldEmissive; it does not apply to the diffuse material below.

At most three overlapping ParticleFields are retained for primary-ray compositing. Each field can contain many Gaussians. Fields are ordered by their mean depth; Gaussians belonging to different fields are not interleaved. Intersecting ParticleFields can therefore show ordering artifacts.

Diffuse Relighting#

ovrtx supplies ParticleFieldSHDiffuse.mdl with its runtime materials, at bin/mdl/ParticleFieldSHDiffuse.mdl in the extracted SDK package. You do not need to write an MDL material. Bind it to a ParticleField to illuminate its Gaussians with scene lights in PT or RTPT. Unlike the default emissive material, this material has no emission. It derives diffuse albedo from the degree-zero spherical-harmonics coefficient (SH0). Light intensity and color then affect the rendered Gaussians.

Cactus with captured emissive appearance and scene-lit SH diffuse material

RTPT after 32 frames through Camera: default emissive material (left), ParticleFieldSHDiffuse with normal tone mapping (right). The scene, camera, and studio lighting are shared.#

Captured illumination remains baked into SH0: this is not a method for recovering unlit surface color from a captured scene, but can serve as a reasonable approximation for many use cases.

The following overrides bind the packaged material to an existing /World/Splats ParticleField and select normal tone mapping on /Render/Camera. The material has no exposed inputs; keep the SH coefficients on the ParticleField itself:

over "World" {
    def Material "SHDiffuse" {
        token outputs:mdl:surface.connect = </World/SHDiffuse/Shader.outputs:out>
        def Shader "Shader" {
            uniform token info:implementationSource = "sourceAsset"
            uniform asset info:mdl:sourceAsset = @ParticleFieldSHDiffuse.mdl@
            uniform token info:mdl:sourceAsset:subIdentifier = "ParticleFieldSHDiffuse"
            token outputs:out (renderType = "material")
        }
    }
    over "Splats" (
        prepend apiSchemas = ["MaterialBindingAPI"]
    ) {
        rel material:binding = </World/SHDiffuse>
    }
}
over "Render" {
    over "Camera" {
        # Render scene-lit color through the normal color pipeline.
        bool omni:rtx:gaussian:skipTonemapping:enabled = false
        bool omni:rtx:rtpt:gaussian:nonStochastic:enabled = false
    }
}

material = UsdShade.Material.Define(stage, "/World/SHDiffuse")
shader = UsdShade.Shader.Define(stage, "/World/SHDiffuse/Shader")
shader.SetSourceAsset(Sdf.AssetPath("ParticleFieldSHDiffuse.mdl"), "mdl")
shader.SetSourceAssetSubIdentifier("ParticleFieldSHDiffuse", "mdl")
output = shader.CreateOutput("out", Sdf.ValueTypeNames.Token)
output.SetRenderType("material")
material.CreateSurfaceOutput("mdl").ConnectToSource(output)
# Bind to the existing ParticleField; its SH0 data becomes diffuse albedo.
splats = stage.GetPrimAtPath("/World/Splats")
UsdShade.MaterialBindingAPI.Apply(splats).Bind(material)
product = stage.GetPrimAtPath("/Render/Camera")
# Render scene-lit color through the normal color pipeline.
product.CreateAttribute("omni:rtx:gaussian:skipTonemapping:enabled", Sdf.ValueTypeNames.Bool).Set(False)
product.CreateAttribute("omni:rtx:rtpt:gaussian:nonStochastic:enabled", Sdf.ValueTypeNames.Bool).Set(False)

These examples keep non-stochastic compositing disabled because diffuse Gaussians use stochastic transparency. Accumulate frames as needed. See Python: Gaussian Splats and C: Gaussian Splats for runnable variants of the same scene.

Supplied Material Source#

The supplied material derives diffuse albedo from SH0 and uses a diffuse reflection BSDF. Its source is shown below and is also available as a download:

mdl 1.6;

import ::df::*;
import ::math::*;
import ::scene::*;

// Treat the view-independent spherical-harmonics color as diffuse albedo.
export material ParticleFieldSHDiffuse() = let {
    float3 sh0 = scene::data_lookup_float3("radiance:sphericalHarmonicsCoefficients_0", float3(0.0));
    // Reconstruct RGB from the degree-zero coefficient: RGB = 0.5 + Y00 * SH0.
    float3 albedo = math::max(float3(0.0), float3(0.5) + 0.28209479177387814 * sh0);
} in material(
    surface: material_surface(
        scattering: df::diffuse_reflection_bsdf(tint: color(albedo))
    )
);

Limitations#

  • With stochastic transparency, a single frame can be noisy. Warm up and inspect the result at the resolution and frame count your application uses.

Further Reading#