Set Up a Depth Product

Goal. Apply the depth model’s API schema to a Camera and configure its RenderProduct to request the three renderer inputs and five sensor outputs.

Before you start. Have a camera scene and understand RenderProducts and RenderVars. New to ovrtx? Start with Your First Lidar Sensor. See Camera single-view stereo depth model for the model’s capabilities and limits.

Apply the schema and request the outputs

The model’s attributes are authored on the Camera prim. Apply OmniSensorGenericDepthSensorSingleViewAPI to it and author the depth parameters there; the RenderProduct points its rel camera at that camera and requests the outputs, and carries no depth attributes of its own. There is no sensor prim. Where the schema applies, and the identity attributes it brings, is on USD Attributes; applying an API schema in general is on Sensor Configuration.

The scene below is complete. The camera applies the schema and authors five of the attributes; the product points at that camera and lists all eight RenderVars in orderedVars: the three inputs the model reads, LdrColor, DistanceToImagePlaneSD and Camera3dPositionSD, then the five DepthSensor* outputs it writes. This example requests all five outputs; you can omit the outputs you do not need. Requesting DepthSensor* outputs enables the post-processing pipeline; applying the schema sets its parameters. The required set explains the input and output requirements.

USDA

#usda 1.0
def Xform "World"
{
    def Camera "Camera" (
        prepend apiSchemas = ["OmniSensorGenericDepthSensorSingleViewAPI"]
    )
    {
        float omni:sensor:depthSensor:baselineMM = 55.0
        float omni:sensor:depthSensor:focalLengthPixel = 897.0
        float omni:sensor:depthSensor:sensorSizePixel = 1280.0
        float omni:sensor:depthSensor:maxDisparityPixel = 110.0
        float omni:sensor:depthSensor:minDistance = 0.5
    }
}

def Scope "Render"
{
    def RenderProduct "CameraDepthSingleView"
    {
        rel camera = </World/Camera>
        rel orderedVars = [
            <LdrColor>,
            <DistanceToImagePlaneSD>,
            <Camera3dPositionSD>,
            <DepthSensorImager>,
            <DepthSensorDistance>,
            <DepthSensorPointCloudPosition>,
            <DepthSensorPointCloudColor>,
            <DepthSensorOutputVisualization>
        ]
        uniform int2 resolution = (1280, 720)

        def RenderVar "LdrColor" {
            string sourceName = "LdrColor"
        }
        def RenderVar "DistanceToImagePlaneSD" {
            string sourceName = "DistanceToImagePlaneSD"
        }
        def RenderVar "Camera3dPositionSD" {
            string sourceName = "Camera3dPositionSD"
        }
        def RenderVar "DepthSensorImager" {
            string sourceName = "DepthSensorImager"
        }
        def RenderVar "DepthSensorDistance" {
            string sourceName = "DepthSensorDistance"
        }
        def RenderVar "DepthSensorPointCloudPosition" {
            string sourceName = "DepthSensorPointCloudPosition"
        }
        def RenderVar "DepthSensorPointCloudColor" {
            string sourceName = "DepthSensorPointCloudColor"
        }
        def RenderVar "DepthSensorOutputVisualization" {
            string sourceName = "DepthSensorOutputVisualization"
        }
    }
}

USD Python

from pxr import Gf, Sdf, UsdGeom, UsdRender

stage.DefinePrim("/World", "Xform")
camera = UsdGeom.Camera.Define(stage, "/World/Camera").GetPrim()
camera.AddAppliedSchema("OmniSensorGenericDepthSensorSingleViewAPI")
for name, value in {
    "baselineMM": 55.0,
    "focalLengthPixel": 897.0,
    "sensorSizePixel": 1280.0,
    "maxDisparityPixel": 110.0,
    "minDistance": 0.5,
}.items():
    camera.CreateAttribute(
        f"omni:sensor:depthSensor:{name}", Sdf.ValueTypeNames.Float, custom=False
    ).Set(value)

stage.DefinePrim("/Render", "Scope")
product = UsdRender.Product.Define(stage, "/Render/CameraDepthSingleView")
prim = product.GetPrim()
product.GetCameraRel().SetTargets([camera.GetPath()])
product.GetResolutionAttr().Set(Gf.Vec2i(1280, 720))

# The three camera outputs the model reads, then the five it writes.
names = [
    "LdrColor",
    "DistanceToImagePlaneSD",
    "Camera3dPositionSD",
    "DepthSensorImager",
    "DepthSensorDistance",
    "DepthSensorPointCloudPosition",
    "DepthSensorPointCloudColor",
    "DepthSensorOutputVisualization",
]
for name in names:
    render_var = UsdRender.Var.Define(stage, f"/Render/CameraDepthSingleView/{name}")
    render_var.GetSourceNameAttr().Set(name)
product.GetOrderedVarsRel().SetTargets(
    [f"/Render/CameraDepthSingleView/{name}" for name in names]
)

The five values shown are the schema defaults, written out; the other ten attributes take theirs. The generated Sensor API Schemas table lists every attribute with its type and default, and USD Attributes explains what each does. Every RenderVar in orderedVars is a prim of its own whose sourceName names the output it carries (RenderVar); here they are children of the product, and they can equally live in a shared Vars scope that several products reference (Independent RenderProducts). The three inputs are ordinary camera outputs, described on Cameras: Outputs.

Note

Match the product’s resolution to the real depth stream’s output mode. Configure focalLengthPixel and sensorSizePixel from the same camera calibration, and keep the rendered Camera prim’s intrinsics consistent with it. The optics group explains the relationship and how to use a different output resolution.

Several products, cameras and projections

The parameters belong to the product, not to the camera. Two depth products rendered in the same step can have different parameter values, so comparing two devices in one scene is one product per device, each with its own attribute values, and the step result contains each product’s frames; stepping several products at once is described on Output Configuration.

Cameras can use different projections. One product’s camera can be a pinhole and another’s an OpenCV fisheye, authored as Lens Calibration describes. The model does not take the projection into account: it shifts every sample along its image row by the same disparity formula whatever lens produced the image. The formula and the attributes in it are on USD Attributes.

A tiled product shares one parameter set. A product whose rel camera lists several cameras renders each camera into its own tile of one frame (The rel camera relationship). The depth simulation then runs once over that whole frame with the one set of parameters authored on the product, shared by every camera in it. When each camera needs its own baseline, optics or noise, give each camera its own product instead (Independent RenderProducts).

Run and read

Step the renderer with the product’s path in its set of render products, as for any camera, and warm it up before you judge the result (Warming up the renderer; Invalid measurements has the note on a capture taken too early). Each returned frame holds one output per RenderVar, keyed by the RenderVar’s full prim path, here /Render/CameraDepthSingleView/DepthSensorDistance, not by its sourceName. Map each output and read its tensor as Sensor Outputs describes for every sensor (Mapping API); the five depth outputs are images at the product’s resolution.

This model ships without a public example. The product authored on this page is the complete configuration, and the steps above run and read it.

Verify it worked

Check the visualization first, then the numbers.

• Look at the visualization. DepthSensorOutputVisualization is the one output meant for viewing. Set rgbDepthOutputMode to a mode that shows distance; mode 3 is the quickest check, because a pixel with no measurement is black there. Author a realistic maxDistance before you judge the gradient itself, or it stays uniform. Outputs lists the modes.

• Read the distance. DepthSensorDistance should be non-zero over textured, lit surfaces inside the range limits. What a zero in it means, and why large empty regions are normal on an untextured scene, is on Invalid measurements.

• Know what you are reading. Outputs explains what each output carries and the two frames the distance image and the point cloud are expressed in.

When it goes wrong

The distance is zero everywhere. Check the request first: a missing input or output stops the simulation and the reason is logged as an error (The required set). Then render more frames before you judge it; where the color has not resolved yet, the confidence test has no texture to pass (Warming up the renderer).

An output is missing from the frame although its RenderVar is authored. You are probably looking it up by sourceName; the frame keys its outputs by the RenderVar’s full prim path (Run and read).

Large regions are empty on a scene that renders. That is the confidence test at work on smooth, untextured surfaces, where it leaves holes by design; Tune Noise, Quantization and Confidence raises coverage through scene texture, lighting contrast, the simulated IR projector or the threshold.

The distance image and the point cloud disagree. They are expressed in different frames and, on a stage whose metersPerUnit is not 1, in different units; Two frames and Units say which is which.

Anything else is on the Symptom Index.

Related

• Camera single-view stereo depth model — what the model reproduces from one rendered view.

• USD Attributes — the fourteen attributes in their groups, with units, clamps and relations.

• Outputs — the required set, the five outputs, what a zero means, the two frames and the visualization modes.

• Build a Profile from a Spec Sheet — making the product behave like a real device, from its datasheet.