Read a Sensor Composite#

Goal. Consume a lidar or radar point cloud in a node.

Before you start. Choose a Resource’s Backing.

The Shape#

A composite arrives as a table of named channels under one render var. Test for it, then take the channels you need by name:

local pc = inputs["PointCloud"]
assert(pc.isComposite, "PointCloud must be connected to a composite AOV")

local coordinates = pc.tensors["Coordinates"]
local counts      = pc.tensors["Counts"]

How It Works#

Highlighted: the test that distinguishes a composite from an ordinary AOV, and the two channels taken out of it. What the node then does with those channels is ordinary buffer work, and is in Choose a Resource’s Backing.

-- Reading a composite AOV.
--
-- A sensor publishes a composite: a table of named channels under one render
-- var, rather than the single resource a camera AOV gives you. Test
-- inputs["X"].isComposite to tell the two apart, then take a channel out of
-- .tensors by name. Each one is an ordinary resource descriptor and binds like
-- any other buffer. .channelNames lists what the composite actually carries,
-- which follows from the channels authored on the RenderVar in the scene.
local WIDTH, HEIGHT = 512, 256
local THREADS = 256

function rangeHistogram(inputs, outputs)
    local pc = inputs["PointCloud"]
    assert(pc.isComposite, "PointCloud must be connected to a composite AOV")

    local coordinates = pc.tensors["Coordinates"]
    local counts = pc.tensors["Counts"]
    assert(coordinates ~= nil, "composite has no Coordinates channel")
    assert(counts ~= nil, "composite has no Counts channel")
    assert(#coordinates.shape == 2, "Coordinates must be [3, Nmax]")
    assert(coordinates.shape[1] == 3, "Coordinates must hold x, y and z runs")

    -- Nmax, the capacity Coordinates is allocated for. Counts says how many of
    -- those entries the sweep actually filled, but a launch script sees
    -- descriptors rather than data, so it cannot read that number. Counts is
    -- passed to the kernel instead and the bound is applied there.
    local maxPoints = coordinates.shape[2]

    -- The sensor publishes scalars alongside the channels. maxPoints is the same
    -- capacity the Coordinates shape reports, so reading it is a cheap check that
    -- this port is carrying the composite the node expects.
    assert(pc.params["maxPoints"] ~= nil, "composite has no maxPoints parameter")

    -- numBins splits the chart width into bars in the kernel, so it has to divide it.
    local numBins = inputs["numBins"].value
    assert(numBins > 0 and WIDTH % numBins == 0, "numBins must divide the chart width evenly")

-- Reading a composite AOV on the Slang backend.
--
-- A sensor publishes a composite: a table of named channels under one render
-- var, rather than the single resource a camera AOV gives you. Test
-- inputs["X"].isComposite to tell the two apart, then take a channel out of
-- .tensors by name. Each one is an ordinary resource descriptor.
--
-- The discovery surface is identical to the CUDA script. The only difference is
-- the binder: slang.StructuredBuffer where cuda.array wraps a raw device
-- pointer.
local WIDTH, HEIGHT = 512, 256
local THREADS = 256

function rangeHistogram(inputs, outputs)
    local pc = inputs["PointCloud"]
    assert(pc.isComposite, "PointCloud must be connected to a composite AOV")

    local coordinates = pc.tensors["Coordinates"]
    local counts = pc.tensors["Counts"]
    assert(coordinates ~= nil, "composite has no Coordinates channel")
    assert(counts ~= nil, "composite has no Counts channel")
    assert(coordinates.rank == 2, "Coordinates must be [3, Nmax]")
    assert(coordinates.shape[1] == 3, "Coordinates must hold x, y and z runs")

    -- Nmax, the capacity Coordinates is allocated for. Counts says how many of
    -- those entries the sweep actually filled, but a launch script sees
    -- descriptors rather than data, so it cannot read that number. Counts is
    -- bound for the shader instead and the bound is applied there.
    local maxPoints = coordinates.shape[2]

    -- The sensor publishes scalars alongside the channels. maxPoints is the same
    -- capacity the Coordinates shape reports, so reading it is a cheap check that
    -- this port is carrying the composite the node expects.
    assert(pc.params["maxPoints"] ~= nil, "composite has no maxPoints parameter")

    -- numBins splits the chart width into bars in the kernel, so it has to divide it.
    local numBins = inputs["numBins"].value
    assert(numBins > 0 and WIDTH % numBins == 0, "numBins must divide the chart width evenly")

The scene decides which channels exist. The channels attribute on the RenderVar lists them. Ask for one and it appears; leave it out and it does not.

lidar_scene.usda, from the runnable composite AOV example#
def Scope "Render"
{
    def RenderProduct "LidarProduct"
    {
        rel camera = </World/Lidar>
        rel orderedVars = [ <PointCloud>, <Histogram> ]

        def RenderVar "PointCloud"
        {
            uniform string sourceName = "PointCloud"
            opaque omni:rtx:aov
            token[] channels = [
                "Coordinates",
                "Intensity",
                "Counts",
                "TimeOffsetNs"
            ]
        }

        def RenderVar "Histogram"
        {
            uniform string sourceName = "Histogram"
            opaque omni:rtx:aov.connect = <../RangeHistogramKernel.outputs:Histogram>
        }

        def Shader "RangeHistogramKernel" (
            references = @RangeHistogramKernel.usda@
        )
        {
            opaque inputs:PointCloud.connect = <../PointCloud.omni:rtx:aov>
        }
    }

    # A second render product, this one an ordinary camera, so the run can save a
    # picture of the scene the lidar is sweeping. It shares nothing with the
    # lidar product but the scene: one step fills both.
    def RenderProduct "SceneView"
    {
        uniform int2 resolution = (608, 342)
        rel camera = </World/ViewCamera>
        rel orderedVars = [ <LdrColor> ]

        def RenderVar "LdrColor"
        {
            uniform string sourceName = "LdrColor"
            opaque omni:rtx:aov
        }
    }
}

Note that the node’s own port is connected like any other. Nothing on the shader side says “composite”.

A RenderVar that lists no channels is not a composite. The same sensor AOV then binds as what it is underneath: a one-dimensional uchar buffer holding the output’s raw bytes, and isComposite is absent. Take that form when the node wants the bytes and will decode them itself. For channels by name, author channels.

The shader definition declares an ordinary port. Nothing in it says “composite”; that follows from what it is connected to, so the same node can be pointed at a different sensor without editing the shader.

An input therefore arrives in one of two shapes, and the script has to handle the one it gets. A camera AOV is a single resource with a shape and a dtype. A sensor composite is a table of named channels, and the individual channels are what carry shapes and dtypes. isComposite is the only way to tell them apart, and a node written for one will not work on the other without that test.

What a composite carries. inputs["X"] is a table rather than a resource:

Field

Contents

.isComposite

true. The only thing that tells a composite from an ordinary AOV.

.tensors[name]

One channel, as a resource descriptor you bind like any other buffer.

.channelNames

The channels this render var actually carries, in order.

.params[name]

A scalar the sensor published alongside the channels, such as a lidar’s maxPoints. A value, not a resource, so it is read in the script and passed to the GPU like any other value-input.

.paramNames

The parameters actually carried, in order.

.status

"empty", "partial" or "complete", as of the moment the launch script runs. Read the note below before branching on it.

.name, .doc

The render output’s own name and description, as the sensor published them.

status is not a test for “did the sensor produce data”. It reports what the render output said at the moment the launch script ran, which is before the frame’s GPU work. In the composite AOV example it reads "empty" on every frame while the channels go on carrying a full sweep of some 26,000 returns. Treat it as information about the launch script’s turn, not about the data the kernel will see, and bound the work by the count channel instead.

Iterate the name lists rather than assuming. channelNames and paramNames say what this render var carries, which is what the scene asked for and not necessarily what the sensor can produce.

Bound the work by the count channel. Point channels are allocated for the worst case, and only the first Counts[0] entries hold a return. The launch script sees descriptors rather than data, so it cannot read that number: pass Counts to the GPU and apply the bound there.

Mind the stride. Coordinates is [3, Nmax], all the x values, then all the y, then all the z. The step between runs is the allocated capacity, not the valid count.

Verify It Worked#

Count what you consumed against what the sensor produced. The composite AOV example prints returns binned: N of N, the same number twice, which only holds if the loop was bounded by Counts. How many returns a sweep produces varies; that the two agree does not. It also compares its output against a host-side computation bar by bar, and that difference is exactly zero.

When It Goes Wrong#

  • Empty output with a valid scene: the sweep returned nothing. A lidar traced across the tick needs the motion BVH enabled or Counts comes back zero. Counts is what answers this; status does not.

  • The node publishes nothing and the log names a channel: a channel or parameter was asked for under a name the composite does not carry. The message names the port, the name you used, the nearest match it found and everything available, and the node fails without publishing rather than binding something else.

  • Plausible but wrong values: the stride was taken as the valid count rather than the capacity.