Shape an Image Like a Camera Sensor#

Goal. Add a mosaic pattern, sensor noise, bit-depth compression and optional image signal processor (ISP) processing to a rendered image. Apply the model’s schema, request its input and a primary output, then inspect the output shapes. Optional diagnostic outputs expose intermediate stages.

Before you start. Have a camera that renders an image and configure its input. See Camera generic core model for the simulated stages, USD Attributes for controls, and Outputs for output names and formats. The Compand to a RAW Format and Run the ISP guide verifies ISP processing.

Apply the schema#

Apply OmniSensorGenericCameraCoreAPI to the Camera prim you already render through. What that implies, that the sensor is the camera itself and its intrinsics keep governing the view, is on USD Attributes. How an API schema is applied to a prim, in USDA or through the usd-core Python API, is described in Sensor Configuration.

This task leaves every attribute of the schema at its default. One default has a scheduling consequence: the schema sets omni:sensor:frameRate to (30, 1), so the camera is scheduled at 30 Hz rather than rendering on every step; author (0, 1) to leave it unscheduled, as Sensor Scheduling explains. The model’s own attributes live in the omni:sensor:core: namespace, spelled with a lower-case core; USD Attributes explains that namespace and what each attribute group does. The sibling how-tos are where you change them: Author a Color Filter Array for the mosaic, Compand to a RAW Format and Run the ISP for the transport bit depth and the ISP program, and Configure the Input Image for the product-side choices that decide what the model sees.

Request the input and a primary output#

The model publishes its results as AOVs on the RenderProduct bound to the camera, requested like any other output: a RenderVar whose sourceName names them (Output Configuration introduces RenderProduct and RenderVar). On that product, request two things:

  • HdrColor, the linear high-dynamic-range render that is the pipeline’s only input, requested on the same product as the model’s outputs. Configure the Input Image states that requirement and what happens without it, the input precision and the render mode behind it.

  • One primary output: OmniCameraSensorPostIsp for the full pipeline, or OmniCameraSensorPreIsp to stop before the ISP and read the sensor’s RAW codes. The linked section of Outputs states which primary output builds how much of the pipeline and what happens when both or neither are requested.

OmniCameraSensorPostIsp carries a processed image only once you have authored an ISP program (Supply the ISP program); until then, use OmniCameraSensorPreIsp as the primary output, which needs none.

The fragment below applies the schema to a camera and binds a RenderProduct to it that requests HdrColor and OmniCameraSensorPostIsp. Swap the primary output’s name for OmniCameraSensorPreIsp to stop before the ISP.

def Camera "SensorCamera" (
    prepend apiSchemas = ["OmniSensorGenericCameraCoreAPI"]
)
{
}

def Scope "Render"
{
    def RenderProduct "SensorCamera"
    {
        int2 resolution = (1920, 1080)
        rel camera = </SensorCamera>
        rel orderedVars = [<HdrColor>, <OmniCameraSensorPostIsp>]

        def RenderVar "HdrColor"
        {
            string sourceName = "HdrColor"
        }

        def RenderVar "OmniCameraSensorPostIsp"
        {
            string sourceName = "OmniCameraSensorPostIsp"
        }
    }
}

In the usd-core Python API the same product is authored as the RenderProduct examples on Output Configuration show, one RenderVar per name listed in orderedVars, with the schema prepended to the camera prim as Sensor Configuration shows.

Add diagnostic taps#

The model has seven output names. Two are the primary outputs of the previous step; the other five are diagnostic taps on the intermediate stages. Add any of them to the same product the same way, one RenderVar per name. Which tap shows which stage, and the rule that a tap behind a stage that was not built publishes nothing, are in the linked section of Outputs.

Requesting every name at once, as the model’s example scene does, lets you follow one frame through the whole pipeline. The product then lists all eight names, with a RenderVar for each written as in the previous step:

def RenderProduct "SensorCamera"
{
    int2 resolution = (1920, 1080)
    rel camera = </SensorCamera>
    rel orderedVars = [
        <HdrColor>,
        <OmniCameraSensorInputSource>,
        <OmniCameraSensorPostColorCorrection>,
        <OmniCameraSensorPostCfa>,
        <OmniCameraSensorPostNoise>,
        <OmniCameraSensorPreIsp>,
        <OmniCameraSensorPostYuvIsp>,
        <OmniCameraSensorPostIsp>,
    ]

    # one RenderVar per name, as in the previous step
}
Four panels of one Cornell box frame: the color HdrColor input; two gray single-component images that look alike, the second carrying a small inset of amplified random grain; and the color processed image.

One frame of the Camera generic core model example scene at four outputs, left to right HdrColor (the rendered input, scaled for display), OmniCameraSensorPostCfa (the single-component mosaic, shown gray), OmniCameraSensorPostNoise (the same mosaic after the noise stage, on the same gray scale; its inset shows the difference to the previous tap, amplified, since the grain is a fraction of a percent of the signal at the example’s settings) and OmniCameraSensorPostIsp (the processed image).#

Verify it worked#

Request OmniCameraSensorPostCfa and OmniCameraSensorPostNoise beside your primary output, then read the outputs back as you would any output: Sensor Outputs covers mapping an output, Decoding camera outputs the image outputs in particular. Check three things, in this order.

  1. The shape of OmniCameraSensorPostCfa, before you look at any value. It is a single-component image of sensor codes where HdrColor is four-component RGBA: a two-dimensional array, or one whose last dimension is 1. If what you map has four components, the color filter array stage did not run, and nothing after it did either.

  2. That the noise stage changed something. Compare OmniCameraSensorPostCfa with OmniCameraSensorPostNoise from the same frame over a lit region; compare within one frame rather than across runs, because the noise pattern repeats from run to run for the same scene and codes. Noise is added in whole sensor codes, so a pixel whose draw falls below one code keeps its value; over a lit region that is rare, and the two taps differ at almost every pixel. If they are identical everywhere, no noise was added at all. Photon shot noise has no attribute and is always on, so identical taps mean the codes are too small for it to register (check colorFilterArrayMaximalValue, which scales them) and the dark-shot noise was authored away (noiseDarkShotNoiseSigma times noiseDarkShotNoiseGain is zero or well below one code).

  3. The sequence of format changes. Absolute pixel values are not a portable check: they depend on the render mode, the sample count and the scene. What is reproducible is the sequence of element formats along the pipeline, one component after the color filter array and four again after the ISP, which the stage table on Outputs lists. The formats do not show that the ISP ran: with no program authored they are the same, and the post-ISP outputs carry no processed image. The verify step of Compand to a RAW Format and Run the ISP checks the ISP itself.

When it goes wrong#

When nothing arrives at all, or an error names the product, the Symptom Index lists the configuration mistakes behind it. Each entry below names the mechanism and the page that states it in full.

  • A camera pipeline is not enabled. Naming one of the seven on a product starts the pipeline only when the product’s camera carries OmniSensorGenericCameraCoreAPI and no other sensor schema; a camera with none, or with two, is rejected and the product delivers none of the outputs. Refer to What enables the pipeline and the entry under Missing output.

  • A camera product is missing its input or its primary output. HdrColor must sit on the same product and one of the two primary outputs has to be present as well; if either is missing, the product is skipped and an error is logged, and a diagnostic tap on its own does not build a pipeline for the tap to sit on. Refer to Configure the Input Image, the primary outputs and the entry under Missing output.

  • The camera does not render on every step. The schema sets omni:sensor:frameRate to (30, 1), so the camera is scheduled at 30 Hz rather than rendering on every step; author (0, 1) to leave it unscheduled. Refer to Sensor Scheduling.

  • The ISP taps produce nothing, or nothing processed, while the earlier taps work. Either OmniCameraSensorPostIsp was not requested, so only the pre-ISP pipeline was built and the ISP-side taps have no stage behind them, or no ISP program is authored in ispSmodelCameraProgram, the schema default, and the post-ISP outputs are published without a processed image. Refer to Supply the ISP program, the primary outputs and the entry under Missing output.

  • A camera image is black or saturated although the mosaic is there. colorFilterArrayMaximalValue is a multiplier, not a clamp, so check it against the range the image actually occupies. Refer to Author a Color Filter Array and the entry under Values look plausible but are wrong.