Python: Projector Light Calibration

This example converts an OpenCV pinhole calibration into an RTX projector-light direction texture. It then renders the same textured projector with its default pinhole mapping and the generated calibration mapping.

The generated direction texture applies radial, tangential, thin-prism, and sensor-tilt terms from the example OpenCV calibration. Projected image: Chelsea the cat, photographed by Stefan van der Walt and dedicated to the public domain under CC0 1.0 Universal.

The generator starts from an OpenCV image size, camera matrix, and distortion vector:

IMAGE_SIZE = (3840, 2160)
CAMERA_MATRIX = np.array(
    [
        [2400.0, 0.0, 1900.0],
        [0.0, 2421.0, 1200.0],
        [0.0, 0.0, 1.0],
    ],
    dtype=np.float64,
)
# OpenCV order: k1, k2, p1, p2, k3, k4, k5, k6,
# s1, s2, s3, s4, tau_x, tau_y.
DISTORTION = np.array(
    [
        -0.32734,
        0.07468,
        -0.00046,
        0.00106,
        0.0,
        0.0,
        0.0,
        0.0,
        -0.0008216463,
        0.0,
        -0.01727441,
        0.0,
        -0.03459962,
        0.001508283,
    ],
    dtype=np.float64,
)

It decodes every square-lookup texel into an octahedral direction:

texel_centers = (np.arange(texture_size, dtype=np.float64) + 0.5) / texture_size
oct_x, oct_y = np.meshgrid(
    texel_centers * 2.0 - 1.0,
    texel_centers * 2.0 - 1.0,
)
direction_z = 1.0 - np.abs(oct_x) - np.abs(oct_y)

folded_x = (1.0 - np.abs(oct_y)) * np.where(oct_x >= 0.0, 1.0, -1.0)
folded_y = (1.0 - np.abs(oct_x)) * np.where(oct_y >= 0.0, 1.0, -1.0)
direction_x = np.where(direction_z <= 0.0, folded_x, oct_x)
direction_y = np.where(direction_z <= 0.0, folded_y, oct_y)

directions = np.stack((direction_x, direction_y, direction_z), axis=-1)
directions /= np.linalg.norm(directions, axis=-1, keepdims=True)

It then uses OpenCV to project front-facing directions into image UVs and marks directions outside the image:

flat_directions = directions.reshape(-1, 3)
uv = np.full((flat_directions.shape[0], 2), -1.0, dtype=np.float32)

# The renderer flips local -Z before indexing the lookup, so valid decoded
# directions face +Z in the coordinate system passed to OpenCV.
front_facing = flat_directions[:, 2] > 1.0e-8
image_points, _ = cv2.projectPoints(
    flat_directions[front_facing],
    np.zeros(3),
    np.zeros(3),
    camera_matrix,
    distortion,
)
projected_uv = image_points[:, 0, :] / np.asarray(image_size)

# Projector lights emit only for UVs strictly inside (0, 1). Mark every
# direction that projects outside the calibrated image with -1.
inside_image = np.all((projected_uv > 0.0) & (projected_uv < 1.0), axis=1)
front_indices = np.flatnonzero(front_facing)
uv[front_indices[inside_image]] = projected_uv[inside_image]
uv = uv.reshape(directions.shape[:2] + (2,))

The final lookup is a square OpenEXR image with float32 R and G channels:

height, width, channels = uv.shape
if width != height or channels != 2:
    raise ValueError("direction textures must be square float RG images")

path.parent.mkdir(parents=True, exist_ok=True)
float_channel = Imath.Channel(Imath.PixelType(Imath.PixelType.FLOAT))
header = OpenEXR.Header(width, height)
header["channels"] = {"R": float_channel, "G": float_channel}
header["compression"] = Imath.Compression(Imath.Compression.ZIP_COMPRESSION)

output = OpenEXR.OutputFile(str(path), header)
output.writePixels(
    {
        "R": np.ascontiguousarray(uv[:, :, 0], dtype=np.float32).tobytes(),
        "G": np.ascontiguousarray(uv[:, :, 1], dtype=np.float32).tobytes(),
    }
)
output.close()

The example generates the lookup before rendering the pinhole and calibrated outputs:

generate_direction_texture(direction_texture, args.texture_size)
pinhole = render_panel("pinhole", None, args.warmup_frames)
calibrated = render_panel(
    "opencv-calibrated", direction_texture, args.warmup_frames
)

Run the complete example with one of these commands:

cd examples/python/projector-light-calibration
uv run main.py
uv run main.py --rrd _output/projector-light-calibration.rrd
uv run main.py --no-rr --png _output/projector-light-calibration.png

Use generate_direction_texture.py directly when you only need the OpenEXR asset:

uv run generate_direction_texture.py \
  --output _output/projector-opencv-direction.exr \
  --texture-size 1024

See Rect Light for the lookup format, shader conventions, and projector-light authoring attributes.