Index of every page on this site: llms.txt
All skills
google-deepmind avatar

/rendering

@eb1ff5e
by Google DeepMindgoogle-deepmind/mujoco18k stars
2,221
Run it yourself

MuJoCo offscreen image rendering using mujoco.Renderer in Python and the MuJoCo C/C++ rendering pipeline (mjrContext, mjvScene, mjr_render, mjr_readPixels): RGB, metric depth maps, segmentation masks (mjtObj), MJCF offwidth/offheight framebuffer sizing, MUJOCO_GL (osmesa, egl), MuJoCo cameras (fixed, tracking, free, MjvCamera), and mjvOption flags/geomgroup. Use only for headless camera rendering, dataset recording, and visual observations in MuJoCo simulations. Don't use for generic OpenGL/EGL/OSMesa offscreen rendering, glReadPixels depth/segmentation post-processing, or synthetic dataset pipelines that do not use the `mujoco` library, or for interactive viewers/Studio (mujoco-studio) or ImGui controls (mujoco-gui).

Requires /studio
  • 1 file
  • 12.4 KB
  • Updated 4 days ago
  • GitHub
Use this Skill: https://skilld.dev/gh/google-deepmind/mujoco/rendering

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈185 tokens for metadata: the name and description. ≈3k when used: this file.

Description uses 9.3% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈746 characters, or 9.3%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

MuJoCo Rendering: Cameras, Contexts & Offscreen Buffers

[!TIP]

Official MuJoCo documentation (doc/*.rst): Read ../../programming/rendering.rst (.. _Rendering:, .. _FilamentRendering:, .. _ClassicRendering:) and ../../programming/visualization.rst (.. _Visualization:) directly when in a MuJoCo checkout; for large reference files like ../../python.rst (.. _PyRender:), ../../XMLreference.rst, and ../../APIreference/functions.rst, grep the literal .. _Label: anchor. Outside a checkout, the same doc/*.rst tree ships offline in pip install mujoco at pathlib.Path(mujoco.__file__).parent / 'doc'; otherwise fetch https://mujoco.readthedocs.io/en/latest/<path>.html (or /en/stable/ for released pages; functions.rst renders inside APIreference/APIfunctions.html, and Sphinx anchors lowercase the label, e.g. .. _PyRender: → #pyrender) and browse repo paths (../../../) under https://github.com/google-deepmind/mujoco/tree/main/.

Related skills:

  • mujoco-python — Core physics simulation, stepping, and state management.
  • mujoco-studio — Interactive simulation, Studio viewer apps, and plugin architecture.
  • mujoco-gui — Dear ImGui UI design and custom controls.

1. Overview: Rendering Backends & Workflows

MuJoCo provides two built-in 3D rendering backends (compared side-by-side in programming/rendering.rst, .. _Rendering:):

  • Classic OpenGL (mujoco.Renderer / C mjr): Immediate-mode OpenGL 1.5 renderer (mjv_updateScene -> mjr_render -> mjr_readPixels), now in maintenance mode. Bundled out of the box in pip install mujoco, making mujoco.Renderer the simplest single-class helper for headless RGB, depth, and segmentation capture. See programming/rendering.rst (.. _ClassicRendering:) and python.rst (.. _PyClassicRender:).
  • Filament PBR (mujoco.mjrf & mujoco.rendering.filament / C mjrf): Retained-mode, asynchronous physically-based renderer supporting OpenGL, Vulkan, Metal, and WebGL (owns its own headless or windowed GPU context and arbitrary-size mjrfRenderTarget buffers). Recommended for PBR materials, image-based/photometric lighting, and new C++/studio applications; see §7 and programming/rendering.rst (.. _FilamentRendering:).
  • Interactive Viewers (mujoco.studio / mujoco.viewer): Desktop or web-streamed interactive viewers; use mujoco-studio instead.

2. Headless Contexts on Linux: MUJOCO_GL

On headless Linux systems (servers, containers), configure the OpenGL context provider via the MUJOCO_GL environment variable before importing mujoco:

import os
os.environ['MUJOCO_GL'] = 'egl'      # Hardware-accelerated offscreen on NVIDIA GPUs
# os.environ['MUJOCO_GL'] = 'osmesa' # Portable CPU software rasterization

import mujoco  # Must import AFTER setting MUJOCO_GL

Or from the command line:

MUJOCO_GL=egl python3 my_script.py

3. The Python Renderer Class (RGB, Depth & Segmentation)

mujoco.Renderer manages GL context creation, scene geometry synchronization, and buffer readback. Always use it as a context manager:

import mujoco
import numpy as np

model = mujoco.MjModel.from_xml_path('scene.xml')
data = mujoco.MjData(model)

with mujoco.Renderer(model, height=480, width=640) as renderer:
  mujoco.mj_forward(model, data)

  # 1. RGB Rendering
  renderer.update_scene(data, camera='overhead_cam')
  rgb = renderer.render()  # Shape: (480, 640, 3), dtype: uint8

  # 2. Metric Depth Rendering (distance in meters)
  renderer.enable_depth_rendering()
  renderer.update_scene(data, camera='overhead_cam')
  depth = renderer.render()  # Shape: (480, 640), dtype: float32
  renderer.disable_depth_rendering()

  # 3. Semantic Segmentation Rendering
  renderer.enable_segmentation_rendering()
  renderer.update_scene(data, camera='overhead_cam')
  seg = renderer.render()  # Shape: (480, 640, 2), dtype: int32
  # seg[:, :, 0] = object ID (-1 for background)
  # seg[:, :, 1] = object type (mjtObj enum, e.g. mjOBJ_GEOM)
  renderer.disable_segmentation_rendering()

Rendering Inside Simulation Loops

Creating a Renderer allocates GPU framebuffers. In RL or evaluation loops, create the renderer once outside the loop:

with mujoco.Renderer(model, height=84, width=84) as renderer:
  for step in range(1000):
    mujoco.mj_step(model, data)
    if step % render_interval == 0:
      renderer.update_scene(data, camera='agent_cam')
      # Call .copy() because the renderer's internal buffer is reused
      frame = renderer.render().copy()

4. C++ Offscreen Rendering Pipeline (mjr & mjv)

In C++, offscreen rendering is performed using the mjvScene and mjrContext APIs:

#include <mujoco/mujoco.h>

// 1. Initialize visualization structures
mjvCamera cam;
mjv_defaultCamera(&cam);
cam.type = mjCAMERA_FIXED;
cam.fixedcamid = mj_name2id(m, mjOBJ_CAMERA, "overhead_cam");

mjvOption opt;
mjv_defaultOption(&opt);

mjvScene scn;
mjv_makeScene(m, &scn, 2000);  // Buffer up to 2000 geoms

// 2. Initialize offscreen rendering context
mjrContext con;
mjr_defaultContext(&con);
mjr_makeContext(m, &con, mjFONTSCALE_100);

// 3. Render frame
mjrRect viewport = {0, 0, 640, 480};
mjv_updateScene(m, d, &opt, nullptr, &cam, mjCAT_ALL, &scn);
mjr_render(viewport, &scn, &con);

// 4. Read pixels into user buffers
unsigned char rgb[640 * 480 * 3];
float depth[640 * 480];
mjr_readPixels(rgb, depth, viewport, &con);

// 5. Clean up
mjr_freeContext(&con);
mjv_freeScene(&scn);

5. Camera Definition & Placement

MuJoCo cameras look down their negative Z-axis:

  • -Z: Viewing forward direction
  • +X: Right
  • +Y: Up

Defining Cameras in MjSpec

# Fixed camera pointing down at ground plane
cam = spec.worldbody.add_camera(
    name='overhead',
    pos=[0, 0, 3.0],
    quat=[0.707, 0.707, 0, 0],  # 90 deg rotation around X
    fovy=60,
)

# Camera attached to robot wrist
wrist_cam = wrist_body.add_camera(
    name='wrist_cam',
    pos=[0.05, 0, 0],
    xyaxes=[0, -1, 0, 0, 0, -1],
    fovy=75,
)

Free Camera (MjvCamera)

For custom interactive or orbital viewpoints:

cam = mujoco.MjvCamera()
cam.type = mujoco.mjtCamera.mjCAMERA_FREE
cam.lookat[:] = [0, 0, 0.5]
cam.distance = 2.5
cam.azimuth = 90.0
cam.elevation = -20.0

renderer.update_scene(data, camera=cam)

6. Framebuffers & Scene Options

Offscreen Framebuffer Resolution

The offscreen rendering resolution cannot exceed the dimensions pre-allocated in the model. Adjust the buffer limit in the spec before compilation:

spec.visual.global_.offwidth = 1920
spec.visual.global_.offheight = 1080
model = spec.compile()

The Single <visual> Block Rule

MuJoCo uses only the last <visual> block defined in an XML file. Multiple visual blocks will silently overwrite earlier settings:

<!-- ❌ WRONG — second visual block silently resets offwidth/offheight to default -->
<visual><global offwidth="1920" offheight="1080"/></visual>
<visual><headlight ambient="0.3 0.3 0.3"/></visual>

<!-- ✅ RIGHT — consolidate into one visual block -->
<visual>
  <global offwidth="1920" offheight="1080"/>
  <headlight ambient="0.3 0.3 0.3"/>
</visual>

Geom Group Visibility & Visualization Flags

Control rendering layers via MjvOption:

opt = mujoco.MjvOption()
opt.geomgroup[:] = False   # Hide all groups
opt.geomgroup[0] = True    # Show primary geometry (default group)
opt.geomgroup[1] = True    # Show secondary visuals

# Toggle visual debugging flags
opt.flags[mujoco.mjtVisFlag.mjVIS_CONTACTFORCE] = True
opt.flags[mujoco.mjtVisFlag.mjVIS_TRANSPARENT] = True

renderer.update_scene(data, camera='overhead', scene_option=opt)

7. Filament Physically-Based Rendering (mjrf)

MuJoCo's Filament backend (mjrfilament.h in C/C++, from mujoco import mjrf and mujoco.rendering.filament in Python) provides retained-mode PBR with OpenGL, Vulkan, Metal, and WebGL support, arbitrary mjrfRenderTarget sizes (independent of <global offwidth offheight/>), and asynchronous mjrfReadRequest pixel readback:

  • Python (ModelScene + Renderer): Create an mjrf.Context, bind model lights and renderables once via mujoco.rendering.filament.model_scene.ModelScene, update in place with ms.update(data), and render into a NumPy target buffer via mujoco.rendering.filament.renderer.Renderer — see the complete example in python.rst (.. _PyFilamentRender:).
  • C/C++ & CMake: Build with -DMUJOCO_USE_FILAMENT=ON (or -DMUJOCO_USE_FILAMENT_MJR_COMPAT=ON for drop-in mjr_ compatibility) and consult programming/rendering.rst (.. _reProgramming:, .. _reSupport:).
  • Photometric light-intensity gotcha: If any <light> has positive intensity, Filament switches the entire scene from classic headlight + default environment lighting to photometric mode (turning off the headlight and darkening zero-intensity lights). For PBR materials, textures, and lighting authoring, read programming/rendering.rst (.. _reAuthoring:, .. _reMaterials:, .. _reLighting:) and inspect the reference PBR/IBL test scenes in ../../../test/render/testdata/ (pbr.xml, lighting.xml, emissive_bloom.xml, reflect_pbr.xml, ibl.ktx) and ../../../model/mug/mug.xml.

8. Common Gotchas & Best Practices

  1. GL context leaks: Always close mujoco.Renderer or use a with block. Failing to free renderers in loops exhausts GL contexts.
  2. Destruction order: In C++, mjr_freeContext must be called before the underlying OpenGL context is destroyed (see programming/rendering.rst, .. _reContext:).
  3. Buffer re-use: renderer.render() returns an internal buffer view. Call .copy() if storing frames in a list.
  4. Camera lookup by name: Pass camera='camera_name' or the integer ID. Avoid passing raw string IDs that are not present in the model.

9. Key References

Documentation

Source Code & Model Examples

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No third-party reports yet.

Signed by skilld at eb1ff5e. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 4 days ago
metadata
{
  "publisher": "google"
}

README badge

README badge for google-deepmind/mujoco/rendering