Candlewick: a modern, cross-platform real-time renderer for robotics based on SDL3

[!WARNING] Candlewick is still under active development. Support will be limited, and the API might break unexpectedly and repeatedly.

Candlewick is a cross-platform framework for real-time visualization in robotics, based on SDL3's graphics API.

Features

Graphical features

• Shadow mapping using directional shadow maps
• Screen-space ambient occlusion (SSAO)
• WIP: Screen-space shadows (SSS)

Interactivitity

• Integration with ImGui
• Take screenshots of the main window
• (optional) Record videos from the main window using FFmpeg

Pinocchio visualizer

Candlewick visualization utilities for robotics based on Pinocchio.

You can load a Pinocchio model, its geometry model, and create a visualizer that can be used similar to the other visualizers included in pinocchio.visualize. Here's a Python example:

```python import examplerobotdata as erd import pinocchio as pin from candlewick import Visualizer, VisualizerConfig, createrecordercontext

robot = erd.load("ur10") model: pin.Model = robot.model visualmodel: pin.GeometryModel = robot.visualmodel

config = VisualizerConfig() config.width = 1280 config.height = 720 viz = Visualizer(config, model, visual_model)

q0 = pin.neutral(model) viz.addFrameViz(model.getFrameId("tool0"))

with createrecordercontext(viz, "ur10_video.mp4"): while not viz.shouldExit: viz.display(q0) ```

Candlewick visualizer runtime and client

The runtime allows users to persist a visualizer window and keep interacting with it asynchronously, even when display commands are not being submitted.

To build the runtime, users must set the BUILD_VISUALIZER_RUNTIME CMake option.

Launch the candlewick runtime:

bash candlewick-visualizer # --help

The corresponding asynchronous visualizer client is used as follows:

```python from candlewick.async_visualizer import AsyncVisualizer

viz = AsyncVisualizer(model, geom_model) viz.initViewer(hostname="127.0.0.1", port=12000) viz.loadViewerModel()

use as any other Pinocchio visualizer class

```

Check out the corresponding example.

Dependencies

Core dependencies

Candlewick depends mainly on:

• SDL3 - Graphics backend, windowing, user input
• Assimp - Loading meshes and importing 3D assets
• EnTT - Implementing the Entity-Component System
• nlohmann_json - JSON parsing (used internally for shader reflection)
• Eigen - Linear algebra
• Coal - Collision detection, geometry primitives
• magic_enum - Enum reflection utilities
• spdlog - Logging

These dependencies can be installed from Conda as follows: bash conda install -c conda-forge sdl3 eigen coal magic_enum assimp entt nlohmann_json spdlog

Additionally, candlewick depends on the following bundled dependencies:

• ImGui - immediate-mode GUI

Optional dependencies

Python bindings

• eigenpy for Python bindings.

Python runtime dependencies: numpy

Pinocchio support

Required for the candlewick::multibody classes and functions:

• Pinocchio, must be built with collision support. conda install -c conda-forge pinocchio

Python runtime dependencies: pinocchio (Python bindings)

Additional dependencies for the candlewick-visualizer runtime:

• cppzmq - C++ ZeroMQ bindings
• msgpack-cxx - MessagePack serialization

Python runtime dependencies: pyzmq msgspec

Video recording

• FFmpeg for support for recording videos from the rendered graphics. | conda install ffmpeg pkg-config | The following components are required:
  • libavformat - Container format handling
  • libavcodec - Video encoding/decoding
  • libswscale - Image scaling and format conversion

Development dependencies (tests, examples)

• GoogleTest for the tests | conda install gtest
• CLI11 for the examples and tests | conda install cli11
• With Pinocchio support: building the C++ examples requires example-robot-data. conda install -c conda-forge example-robot-data

Building Candlewick

For building the library, you will need CMake (version at least 3.26) and a C++20-compliant compiler. These can also be obtained through Conda.

In the directory where you have checked out the code, perform the following steps: ```bash

1. Create a CMake build directory

cmake -S . -B build/ \ -DCMAKEBUILDTYPE=Release \ -DBUILDPINOCCHIOVISUALIZER:BOOL=ON \ # For Pinocchio support -DBUILDVISUALIZERRUNTIME:BOOL=ON \ # Async visualizer runtime -DBUILDPYTHONINTERFACE:BOOL=ON \ # For Python bindings -DGENERATEPYTHONSTUBS:BOOL=ON \ # Python binding type stubs -GNinja \ # or -G"Unix Makefiles" to use Make -DBUILDTESTING=OFF \ # or ON not build the tests -DCMAKEINSTALLPREFIX= \ # e.g. ~/.local/, or $CONDAPREFIX

2. Move into it and build (generator-independent)

cd build/ cmake --build . -j

3. Install

cmake --install . ```

A summary of build options

| Build Option | Default | Description | Additional Dependencies | Requires | |-------------|---------|-------------|------------------------|----------| | BUILD_PYTHON_INTERFACE | ON | Build Python bindings | eigenpy | - | | BUILD_PINOCCHIO_VISUALIZER | ON | Enable Pinocchio robot visualization | pinocchio >= 3.5 | - | | BUILD_VISUALIZER_RUNTIME | OFF | Build candlewick-visualizer executable | cppzmq, msgpack-cxx | BUILD_PINOCCHIO_VISUALIZER=ON | | BUILD_WITH_FFMPEG_SUPPORT | OFF | Enable video recording capabilities | FFmpeg | - | | GENERATE_PYTHON_STUBS | OFF | Generate Python type stubs | (handled by cmake submodule) | BUILD_PYTHON_INTERFACE=ON |

Credits

Many of the design choices of this library are heavily inspired by other, more mature work in the open-source 3D graphics middleware space.

Here are some of the resources I've looked at:

• the Magnum graphics middleware (the design ideas around mesh layouts, how to load geometry primitives, and the type-erased MeshData type)
• bgfx
• Sascha Willems' Vulkan examples: https://github.com/SaschaWillems/Vulkan/

Font Credit/License

Candlewick embeds the Inter Medium font as part of its source code:

Inter-Medium.ttf, by Rasmus Andersson.
SIL OPEN FONT LICENSE Version 1.1
Copyright (c) 2016 The Inter Project Authors
https://rsms.me/inter