About

This code has been produced during while writing my Ph.D. (Dr.-Ing.) thesis at the institut of automatic control, RWTH Aachen University. If you find it helpful for your research please cite this:

T. Redick, „Bayesian inference for CAD-based pose estimation on depth images for robotic manipulation“, RWTH Aachen University, 2024. doi: 10.18154/RWTH-2024-04533.

SciGL.jl

Port of scigl_render to julia.

The primary goal is to enable efficient rendering of multiple scenes and transferring the images to a compute device (CPU or CUDA) for Scientific calculations.

examples

To run the examples, you need to activate the examples environment! I do not want to contaminate the main environment with the Images.jl packages.

Design Choices

I try to incorporate existing Julia packages wherever possible. The next section contains a list and use cases of the packages.

For performance, I use Persistent mapping and glGetTextureSubImage to transfer data between the GPU and CPU. These functions require OpenGL 4.5, but I set the minimum version to 4.1 to support WSL2. It seems like the drivers support the functions nevertheless.

Package Dependencies

• CoordinateTransformations: Representing and chaining transformations like rotations, translations, and perspective transformations. Rotations are handled by the equally named package.
• GLAbstractions: Takes some of the low-level OpenGL pain away. Manages the context, compiles shaders, and handles the buffers.
• ModernGL: Used by GLAbstractions to interface with the OpenGL driver.
• GLFW: OpenGL context handling.
• MeshIO: Load mesh files like .obj, .ply, and .stl. It uses the FileIO interface, so this package is also included.

Reexport

For convenience commonly used symbols are reexported:

• ColorTypes: AbstractRGBA, RGB, RGBA, Gray, red, blue, green, alpha
• CoordinateTransformations: Translation
• GLAbstraction
• GLFW
• Rotations: all symbols

HPC on Headless Server with VirtualGL

Install TurboVNC on the server which will be used to instantiate a render context without an attached display. There are also good instructions on the GLVisualize github.

Use the following script to launch julia with TurboVNC and NVIDIA as OpenGL vendor:

#!/bin/sh
DIR="$(cd "$(dirname "$0")" && pwd)"
JULIA=$DIR/julia
# VSCode reads the ouputs of julia -e using Pkg; println.(Pkg.depots())
/opt/TurboVNC/bin/vncserver :6
DISPLAY=:6 __GLX_VENDOR_LIBRARY_NAME=nvidia $JULIA "$@"%

Make the file executable via chmod +x julia_nvidia.sh

Moreover, you can trick vscode that this file is the julia executable via the setting: "julia.executablePath": "/path/to/julia/bin/julia_nvidia.sh"

Tipp: If you get an unknown CUDA Error (999) during OpenGL interop, you probably render to the integrated device instead of the NVIDIA

Devcontainer

Recommended: Install the vscode Remote - Containers plugin and load the devcontainer. Alternatively install julia locally, activate and instantiate the SciGL.jl environment.

Switching GPUs

On Ubuntu 20.04 and other recent Linux distros, NVIDIA allows for on-demand switching between dedicated and integrated graphics. Rendering on the NVIDIA GPU is mandatory if OpenGL-CUDA interop is required. You can run julia with the environment variables set:

# NVIDIA GPU
__NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia julia script.
# Integrated graphics
__GLX_VENDOR_LIBRARY_NAME=mesa julia script.jl

You can verify whether the NVIDIA GPU is used in a Julia program by the following command on the host:

nvidia-smi | grep julia

Windows Subsystem for Linux (WSL2)

Microsoft added the Direct3D backend to the mesa driver in WSL2. One drawback is that the driver version is not the latest, e.g., 4.2 for NVIDIA and 4.1 for Intel at the time of writing. You can switch between the GPUs by setting the following environment variable:

MESA_D3D12_DEFAULT_ADAPTER_NAME=NVIDIA julia
MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel julia

Debug in vscode

Later versions of the Julia extension seem to have fixed the issue.

The vscode julia debugger crashes when loading the native OpenGL functions. Enabling the Compiled Mode as described here seems to be a workaround.

IJupyter

Based on Jupyter, IJupyter can be used for explorative coding. To use IJupyter, you have two choices:

• Create a .ipynb file and open it in vscode. The Jupyter extension will automatically launch the Jupyter server.
• Launch jupyter lab --allow-root from an integrated terminal. Hold Alt + Click the link to open the notebook.