RealtimeSanitizer

Introduction

RealtimeSanitizer (a.k.a. RTSan) is a real-time safety testing tool for C and C++ projects. RTSan can be used to detect real-time violations, i.e. calls to methods that are not safe for use in functions with deterministic run time requirements. RTSan considers any function marked with the [[clang::nonblocking]] attribute to be a real-time function. At run-time, if RTSan detects a call to malloc, free, pthread_mutex_lock, or anything else known to have a non-deterministic execution time in a function marked [[clang::nonblocking]] it raises an error.

RTSan performs its analysis at run-time but shares the [[clang::nonblocking]] attribute with the Function Effect Analysis system, which operates at compile-time to detect potential real-time safety violations. For comprehensive detection of real-time safety issues, it is recommended to use both systems together.

The runtime slowdown introduced by RealtimeSanitizer is negligible.

How to build

Build LLVM/Clang with CMake and enable the compiler-rt runtime. An example CMake configuration that will allow for the use/testing of RealtimeSanitizer:

$ cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_RUNTIMES="compiler-rt" <path to source>/llvm

Usage

There are two requirements:

  1. The code must be compiled with the -fsanitize=realtime flag.

  2. Functions that are subject to real-time constraints must be marked with the [[clang::nonblocking]] attribute.

Typically, these attributes should be added onto the functions that are entry points for threads with real-time priority. These threads are subject to a fixed callback time, such as audio callback threads or rendering loops in video game code.

% cat example_realtime_violation.cpp
#include <vector>

void violation() [[clang::nonblocking]]{
  std::vector<float> v;
  v.resize(100);
}

int main() {
  violation();
  return 0;
}
# Compile and link
% clang++ -fsanitize=realtime example_realtime_violation.cpp

If a real-time safety violation is detected in a [[clang::nonblocking]] context, or any function invoked by that function, the program will exit with a non-zero exit code.

% clang++ -fsanitize=realtime example_realtime_violation.cpp
% ./a.out
==76290==ERROR: RealtimeSanitizer: unsafe-library-call
Intercepted call to real-time unsafe function `malloc` in real-time context!
    #0 0x000102a7b884 in malloc rtsan_interceptors.cpp:426
    #1 0x00019c326bd0 in operator new(unsigned long)+0x1c (libc++abi.dylib:arm64+0x16bd0)
    #2 0xa30d0001024f79a8  (<unknown module>)
    #3 0x0001024f794c in std::__1::__libcpp_allocate[abi:ne200000](unsigned long, unsigned long)+0x44
    #4 0x0001024f78c4 in std::__1::allocator<float>::allocate[abi:ne200000](unsigned long)+0x44
    ... snip ...
    #9 0x0001024f6868 in std::__1::vector<float, std::__1::allocator<float>>::resize(unsigned long)+0x48
    #10 0x0001024f67b4 in violation()+0x24
    #11 0x0001024f68f0 in main+0x18 (a.out:arm64+0x1000028f0)
    #12 0x00019bfe3150  (<unknown module>)
    #13 0xed5efffffffffffc  (<unknown module>)

Blocking functions

Calls to system library functions such as malloc are automatically caught by RealtimeSanitizer. Real-time programmers may also write their own blocking (real-time unsafe) functions that they wish RealtimeSanitizer to be aware of. RealtimeSanitizer will raise an error at run time if any function attributed with [[clang::blocking]] is called in a [[clang::nonblocking]] context.

$ cat example_blocking_violation.cpp
#include <atomic>
#include <thread>

std::atomic<bool> has_permission{false};

int wait_for_permission() [[clang::blocking]] {
  while (has_permission.load() == false)
    std::this_thread::yield();
  return 0;
}

int real_time_function() [[clang::nonblocking]] {
  return wait_for_permission();
}

int main() {
  return real_time_function();
}

$ clang++ -fsanitize=realtime example_blocking_violation.cpp && ./a.out
==76131==ERROR: RealtimeSanitizer: blocking-call
Call to blocking function `wait_for_permission()` in real-time context!
    #0 0x0001000c3db0 in wait_for_permission()+0x10 (a.out:arm64+0x100003db0)
    #1 0x0001000c3e3c in real_time_function()+0x10 (a.out:arm64+0x100003e3c)
    #2 0x0001000c3e68 in main+0x10 (a.out:arm64+0x100003e68)
    #3 0x00019bfe3150  (<unknown module>)
    #4 0x5a27fffffffffffc  (<unknown module>)

Run-time flags

RealtimeSanitizer supports a number of run-time flags, which can be specified in the RTSAN_OPTIONS environment variable:

% RTSAN_OPTIONS=option_1=true:path_option_2="/some/file.txt" ./a.out
...

Or at compile-time by providing the symbol __rtsan_default_options:

__attribute__((__visibility__("default")))
extern "C" const char *__rtsan_default_options() {
  return "symbolize=false:abort_on_error=0:log_to_syslog=0";
}

You can see all sanitizer options (some of which are unsupported) by using the help flag:

% RTSAN_OPTIONS=help=true ./a.out

A partial list of flags RealtimeSanitizer respects:

Run-time Flags

Flag name

Default value

Type

Short description

halt_on_error

true

boolean

Exit after first reported error. If false (continue after a detected error), deduplicates error stacks so errors appear only once.

print_stats_on_exit

false

boolean

Print stats on exit. Includes total and unique errors.

color

"auto"

string

Colorize reports: (always|never|auto).

fast_unwind_on_fatal

false

boolean

If available, use the fast frame-pointer-based unwinder on detected errors. If true, ensure the code under test has been compiled with frame pointers with -fno-omit-frame-pointers or similar.

abort_on_error

OS dependent

boolean

If true, the tool calls abort() instead of _exit() after printing the error report. On some OSes (OSX, for exmple) this is beneficial because a better stack trace is emitted on crash.

symbolize

true

boolean

If set, use the symbolizer to turn virtual addresses to file/line locations. If false, can greatly speed up the error reporting.

suppressions

“”

path

If set to a valid suppressions file, will suppress issue reporting. See details in “Disabling”, below.

Some issues with flags can be debugged using the verbosity=$NUM flag:

% RTSAN_OPTIONS=verbosity=1:misspelled_flag=true ./a.out
WARNING: found 1 unrecognized flag(s):
misspelled_flag
...

Disabling and suppressing

There are multiple ways to disable error reporting when using RealtimeSanitizer.

In general, ScopedDisabler should be preferred, as it is the most performant.

Suppression methods

Method

Specified at?

Scope

Run-time cost

Description

ScopedDisabler

Compile-time

Stack

Very low

Violations are ignored for the lifetime of the ScopedDisabler object.

function-name-matches suppression

Run-time

Single function

Medium

Suppresses intercepted and [[clang::blocking]] function calls by name.

call-stack-contains suppression

Run-time

Stack

High

Suppresses any stack trace contaning the specified pattern.

ScopedDisabler

At compile time, RealtimeSanitizer may be disabled using __rtsan::ScopedDisabler. RTSan ignores any errors originating within the ScopedDisabler instance variable scope.

#include <sanitizer/rtsan_interface.h>

void process(const std::vector<float>& buffer) [[clang::nonblocking]] {
    {
        __rtsan::ScopedDisabler d;
        ...
    }
}

If RealtimeSanitizer is not enabled at compile time (i.e., the code is not compiled with the -fsanitize=realtime flag), the ScopedDisabler is compiled as a no-op.

In C, you can use the __rtsan_disable() and rtsan_enable() functions to manually disable and re-enable RealtimeSanitizer checks.

#include <sanitizer/rtsan_interface.h>

int process(const float* buffer) [[clang::nonblocking]]
{
    {
        __rtsan_disable();

        ...

        __rtsan_enable();
    }
}

Each call to __rtsan_disable() must be paired with a subsequent call to __rtsan_enable() to restore normal sanitizer functionality. If a corresponding rtsan_enable() call is not made, the behavior is undefined.

Suppression file

At run-time, suppressions may be specified using a suppressions file passed in RTSAN_OPTIONS. Run-time suppression may be useful if the source cannot be changed.

> cat suppressions.supp
call-stack-contains:MallocViolation
call-stack-contains:std::*vector
function-name-matches:free
function-name-matches:CustomMarkedBlocking*
> RTSAN_OPTIONS="suppressions=suppressions.supp" ./a.out
...

Suppressions specified in this file are one of two flavors.

function-name-matches suppresses reporting of any intercepted library call, or function marked [[clang::blocking]] by name. If, for instance, you know that malloc is real-time safe on your system, you can disable the check for it via function-name-matches:malloc.

call-stack-contains suppresses reporting of errors in any stack that contains a string matching the pattern specified. For example, suppressing error reporting of any non-real-time-safe behavior in std::vector may be specified call-stack-contains:std::*vector. You must include symbols in your build for this method to be effective, unsymbolicated stack traces cannot be matched. call-stack-contains has the highest run-time cost of any method of suppression.

Patterns may be exact matches or are “regex-light” patterns, containing special characters such as ^$*.

The number of potential errors suppressed via this method may be seen on exit when using the print_stats_on_exit flag.

Compile-time sanitizer detection

Clang provides the pre-processor macro __has_feature which may be used to detect if RealtimeSanitizer is enabled at compile-time.

#if defined(__has_feature) && __has_feature(realtime_sanitizer)
...
#endif