RealtimeSanitizer¶

Introduction ¶

RealtimeSanitizer (a.k.a. RTSan) is a real-time safety testing tool for C and C++ projects. RTSan can be used to detect real-time violations, i.e. calls to methods that are not safe for use in functions with deterministic run time requirements. RTSan considers any function marked with the [[clang::nonblocking]] attribute to be a real-time function. At run-time, if RTSan detects a call to malloc, free, pthread_mutex_lock, or anything else known to have a non-deterministic execution time in a function marked [[clang::nonblocking]] it raises an error.

RTSan performs its analysis at run-time but shares the [[clang::nonblocking]] attribute with the Function Effect Analysis system, which operates at compile-time to detect potential real-time safety violations. For comprehensive detection of real-time safety issues, it is recommended to use both systems together.

The runtime slowdown introduced by RealtimeSanitizer is negligible.

How to build ¶

Build LLVM/Clang with CMake and enable the compiler-rt runtime. An example CMake configuration that will allow for the use/testing of RealtimeSanitizer:

$ cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_RUNTIMES="compiler-rt" <path to source>/llvm

Usage ¶

There are two requirements:

The code must be compiled with the -fsanitize=realtime flag.
Functions that are subject to real-time constraints must be marked with the [[clang::nonblocking]] attribute.

Typically, these attributes should be added onto the functions that are entry points for threads with real-time priority. These threads are subject to a fixed callback time, such as audio callback threads or rendering loops in video game code.

% cat example_realtime_violation.cpp
#include <vector>

void violation() [[clang::nonblocking]]{
  std::vector<float> v;
  v.resize(100);
}

int main() {
  violation();
  return 0;
}
# Compile and link
% clang++ -fsanitize=realtime example_realtime_violation.cpp

If a real-time safety violation is detected in a [[clang::nonblocking]] context, or any function invoked by that function, the program will exit with a non-zero exit code.

% clang++ -fsanitize=realtime example_realtime_violation.cpp
% ./a.out
==76290==ERROR: RealtimeSanitizer: unsafe-library-call
Intercepted call to real-time unsafe function `malloc` in real-time context!
    #0 0x000102a7b884 in malloc rtsan_interceptors.cpp:426
    #1 0x00019c326bd0 in operator new(unsigned long)+0x1c (libc++abi.dylib:arm64+0x16bd0)
    #2 0xa30d0001024f79a8  (<unknown module>)
    #3 0x0001024f794c in std::__1::__libcpp_allocate[abi:ne200000](unsigned long, unsigned long)+0x44
    #4 0x0001024f78c4 in std::__1::allocator<float>::allocate[abi:ne200000](unsigned long)+0x44
    ... snip ...
    #9 0x0001024f6868 in std::__1::vector<float, std::__1::allocator<float>>::resize(unsigned long)+0x48
    #10 0x0001024f67b4 in violation()+0x24
    #11 0x0001024f68f0 in main+0x18 (a.out:arm64+0x1000028f0)
    #12 0x00019bfe3150  (<unknown module>)
    #13 0xed5efffffffffffc  (<unknown module>)

Blocking functions ¶

Calls to system library functions such as malloc are automatically caught by RealtimeSanitizer. Real-time programmers may also write their own blocking (real-time unsafe) functions that they wish RealtimeSanitizer to be aware of. RealtimeSanitizer will raise an error at run time if any function attributed with [[clang::blocking]] is called in a [[clang::nonblocking]] context.

$ cat example_blocking_violation.cpp
#include <atomic>
#include <thread>

std::atomic<bool> has_permission{false};

int wait_for_permission() [[clang::blocking]] {
  while (has_permission.load() == false)
    std::this_thread::yield();
  return 0;
}

int real_time_function() [[clang::nonblocking]] {
  return wait_for_permission();
}

int main() {
  return real_time_function();
}

$ clang++ -fsanitize=realtime example_blocking_violation.cpp && ./a.out
==76131==ERROR: RealtimeSanitizer: blocking-call
Call to blocking function `wait_for_permission()` in real-time context!
    #0 0x0001000c3db0 in wait_for_permission()+0x10 (a.out:arm64+0x100003db0)
    #1 0x0001000c3e3c in real_time_function()+0x10 (a.out:arm64+0x100003e3c)
    #2 0x0001000c3e68 in main+0x10 (a.out:arm64+0x100003e68)
    #3 0x00019bfe3150  (<unknown module>)
    #4 0x5a27fffffffffffc  (<unknown module>)

Run-time flags ¶

RealtimeSanitizer supports a number of run-time flags, which can be specified in the RTSAN_OPTIONS environment variable:

% RTSAN_OPTIONS=option_1=true:path_option_2="/some/file.txt" ./a.out
...

Or at compile-time by providing the symbol __rtsan_default_options:

__attribute__((__visibility__("default")))
extern "C" const char *__rtsan_default_options() {
  return "symbolize=false:abort_on_error=0:log_to_syslog=0";
}

You can see all sanitizer options (some of which are unsupported) by using the help flag:

% RTSAN_OPTIONS=help=true ./a.out

A partial list of flags RealtimeSanitizer respects:

Run-time Flags¶
Flag name	Default value	Type	Short description
`halt_on_error`	`true`	boolean	Exit after first reported error.
`suppress_equal_stacks`	`true`	boolean	If true, suppress duplicate reports (i.e. only print each unique error once). Only particularly useful when `halt_on_error=false`.
`print_stats_on_exit`	`false`	boolean	Print stats on exit. Includes total and unique errors.
`color`	`"auto"`	string	Colorize reports: (always\|never\|auto).
`fast_unwind_on_fatal`	`false`	boolean	If available, use the fast frame-pointer-based unwinder on detected errors. If true, ensure the code under test has been compiled with frame pointers with `-fno-omit-frame-pointers` or similar.
`abort_on_error`	OS dependent	boolean	If true, the tool calls `abort()` instead of `_exit()` after printing the error report. On some OSes (MacOS, for exmple) this is beneficial because a better stack trace is emitted on crash.
`symbolize`	`true`	boolean	If set, use the symbolizer to turn virtual addresses to file/line locations. If false, can greatly speed up the error reporting.
`suppressions`	`""`	path	If set to a valid suppressions file, will suppress issue reporting. See details in Disabling and Suppressing.
`verify_interceptors`	`true`	boolean	If true, verifies interceptors are working at initialization. The program will abort with error `==ERROR: Interceptors are not working. This may be because RealtimeSanitizer is loaded too late (e.g. via dlopen)` if an issue is detected.

Some issues with flags can be debugged using the verbosity=$NUM flag:

% RTSAN_OPTIONS=verbosity=1:misspelled_flag=true ./a.out
WARNING: found 1 unrecognized flag(s):
misspelled_flag
...

Additional customization ¶

In addition to __rtsan_default_options outlined above, you can provide definitions of other functions that affect how RTSan operates.

To be notified on every error reported by RTsan, provide a definition of __sanitizer_report_error_summary.

extern "C" void __sanitizer_report_error_summary(const char *error_summary) {
   fprintf(stderr, "%s %s\n", "In custom handler! ", error_summary);
   /* do other custom things */
}

The error summary will be of the form:

SUMMARY: RealtimeSanitizer: unsafe-library-call main.cpp:8 in process(std::__1::vector<int, std::__1::allocator<int>>&)

To register a callback which will be invoked before a RTSan kills the process:

extern "C" void __sanitizer_set_death_callback(void (*callback)(void));

void custom_on_die_callback() {
  fprintf(stderr, "In custom handler!")
  /* do other custom things */
}

int main()
{
  __sanitizer_set_death_callback(custom_on_die_callback);
  ...
}

Disabling and suppressing ¶

There are multiple ways to disable error reporting when using RealtimeSanitizer.

In general, ScopedDisabler should be preferred, as it is the most performant.

Suppression methods¶
Method	Specified at?	Scope	Run-time cost	Description
`ScopedDisabler`	Compile-time	Stack	Very low	Violations are ignored for the lifetime of the `ScopedDisabler` object.
`function-name-matches` suppression	Run-time	Single function	Medium	Suppresses intercepted and `[[clang::blocking]]` function calls by name.
`call-stack-contains` suppression	Run-time	Stack	High	Suppresses any stack trace contaning the specified pattern.

`ScopedDisabler`¶

At compile time, RealtimeSanitizer may be disabled using __rtsan::ScopedDisabler. RTSan ignores any errors originating within the ScopedDisabler instance variable scope.

#include <sanitizer/rtsan_interface.h>

void process(const std::vector<float>& buffer) [[clang::nonblocking]] {
    {
        __rtsan::ScopedDisabler d;
        ...
    }
}

If RealtimeSanitizer is not enabled at compile time (i.e., the code is not compiled with the -fsanitize=realtime flag), the ScopedDisabler is compiled as a no-op.

In C, you can use the __rtsan_disable() and rtsan_enable() functions to manually disable and re-enable RealtimeSanitizer checks.

#include <sanitizer/rtsan_interface.h>

int process(const float* buffer) [[clang::nonblocking]]
{
    {
        __rtsan_disable();

        ...

        __rtsan_enable();
    }
}

Each call to __rtsan_disable() must be paired with a subsequent call to __rtsan_enable() to restore normal sanitizer functionality. If a corresponding rtsan_enable() call is not made, the behavior is undefined.

Suppression file ¶

At run-time, suppressions may be specified using a suppressions file passed in RTSAN_OPTIONS. Run-time suppression may be useful if the source cannot be changed.

> cat suppressions.supp
call-stack-contains:MallocViolation
call-stack-contains:std::*vector
function-name-matches:free
function-name-matches:CustomMarkedBlocking*
> RTSAN_OPTIONS="suppressions=suppressions.supp" ./a.out
...

Suppressions specified in this file are one of two flavors.

function-name-matches suppresses reporting of any intercepted library call, or function marked [[clang::blocking]] by name. If, for instance, you know that malloc is real-time safe on your system, you can disable the check for it via function-name-matches:malloc.

call-stack-contains suppresses reporting of errors in any stack that contains a string matching the pattern specified. For example, suppressing error reporting of any non-real-time-safe behavior in std::vector may be specified call-stack-contains:std::*vector. You must include symbols in your build for this method to be effective, unsymbolicated stack traces cannot be matched. call-stack-contains has the highest run-time cost of any method of suppression.

Patterns may be exact matches or are “regex-light” patterns, containing special characters such as ^$*.

The number of potential errors suppressed via this method may be seen on exit when using the print_stats_on_exit flag.

Compile-time sanitizer detection ¶

Clang provides the pre-processor macro __has_feature which may be used to detect if RealtimeSanitizer is enabled at compile-time.

#if defined(__has_feature) && __has_feature(realtime_sanitizer)
...
#endif