Recently, at work, I encountered a strange bug with GCC 7.2 and clang 6 (I didn’t test it with Visual Studio 2017 for different reasons). The bug was not visible on “old” compilers like gcc 4, Visual Studio 2013 or even Intel Compiler 2017. In debug mode, everything was fine, but in release mode, the application crashed. But not always at the same location.

Tools to debug

As we run valgrind all the time, I knew that the error could not be found with valgrind. When debugging the error, there was nothing that was wrong. All the variables were defined properly, were local or passed by value (for shared pointers), so nothing popped up.

But I had a feeling I would be able to find it with Address Sanitizer. So I ran it with the option ASAN_OPTIONS=detect_stack_use_after_return=1. And then I found it. Use after stack, and where ASAN found the error, I could figure out that we kept a reference to a stack variable that was removed.

What Address Sanitizer found and how to understand the reports

The following piece of code is a simplification of what was written. It may well be that the code was correct the first time it was written because Foo was supposed to be used locally. But in this context, it is not correct.

#include <iostream>
 
struct Foo
{
  Foo(const int& bar)
  : bar(bar)
  {}
 
  const int& bar;
};
 
Foo generate()
{
  int i = 99;
  return Foo(i);
}
 
 
int main()
{
  Foo foo = generate();
 
  std::cout << foo.bar << std::endl;
}

As you can see, Foo keeps a reference to an int, and in this case, that integer was allocated on the stack and was destroyed when we access the reference. In debug mode, you would get 99. In optimized mode, you get anything. Literally.

To compile it, just do

clang++ test.cpp -fsanitizer=address

OK, so what does ASAN returns?

=================================================================
==24406==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f2db2c00040 at pc 0x0000005172e0 bp 0x7ffe043cf770 sp 0x7ffe043cf768
READ of size 4 at 0x7f2db2c00040 thread T0
    #0 0x5172df in main (/home/mbrucher/local/temp/a.out+0x5172df)
    #1 0x7f2db60ffc04 in __libc_start_main (/lib64/libc.so.6+0x21c04)
    #2 0x41a757 in _start (/home/mbrucher/local/temp/a.out+0x41a757)

Address 0x7f2db2c00040 is located in stack of thread T0 at offset 64 in frame
    #0 0x516fcf in generate() (/home/mbrucher/local/temp/a.out+0x516fcf)

  This frame has 2 object(s):
    [32, 40) 'retval'
    [64, 68) 'i' <== Memory access at offset 64 is inside this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-return (/home/mbrucher/local/temp/a.out+0x5172df) in main
Shadow bytes around the buggy address:
  0x0fe636577fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe636577fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe636577fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe636577fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe636577ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0fe636578000: f5 f5 f5 f5 f5 f5 f5 f5[f5]f5 f5 f5 f5 f5 f5 f5
  0x0fe636578010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe636578020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe636578030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe636578040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe636578050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==24406==ABORTING

The report can be confusing. The trick is to compile with -g to have proper stack information. Here, I get where the bad memory access occurs AND where I stored the wrong reference. Then, the color system allows to check what happens in the memory. Here, it’s only f5, so stack after return information (you could get bound check, deallocated memory…). So we can look for a stack variable that was used, hence the reference that is the culprit.

Conclusion

Address Sanitizer is great. Of course, by default, it checks memory leaks, bound checks… But it can do far more than just these. It is better than valgrind on several aspects, like speed (as it’s not emulation based) but also on what it can check. It saved me lots of time already despite having used it only for a few months, so consider adopting it. It’s a puppy that doesn’t require much time.