Show multiple levels of source-line mapping for C++ stack traces

[Twinklebear]

Problem Statement

We're shipping a C++/WASM app that I recently hooked up debug symbols for in Sentry, so now I can start seeing source line mapping in Sentry for stack traces in the WASM which is great. However, I noticed that when template calls are involved in the stack trace I'm only shown the first line mapping in Sentry, which isn't useful since it's usually referring to some standard library code and not my app's code.

For example, I can have a stack frame entry like:

at code.wasm.std::__2::__function::__func<code::DatasetFilter::Execute()::$_1, std::__2::allocator<code::DatasetFilter::Execute()::$_1>, void (code::FetchDataset::Result, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>> const&)>::operator()(code::FetchDataset::Result&&, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char>> const&) (http://localhost:8080/v0.6.15.352e2231a07166714eb3.wasm:wasm-function[3518]:0x11d271)

If I run wasm addr2line on this memory address it expands to multiple source lines:

code::DatasetFilter::Execute()::$_1::operator()(code::FetchDataset::Result, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) const /Users/will/repos/code/src/code/filter/import_dataset.cpp:196:11
decltype (((std::declval<code::DatasetFilter::Execute()::$_1&>)())(((std::declval<std::__2::__invoke[abi:ne180100]::FetchDataset::Result, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&>)())...)) std::__2::__invoke[abi:ne180100]<code::DatasetFilter::Execute()::$_1&, std::__2::__invoke[abi:ne180100]::FetchDataset::Result, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&>(std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&, code::DatasetFilter::Execute()::$_1&) /Users/will/libs/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__type_traits/invoke.h:344:25
void std::__2::__invoke_void_return_wrapper<void, true>::__call[abi:ne180100]<code::DatasetFilter::Execute()::$_1&, std::__2::__invoke_void_return_wrapper<void, true>::__call[abi:ne180100]::FetchDataset::Result, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&>(code::DatasetFilter::Execute()::$_1&, std::__2::__invoke_void_return_wrapper<void, true>::__call[abi:ne180100]::FetchDataset::Result, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) /Users/will/libs/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__type_traits/invoke.h:419:5
std::__2::__function::__alloc_func<code::DatasetFilter::Execute()::$_1, std::__2::allocator<code::DatasetFilter::Execute()::$_1>, void (code::FetchDataset::Result, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&)>::operator()[abi:ne180100](code::FetchDataset::Result&&, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) /Users/will/libs/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__functional/function.h:169:12
std::__2::__function::__func<code::DatasetFilter::Execute()::$_1, std::__2::allocator<code::DatasetFilter::Execute()::$_1>, void (code::FetchDataset::Result, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&)>::operator()(code::FetchDataset::Result&&, std::__2::basic_string<char, std::__2::char_traits<char>, std::__2::allocator<char> > const&) /Users/will/libs/emsdk/upstream/emscripten/cache/sysroot/include/c++/v1/__functional/function.h:311:10

In Sentry, I'm shown the bottom most entry in the list as the source line for memory address 0x11d271 (pic below) but our app's code is actually the top-most entry in the list and that's what would actually be useful for me to see.

So the information shown in Sentry isn't actionable, I still need to go back to our addr2line script to reconstruct the stack trace to debug the crash. In Sentry what I'd really want to see is all source lines listed here, w/ the top-most one shown as the default and then something to click to expand to show all the source line mappings here.

I also noticed that the C++ template symbols in the stack trace seem to break some regex being done by Sentry to parse and split up the stack trace, so we have a thing that catches exceptions from the WASM and splits the stack trace up on \n to build the stack frames for Sentry instead of using Sentry's exception parsing.

Solution Brainstorm

Support for multiple source lines mapping to a single memory address to be shown in Sentry, the first source line returned by addr2line should be shown first, with something to click to expand to see all source lines

Product Area

Issues - Source Maps

[getsantry]

Auto-routing to @getsentry/product-owners-issues for triage ⏲️

[armenzg]

@Twinklebear could you please get me a link to the issue in your account?
You could also paste the "exception" from the event if you prefer.
What SDK are you using?

[Twinklebear]

Is there a way I can share that internally in the Sentry app with you? The actual error is from my work project which I've tweaked the code listing a bit to make it more generic/anonymized, the company is on a paid plan but I wasn't sure what support route to go through (in app support link, or Github issue, etc.)

[Twinklebear]

I submitted a ticket through zendesk in the app w/ some more info, it's ticket # 140747

[jjbayer]

Thanks for the report, I transferred the issue to the symbolic library and we will triage it soon.

[loewenheim]

Hi! What you're describing is the resolution of inlined functions, which we already support. However, your report uncovered a bug in our cache format which led us to not resolve inlinees correctly in some cases. The fix is #883.

[Twinklebear]

Awesome thanks @loewenheim ! When should I expect to see this show up in the app? (not sure what your release timeline/frequency is like)

[jjbayer]

@Twinklebear we will try to get the fix deployed this week, though I cannot make any promises.

[Twinklebear]

No worries, I was just curious when to expect seeing the update in the app 👍