-
Notifications
You must be signed in to change notification settings - Fork 533
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SIGSEGV reported by Sentry when a NullReferenceException is thrown in Release mode #9055
Comments
@tranb3r considering I have no idea what Sentry is and how to use it, I will need more info from you :) Can you add to this issue The fact that you don't use native code directly doesn't mean it isn't involved, in fact, it's always involved in one manner or another. If you're able to reproduce this issue locally, please capture logcat output using the following commands: > adb shell setprop debug.mono.log default,assembly,mono_log_level=debug,mono_log_mask=all
> adb logcat -G 64M
> adb logcat -c
rem Start and crash the app here, wait 2-3 seconds and then:
> adb logcat -d > logcat.txt |
@grendello
The manage exception is catched. I've added an INFO log so you can see it in the logcat.
The application is not crashing, and I haven't seen a trace for the Segfault. So here are two logcats:
|
@tranb3r thanks, I'd still like to see the error your instance of Sentry records, the one with |
@grendello |
@tranb3r thanks! They clearly have the whole stack trace somewhere, since without it they wouldn't show the registers nor the frame where it happens. Alas, their UI made the trace useless - could you try digging in the UI to find the raw data they parse? |
I really think that's everything I can get from the UI. Adding @jamescrosswell , hope you don't mind. |
It's too bad the raw data is missing. With these crashes the context is everything - not just the header, but most importantly the frames themselves. With some kinds of signals (e.g. SIGABRT) lines preceding the native trace are often crucial. Contents of registers, as shown in the screenshot, is mostly of secondary interest - they don't give us any information about the location of the crash (except of, in this case, the |
Here is the data that is saved by Sentry when the error occurs. It contains a bit more than what is visible in the UI. Maybe you can take a look? |
The only frame in this data is this: "frames": [{
"instruction_addr": "0x7085005af665",
"package": "/data/app/~~lhSuplZuwh7nocNaMhXmWw==/com.companyname.mauiappsegfault-henAPYLgWXjjYhjFwUCWbQ==/split_config.x86_64.apk!/lib/x86_64/libaot-MauiAppSegfault.dll.so",
"image_addr": "0x7085005ae000"
}] which is weird, because AOT libraries don't contain directly executable code. They contain code-as-data that is loaded by the runtime, patched up and made executable so the frame shouldn't point to anything in that shared library. However, try without AOT and see if you still get the segfault? I don't know if this is the only frame that was logged by Android or the only frame that was deemed worthy of being saved by Sentry, but without the rest it's very, VERY, hard to see what's going on. The fact that the segfault is reported by Sentry, but is not in your logcat is very weird. The only explanation that comes to me is that Sentry intercepts it and prevents it from ending up in the logcat, exiting the application "cleanly" instead. Is such scenario possible? |
Yes. Still getting the segfault without AOT (
This is why I also pasted the logcat without sentry. I don't see the segfault in this log. |
@tranb3r Sentry could intercept the signal if it installed its own handler and didn't chain it up to the previous one before exiting the application. This could prevent the signal from being logged in the logcat. |
Sure. |
It may also mean that Sentry is causing and catching the segfault. But we don't have enough information to figure that out yet. |
I will consider opening an issue in Sentry SDK repository if I do not get an answer from James or Stefan here. |
sentry-native should chain the handlers so this shouldn't be the case. @Swatinem or @markushi could confirm this
Looking at the screenshot of this event in Sentry I noticed it's missing symbols. Any chance you can upload debug symbols when building the app so we can see the stack trace? Upload is done automatically via msbuild if you configure your .NET project. Thanks for the repro though, at least we have something to dig into it. @bitsandfoxes said he's taking a look |
Running the repro without any problems and I can't seem to reproduce the segfault being sent. I'm on Android 14 on a Pixel 6, might this be a device-specific issue? |
Did you follow all the steps ? (release mode ; tap the run button; restart the app) |
Let me try that again. |
I can confirm my repro on:
As soon as the Run button is pressed and the exception is triggered, a Is Sentry capturing a crash of the app? Or is it Sentry SDK that is actually crashing? |
So after a whole bunch of testing:
@grendello is this intended behavior? And if so, how does the runtime handle the signal? What is getting checked to ignore the signal safely? For context, the SDK is hooking itself up to the signal handler and it receives a signal, thus creating an event from it. That signal then gets forwarded and seems to get swallowed. |
@bitsandfoxes no, the behavior isn't correct... I wonder if your handler catches the signal on a thread that's not attached to MonoVM and thus there are no handlers to chain to. |
To confirm, a C# |
@bruno-garcia no, a managed |
Doesn't seem to be the case as OP mentioned without Sentry still nothing shows up on logcat. Unless I misunderstood things.
That's possible (except the signal might still exist even without Sentry since nothing shows up logcat either with or without Sentry). I'm saying that after reading:
|
It's a very, very, rare case that an Android application crashes with segfault and nothing gets logged. I think, I've seen only one such instance over the years. Sometimes there's very little logged, but there's always a trace. If MonoVM misses to catch the signal, ART or Zygote will. Mono won't silently handle and ignore a segfault, so the fact that we see nothing in the logcat without Sentry means that most likely (very likely) there's no signal raised.
This is very unlikely, as explained above. The only way for that to happen is if there were a signal handler installed somewhere which would handle and not log the segfault. Neither ART nor MonoVM would do that, and also there's no good reason to swallow such destructive and dangerous signals, I can't imagine why a legitimate piece of software would do that. The only scenario I can imagine is where we have a corrupted chain of signal handlers. For instance, let's consider a situation where both ART and MonoVM uninstall their own handlers and the chain is left with a handler in the middle that does some processing when it captures a signal and then passes it on, if there is another handler installed. If we assume that other signals are gone and our hypothetical handler doesn't have logging code, nor was it designed to abort the application, the app can keep running. I can imagine this scenario with software like Sentry running in the application, but I can't imagine it without it present (barring application itself doing that, of course). |
Is there anything I can do to help make some progress? |
Hi, I'm a Sentry Native SDK maintainer. I had the chance to investigate this topic further to understand what was happening. In short, what is happening is the following:
This is why you will get two events and see no crashes in logcat. The The Native SDK will also receive that In the case of a native-provoked managed code exception, our handler would never execute (and, as a result, produce a crash event). Only if the runtime handler continues the signal chain (which would either be an unintended CLR crash or, more likely, a crash in some other native code) will we send a native crash. If you want to follow along the process in the dotnet runtime:
I couldn't see any significant differences between a Linux/Android handling, rather that the same is true for other POSIX systems as well (except where that layer only emulates lower level mechanisms, i.e., Mach and SEH). I also created a trivial dotnet program (running on a Linux GHA runner) that installs a signal handler similar ours (without all the handling code, but maintaining the signal chain): https://github.com/supervacuus/signals_dotnet This program doesn't use any sentry code and shows the same behavior, the last installed signal handler will receive the signal provoked by the generated code while the dotnet runtime handler (if invoked) will produce a |
Awesome - thank you @supervacuus !
It sounds like we can close this issue in the |
Yes, I think this entirely a sentry issue. |
@supervacuus thanks for the analysis, however I have one thing that must be noted. In the case of .NET for Android, CLR isn't used, we use the MonoVM runtime, so while overall behavior might be the same, the details may differ. |
Yeah, sorry, you are absolutely right @grendello. I did not mention this because I did not want to extend the already long comment. While the concrete implementation differs,
My point was also that we (Sentry) should probably have a more abstract view of the differences between the two implementations and care more about the similarity in behavior as seen from the chaining in our handler. But I am not a runtime dev, and any input you can give us on the implementation details and how they could affect choices in our handler is more than welcome! |
@supervacuus got it :) The details I presume will be more or less the same, as far as the mechanics are concerned, after all we're dealing with standard POSIX way of chaining signals. I mentioned MonoVM just for the record and fullness of information, so that future readers of this issue have a clear image on what's involved here. |
I don't have much value to add except to say it's happening to me too. Thanks @tranb3r for submitting this and giving me a hint as to what might have triggered the intermittent SIGSEGV I'm seeing in release code. So far, it's highly intermittent and not reproducible but this is at least a clue as to what to look for! |
This does not crash the actual app, it just shows an error in Sentry. |
I think this is a bug in Sentry. How will Sentry fix this? |
That's being tracked here: And the PR addressing it is already open: It's available on 7.15.0-alpha.1 of Sentry for Android (not the .NET SDK yet) while we test things out. But once it's all validated, we'll merge this to |
@supervacuus @jamescrosswell |
Could you share some context about the crash? The stack trace would help. |
Here is the logcat.
|
@bruno-garcia |
@tranb3r we had this issue and PR related in the sentry-dotnet repo:
I think that issue was a bit confusing though as it described two things - the SIGSEGV and the missing logcat attachments. In any event, yes I think you could open a new issue in the |
|
Android framework version
net8.0-android
Affected platform version
VS 2022 17.10.0
Description
I'm getting SIGSEGV reports from Sentry when running my Maui app on Android.
I've norrowed it down to a NullReferenceException happening in Release mode.
However, I don't understand why I get both a NullReferenceException and a SIGSEGV, since I'm using only managed code.
Even when I catch the NullReferenceException, the SIGSEGV still occurs, without the app actually crashing.
Also, I haven't seen anything weird in the log. How is Sentry catching this SIGSEGV? I'm lost here...
Could you please take a look at my repro? https://github.com/tranb3r/Issues/tree/main/MauiAppSegfault
Steps to Reproduce
Did you find any workaround?
No workaround.
Relevant log output
No response
The text was updated successfully, but these errors were encountered: