[Ubuntu] Segmentation fault on quit starting from #5892

[nclok1405]

Starting from libultraship update at #5892 (and 9c1a172 in nightly develop) SoH started to crash after exiting the game on Ubuntu 22.04.

In the log file the last message before the crash is:
[ResourceManager.cpp:65] [info] destruct ResourceManager

The last build without crash at exit for me is "Owl Travel cutscenes skipped as One Point cutscenes" #6141 (c7e4e8c in nightly develop)

EDIT: I built a debug build with latest commit and used gdb to investigate.

The actual last log message before crash is:
[ConsoleVariable.cpp:20] [trace] destruct console variables

Backtrace is:

(gdb) bt
#0  std::__atomic_base<int>::load (__m=std::memory_order::relaxed, this=0x40) at /usr/include/c++/11/bits/atomic_base.h:488
#1  spdlog::logger::should_log (this=0x0, msg_level=spdlog::level::debug) at /usr/include/spdlog/logger.h:268
#2  0x00005555567bad50 in spdlog::logger::log (this=0x0, loc=..., lvl=spdlog::level::debug, msg=...) at /usr/include/spdlog/logger.h:127
#3  0x0000555556d99226 in spdlog::logger::log<char [22], 0> (this=0x0, loc=..., lvl=spdlog::level::debug, msg=...) at /usr/include/spdlog/logger.h:102
#4  0x0000555558d13b22 in Fast::Fast3dWindow::~Fast3dWindow (this=0x55555b044e50, __in_chrg=<optimized out>) at /home/myuser/app/shipfork/libultraship/src/fast/Fast3dWindow.cpp:47
#5  0x0000555558250e9f in std::destroy_at<Fast::Fast3dWindow> (__location=0x55555b044e50) at /usr/include/c++/11/bits/stl_construct.h:88
#6  0x0000555558250726 in std::allocator_traits<std::allocator<Fast::Fast3dWindow> >::destroy<Fast::Fast3dWindow> (__a=..., __p=0x55555b044e50) at /usr/include/c++/11/bits/alloc_traits.h:537
#7  0x000055555824f58d in std::_Sp_counted_ptr_inplace<Fast::Fast3dWindow, std::allocator<Fast::Fast3dWindow>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x55555b044e40)
    at /usr/include/c++/11/bits/shared_ptr_base.h:528
#8  0x0000555556883c15 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x55555b044e40) at /usr/include/c++/11/bits/shared_ptr_base.h:168
#9  0x000055555688374f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x55555a744788 <sohFast3dWindow+8>, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/shared_ptr_base.h:705
#10 0x0000555556de0314 in std::__shared_ptr<Fast::Fast3dWindow, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x55555a744780 <sohFast3dWindow>, __in_chrg=<optimized out>)
    at /usr/include/c++/11/bits/shared_ptr_base.h:1154
#11 0x0000555556de0334 in std::shared_ptr<Fast::Fast3dWindow>::~shared_ptr (this=0x55555a744780 <sohFast3dWindow>, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/shared_ptr.h:122
#12 0x00007ffff76e6495 in __run_exit_handlers (status=0, listp=0x7ffff78bb838 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:113
#13 0x00007ffff76e6610 in __GI_exit (status=<optimized out>) at ./stdlib/exit.c:143
#14 0x00007ffff76cad97 in __libc_start_call_main (main=main@entry=0x55555880f07b <main>, argc=argc@entry=1, argv=argv@entry=0x7fffffffd1d8) at ../sysdeps/nptl/libc_start_call_main.h:74
#15 0x00007ffff76cae40 in __libc_start_main_impl (main=0x55555880f07b <main>, argc=1, argv=0x7fffffffd1d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd1c8)
    at ../csu/libc-start.c:392
#16 0x0000555556700b25 in _start ()

Around Fast3dWindow.cpp:47:

Fast3dWindow::~Fast3dWindow() {
     SPDLOG_DEBUG("destruct fast3dwindow");
     mInterpreter->Destroy();
     delete mRenderingApi;
     delete mWindowManagerApi;
}

it looks to me SpdLog is being used after it already got unloaded from memory. If I commenting this line out ("destruct fast3dwindow"), another call to SpdLog from Resource.cpp causes crash at exit. I'm still not sure if this crash actually originates from SoH-side code or libultraship yet.

[nclok1405]

I've tried the latest nightly on a completely different laptop, different OS version (24.04.3), and clean Live USB environment, yet the exact same crash happens on exit, so it is not Ubuntu 22.04 specific.

PC1 (Desktop)
OS: Ubuntu 22.04.5 LTS (Installed)
CPU: AMD Ryzen 5 3500 6-Core Processor
RAM: 32GB
GPU: NVIDIA GeForce RTX 3060 (12GB)
Driver: NVIDIA Proprietary 535.274.02

PC2 (Laptop)
OS: Xubuntu 24.04.3 LTS (Live USB Session)
CPU: 13th Gen Intel(R) Core(TM) i7-13620H
RAM: 32GB
GPU: NVIDIA GeForce RTX 3050 (6GB)
Driver: Mesa 25.0.7 (open source driver)

[nclok1405]

Using the last non-crashing build as the base, I did an experiment to update libultraship and ZAPDTR versions to match the current develop. The game still doesn't crash at exit, so these submodules are not triggering the issue, at least not directly.

Although I haven't looked deeply yet, I noticed #5892 introduced new usage of <atomic>. Spdlog also uses <atomic> internally (including Spdlog's should_log function) so there might be a conflict. Spdlog does have a compile-time option to avoid <atomic> usage (SPDLOG_NO_ATOMIC_LEVELS) but if doing so we'll likely lose the ability to change log level mid-game.

[Malkierian]

@serprex Any ideas on how to implement that without ? We could at least give it a shot for nclok to test. I'm just at a loss for proper thread safety in that instance.

[serprex]

You'd have to use rwlock or mutex. Or just embrace UB implementation. I think your suggestion to remove the log line from LUS was the better idea

[ShishuTheDragon]

@nclok1405 Can you please test #6212 to see if this fixes your issue. (It fixed it for me, but I’m on macOS, so just want to confirm).

Although the stack trace says it looks like it was actually because the logger was null (note the spdlog::logger::log (this=0x0 in the stack trace).

There was a lingering reference to the window meant that the destruction happened after spdlog::shutdown.

[Malkierian]

Yes, it fixes this.

[nclok1405]

Tested #6212 appimage build on Ubuntu 22.04. It worked! Thank you!