Opcache Optimization triggering Segmentation Faults when using OpenTelemetry hooks

[rutek]

I don't have a code that can be used for reproducing the issue, so I just want to share my findings with you. I hope it may help someone else to finally find the source of the issue, as I already spent on it many hours. I tried creating simple PHP projects that should reproduce the issue, but I couldn't do so.

Issue & debugging

I found out that my client's application triggers Segmentation Faults since we started using opentelemetry PHP extension. It happens for each Apache2 Worker that triggers some scenario. From my testing, Segmentation Faults happen when there is a registered hook that overrides the function return type, but an exception happens when the instrumented function is invoked. We removed our hooks that did that to ensure that's not fault from our code, but it was still occurring in other places. We found out that the hook that was causing it was coming from the open-telemetry/opentelemetry-auto-slim package:

https://github.com/opentelemetry-php/contrib-auto-slim/blob/2a4b89b86948f8750013d82569a6d8596f733e38/src/SlimInstrumentation.php#L79-L114

It matches the same requirements for the bug to happen that we found in our code:

• Return type override defined.
• Fails when uncaught Exception is thrown within the code.

I ran gdb for the failing Apache2 Worker and I got the following trace:

0x00007f7e9b3f388a in _emalloc_256 () from /usr/lib/apache2/modules/libphp.so
(gdb) bt
#0  0x00007f7e9b3f388a in _emalloc_256 () from /usr/lib/apache2/modules/libphp.so
#1  0x00007f7e9aed503d in zend_accel_load_script ()
   from /usr/local/lib/php/extensions/no-debug-non-zts-20220829/opcache.so
#2  0x00007f7e9c445e4a in xdebug_compile_file (file_handle=<optimized out>, type=<optimized out>)
    at /tmp/pear/temp/xdebug/src/base/base.c:94
#3  0x00007f7e9b3df8b6 in compile_filename () from /usr/lib/apache2/modules/libphp.so
#4  0x00007f7e9b4507b0 in zend_include_or_eval () from /usr/lib/apache2/modules/libphp.so
#5  0x00007f7e9b45e949 in ZEND_INCLUDE_OR_EVAL_SPEC_OBSERVER_HANDLER ()
   from /usr/lib/apache2/modules/libphp.so
#6  0x00007f7e9b485945 in execute_ex () from /usr/lib/apache2/modules/libphp.so
#7  0x00007f7e9b40ff3d in zend_call_function () from /usr/lib/apache2/modules/libphp.so
#8  0x00007f7e9b41031d in zend_call_known_function () from /usr/lib/apache2/modules/libphp.so
#9  0x00007f7e9b2eb7a6 in spl_perform_autoload () from /usr/lib/apache2/modules/libphp.so
#10 0x00007f7e9b40f18e in zend_lookup_class_ex () from /usr/lib/apache2/modules/libphp.so
#11 0x00007f7e9b410720 in zend_fetch_class_by_name () from /usr/lib/apache2/modules/libphp.so
#12 0x00007f7e9b45c0b7 in ZEND_NEW_SPEC_CONST_UNUSED_HANDLER ()
   from /usr/lib/apache2/modules/libphp.so
#13 0x00007f7e9b485afc in execute_ex () from /usr/lib/apache2/modules/libphp.so
#14 0x00007f7e9b40ff3d in zend_call_function () from /usr/lib/apache2/modules/libphp.so
#15 0x00007f7e9b410218 in _call_user_function_impl () from /usr/lib/apache2/modules/libphp.so
#16 0x00007f7e9b0ec662 in zend_user_exception_handler () from /usr/lib/apache2/modules/libphp.so
#17 0x00007f7e9b0ec6e8 in zend_execute_scripts[cold] () from /usr/lib/apache2/modules/libphp.so
#18 0x00007f7e9b3b8efe in php_execute_script () from /usr/lib/apache2/modules/libphp.so
#19 0x00007f7e9b503958 in php_handler () from /usr/lib/apache2/modules/libphp.so
#20 0x0000563560d0df00 in ap_run_handler ()
#21 0x0000563560d0e4e6 in ap_invoke_handler ()
#22 0x0000563560d2622e in ap_internal_redirect ()
#23 0x00007f7e9c6dd2a4 in ?? () from /usr/lib/apache2/modules/mod_rewrite.so
#24 0x0000563560d0df00 in ap_run_handler ()
#25 0x0000563560d0e4e6 in ap_invoke_handler ()
#26 0x0000563560d26dd7 in ap_process_async_request ()
#27 0x0000563560d26fdf in ap_process_request ()

The first thing I did was removing Xdebug completely. It did not help, which was expected, as the issue happened for both production and development PHP configurations.

The second thing I tried was to disable Opcache Optimizations completely by setting opcache.optimization_level=0. It stopped Segmentation Faults.

Finally, I managed to disable Opcache Optimization one by one and found out that the following one is causing these Segmentation Faults:

0x0020 (dfa_pass.c): SSA-based optimization.
(Full list: https://www.npopov.com/2022/05/22/The-opcache-optimizer.html)

Test platform: PHP 8.2 @ Linux (WSL2, happened on Ubuntu-based VMs as well). OpenTelemetry extension version: 1.1.2.

Temporary solution

In the end, I resolved the issue by setting the following value in our PHP config:

opcache.optimization_level=0x7ffebfdf

It is a default value of 0x7FFEBFFF with SSA-based optimization disabled.

Additional scenario (without debugging)

I did not have opportunity to re-test other scenarios in which the Segmentation Fault was happening, but a month ago I noted down some strange errors in PHPUnit 11.x tests when I ran tests for our OpenTelemetry hooks. It was reporting the issue and then failing with Segmentation Fault as well. The caught exception was really strange because it reported a class name from OpenTelemetry SDK, but the method name and its location was referring to the PHPUnit code.

Message: Call to undefined method OpenTelemetry\SDK\Trace\Event::snapshot()
Location: vendor/phpunit/phpunit/src/Event/Emitter/DispatchingEmitter.php:1293

Downgrade to PHPUnit 10.x resolved this issue, so I did not investigate it further. However, it looks like some memory leaks/invalid object reference, so I am mentioning it there as well. It only happened if the tested hook had return type override. Without it, it always worked.