Crash with opcache_reset() under high load in fpm

[ndossche]

Description

The following code:

<?php
class T {
}

new T;
opcache_reset();

Executed under php-fpm with the following fpm configuration (save this as custom.conf):

[global]
error_log = /dev/stderr

[www]
listen = 127.0.0.1:9000
pm = static
pm.max_children = 10

Ran using: valgrind ./php-fpm -F -y custom.conf with opcache enabled.

Put under load with: ab -c 24 -n 200000 http://localhost/index.php

Resulted in this output:

==150641== Process terminating with default action of signal 6 (SIGABRT)
==150641==    at 0x4CCDE44: __pthread_kill_implementation (pthread_kill.c:44)
==150641==    by 0x4C75A2F: raise (raise.c:26)
==150641==    by 0x4C5D4C2: abort (abort.c:79)
==150641==    by 0x4C5D3DE: __assert_fail_base.cold (assert.c:94)
==150641==    by 0x4C6DC66: __assert_fail (assert.c:103)
==150641==    by 0x702B6D: zend_gc_delref (zend_types.h:1228)
==150641==    by 0x70317D: zend_string_release_ex (zend_string.h:332)
==150641==    by 0x708F6E: _php_stream_open_wrapper_ex (streams.c:2291)
==150641==    by 0x6E412B: php_stream_open_for_zend_ex (main.c:1564)
==150641==    by 0x6E407E: php_stream_open_for_zend (main.c:1552)
==150641==    by 0x7B538F9: persistent_stream_open_function (ZendAccelerator.c:2520)
==150641==    by 0x8514F6: zend_stream_open (zend_stream.c:88)

And some similar crashes, also got one crash on time in zend_accelerator_util_funcs.c in ZSTR_SET_CE_CACHE_EX(ce->name, ce, 0);.

But I expected this output instead:

Obviously not that

PHP Version

8.2

Operating System

No response

[bukka]

I have done a bit of testing on master and created a WST test for this . The test actually works on normal debug build but it helps me to quickly extract commands to run it with valgrind and was able to get few more different traces:

This is class hash copy:

==3377480== Process terminating with default action of signal 6 (SIGABRT)
==3377480==    at 0x5EBDB2C: __pthread_kill_implementation (pthread_kill.c:44)
==3377480==    by 0x5EBDB2C: __pthread_kill_internal (pthread_kill.c:78)
==3377480==    by 0x5EBDB2C: pthread_kill@@GLIBC_2.34 (pthread_kill.c:89)
==3377480==    by 0x5E6427D: raise (raise.c:26)
==3377480==    by 0x5E478FE: abort (abort.c:79)
==3377480==    by 0x5E4781A: __assert_fail_base.cold (assert.c:96)
==3377480==    by 0x5E5A516: __assert_fail (assert.c:105)
==3377480==    by 0x7A7882: _zend_accel_class_hash_copy (zend_accelerator_util_funcs.c:237)
==3377480==    by 0x7A7930: zend_accel_class_hash_copy (zend_accelerator_util_funcs.c:250)
==3377480==    by 0x7A8366: zend_accel_load_script (zend_accelerator_util_funcs.c:407)
==3377480==    by 0x7DC153: persistent_compile_file (ZendAccelerator.c:2264)
==3377480==    by 0xFE3F4D: zend_execute_script (zend.c:1969)
==3377480==    by 0xD542F1: php_execute_script_ex (main.c:2600)
==3377480==    by 0xD5450E: php_execute_script (main.c:2640)

Then memory coruption:

==3377928== Process terminating with default action of signal 11 (SIGSEGV)
==3377928==    at 0x5E6453B: kill (syscall-template.S:120)
==3377928==    by 0xE03994: zend_mm_panic (zend_alloc.c:408)
==3377928==    by 0xE05BB9: zend_mm_free_heap (zend_alloc.c:1547)
==3377928==    by 0xE08C5D: _efree (zend_alloc.c:2812)
==3377928==    by 0xD498D2: zend_string_release_ex (zend_string.h:353)
==3377928==    by 0xD4AE17: php_fopen_primary_script (fopen_wrappers.c:464)
==3377928==    by 0xFF8C4D: main (fpm_main.c:1873)

Then this:

==3310952== Process terminating with default action of signal 11 (SIGSEGV)
==3310952==  General Protection Fault
==3310952==    at 0x7D2F7B: zend_string_copy (zend_string.h:209)
==3310952==    by 0x7DD5B5: persistent_zend_resolve_path (ZendAccelerator.c:2571)
==3310952==    by 0xD4AD58: php_fopen_primary_script (fopen_wrappers.c:447)
==3310952==    by 0xFF8C4D: main (fpm_main.c:1873)

And finally

==3310956== Process terminating with default action of signal 6 (SIGABRT)
==3310956==    at 0x5EBDB2C: __pthread_kill_implementation (pthread_kill.c:44)
==3310956==    by 0x5EBDB2C: __pthread_kill_internal (pthread_kill.c:78)
==3310956==    by 0x5EBDB2C: pthread_kill@@GLIBC_2.34 (pthread_kill.c:89)
==3310956==    by 0x5E6427D: raise (raise.c:26)
==3310956==    by 0x5E478FE: abort (abort.c:79)
==3310956==    by 0x5E4781A: __assert_fail_base.cold (assert.c:96)
==3310956==    by 0x5E5A516: __assert_fail (assert.c:105)
==3310956==    by 0xE82928: zend_vm_init_call_frame (zend_execute.h:330)
==3310956==    by 0xE82BFF: zend_vm_stack_push_call_frame_ex (zend_execute.h:350)
==3310956==    by 0xE82CA2: zend_vm_stack_push_call_frame (zend_execute.h:369)
==3310956==    by 0xF267D5: zend_execute (zend_vm_execute.h:64388)
==3310956==    by 0xFE3FB2: zend_execute_script (zend.c:1976)
==3310956==    by 0xD542F1: php_execute_script_ex (main.c:2600)
==3310956==    by 0xD5450E: php_execute_script (main.c:2640)

[bukka]

I can also confirm that after applying changes in #14803 I can no longer recreate the issue.

[ndossche]

I remember this. This can also happen without an explicit call to opcache_reset when shm gets full and opcache does call it internally.
I had trouble getting it reliably reproduced as I just used an infinite loop in bash to trigger requests, not very pretty.
I'll take a look at your wst test, that may come in handy, I don't have experience (yet) with your toolchain though

[bukka]

I actually didn't manage to recreate it using WST run as it currently does not support running servers (FPM in this case) under Valgrind. So I used it more just to setup the environment (creates php file, configs and everything needed for execution). This happens when the instance (test) is run which runs successfuly for me because it doesn't show anything when not run with Valgrind. It also prints all the commands to the output which I just copy, then open shell and run nginx. In my case the nginx is run as:

/usr/sbin/nginx -c /home/jakub/prog/wst/wst-php-fpm/workspace/opcache-reset-race/nginx/_env/etc/nginx/nginx.conf

And FPM run with valgrind as

valgrind /usr/local/sbin/php-fpm -c /home/jakub/prog/wst/wst-php-fpm/workspace/opcache-reset-race/fpm/_env/etc/fpm/php.ini -F -R -y /home/jakub/prog/wst/wst-php-fpm/workspace/opcache-reset-race/fpm/_env/etc/fpm/fpm.conf

And it also gives me vegeta (something like ab but in go) command to run:

echo "GET http://localhost:8526/index.php" | vegeta attack -duration=3s -rate=400/1s | vegeta report

And this quite reliably crashing.

I haven't tried with ASAN which could potentially recreate it directly in the test. Adding Valgrind support to the tool shouldn't be too hard so I will look into it. But for debugging I usually use the extracted commands anyway because it gives me a bit more control. The test run is more important for later regression testing.

If you want to set it up, the install notes are in https://github.com/wstool/wst#installation (requires go 1.23). It will also require checking out wst-php-fpm and then execute the test:

wst run --debug -c /path/to/wst-php-fpm/wst.yaml opcache-reset-race

The -c for path to wst.yaml is not necessary if cwd is in that wst-php-fpm install dir. If you need any help, feel free to let me know.