`PageFaultManagerLinux::callPreviousHandler` is not thread-safe: concurrent SIGSEGVs on non-USM addresses crash the process

[michel2323]

Summary

PageFaultManagerLinux installs a process-wide SIGSEGV handler. For a fault on an address the page-fault manager doesn't own, it chains to the previously installed handler through callPreviousHandler(). That function tracks its position in the handler chain with handlerIndex, a plain int member shared by all threads:

• cpu_page_fault_manager_linux.h#L41: int handlerIndex = 0;
• cpu_page_fault_manager_linux.cpp#L115-L132

void PageFaultManagerLinux::callPreviousHandler(int signal, siginfo_t *info, void *context) {
    handlerIndex++;
    UNRECOVERABLE_IF(handlerIndex < 0 && handlerIndex >= static_cast<int>(previousPageFaultHandlers.size()));
    auto previousPageFaultHandler = previousPageFaultHandlers[previousPageFaultHandlers.size() - handlerIndex];
    if (previousPageFaultHandler.sa_flags & SA_SIGINFO) {
        previousPageFaultHandler.sa_sigaction(signal, info, context);
    } ...
    handlerIndex--;
}

SIGSEGV can be delivered to several threads at the same time. Suppose thread A is inside the chained application handler, so handlerIndex == 1. If thread B faults now, it computes handlerIndex == 2 and reads previousPageFaultHandlers[size - 2]. With one previous handler, that is index SIZE_MAX, before the start of the vector. B then calls through a garbage function pointer, and the process dies with SIGSEGV and no diagnostic.

This affects any application that installs its own SIGSEGV handler before Level Zero initializes and can take SIGSEGV on several threads concurrently. A typical example is a managed runtime that uses a protected page for GC safepoints: all threads fault on the page together, and each waits inside the handler until the collection is done. There the crash is deterministic.

Related problems in the same code

1. The bounds check can never fire. handlerIndex < 0 && handlerIndex >= size is always false (&& instead of ||), so the out-of-bounds read goes undetected. The valid range is 1 <= handlerIndex <= size.
2. SA_ONSTACK is dropped. The handler is registered with sa_flags = SA_SIGINFO only (#L80). After NEO takes over, SIGSEGV runs on the faulting thread's stack, even if the application registered its own handler with SA_ONSTACK and set up alternate signal stacks. A fault caused by a stack overflow can then no longer be handled.
3. Shared state is mutated inside the handler. The SIG_DFL branch of callPreviousHandler calls sigaction() and previousPageFaultHandlers.clear() from inside the signal handler, while other threads may be reading the vector.

When the handler gets installed

• master: lazily, from SVMAllocsManager::createUnifiedAllocationWithDeviceStorage() → MemoryManager::initPageFaultManager(), i.e. on the first shared-USM allocation.
• 25.18 (1.6.33578.x): eagerly, in the MemoryManager constructor, whenever a device has local memory (during zeInit).

In both cases, PageFaultManagerLinux's constructor always registers the handler. I found no debug key that prevents this. Tested on 1.6.33578.42: UseKmdMigration=0/1, RegisterPageFaultHandlerOnMigration=1, EnableRecoverablePageFaults=0/1 and EnableImplicitMigrationOnFaultableHardware=1 all still install the handler and crash. EnableLocalMemory=0 makes the driver itself segfault during init.

Reproducer

mwe_neo_sigsegv.c is below. It installs its own SA_SIGINFO | SA_ONSTACK handler, then initializes Level Zero. After that, N threads read a PROT_NONE guard page at the same moment. The application handler waits until all N threads have arrived; then the last one unprotects the page and they all return.

gcc -O1 -pthread mwe_neo_sigsegv.c -o mwe_neo_sigsegv -lze_loader -ldl
./mwe_neo_sigsegv 8 none     # no Level Zero: OK
./mwe_neo_sigsegv 1 shared   # one thread: OK
./mwe_neo_sigsegv 8 init     # zeInit + device alloc (25.18 installs the handler here)
./mwe_neo_sigsegv 8 shared   # + zeMemAllocShared (master installs the handler here)

Output of the passing control:

threads=8 mode=none
app installed:   SIGSEGV handler 0x400f0e in ./mwe_neo_sigsegv, flags 0xc000004
OK: 8 faults handled, max 8 threads inside the app handler at once

With NEO's handler installed, the process is killed by SIGSEGV after:

after dev alloc: SIGSEGV handler 0x14671abdc3e0 in /usr/lib64/libze_intel_gpu.so.1, flags 0x4000004 (no SA_ONSTACK)

Results on Intel Data Center GPU Max 1550 nodes, 3 runs per cell, showing the number of runs that crashed:

NEO	none, 4 / 8 threads	init, 1 / 4 / 8 threads	shared, 1 / 4 / 8 threads
1.6.33578.42	0 / 0	0 / 3 / 3	0 / 3 / 3
1.6.33578.77 (SLES 15 SP7, kernel 6.4, loader 1.24)	0 / 0	0 / 3 / 3	0 / 2 / 3

The reproducer also crashes with 4 and 8 threads, and passes with 1, under every device hierarchy setting: ZE_FLAT_DEVICE_HIERARCHY=FLAT, COMPOSITE (default, EnableImplicitScaling=0, or 1) and COMBINED. At 4 threads on .77, a few runs out of 3 pass by timing.

The chaining code on master (4cdaa40) is identical to 25.18. I also checked it without a GPU: a copy of master's registerFaultHandler / pageFaultHandlerWrapper / callPreviousHandler, with verifyAndHandlePageFault() stubbed to false, crashes 5/5 with 2, 4, 8 and 32 threads.

mwe_neo_sigsegv.c

// Minimal reproducer: NEO's SIGSEGV handler chaining is not thread-safe.
//
// The application installs its own SIGSEGV handler first, as managed runtimes with
// garbage collection do. It uses a PROT_NONE guard page the way such a runtime uses a GC
// safepoint page: N threads touch the page at the same moment, and each one waits *inside
// the signal handler* until all have arrived. The last one unprotects the page and they
// all return.
//
// Without Level Zero, or with NEO installed but one thread, this works. Once NEO's
// PageFaultManagerLinux has installed its handler, NEO calls the app handler through
// callPreviousHandler(), which uses a shared, non-atomic `handlerIndex`. The second
// thread in the handler indexes previousPageFaultHandlers[size - 2] (out of bounds)
// and the process dies with SIGSEGV.
//
//   gcc -O1 -pthread mwe_neo_sigsegv.c -o mwe_neo_sigsegv -lze_loader -ldl
//   ./mwe_neo_sigsegv <nthreads> <none|init|shared>
//
//   none   : no Level Zero at all                       (control, passes)
//   init   : zeInit + context + device allocation       (LTS 25.18 installs the handler here)
//   shared : init + one zeMemAllocShared                (rolling 26.x installs it here)

#define _GNU_SOURCE
#include <level_zero/ze_api.h>
#include <dlfcn.h>
#include <pthread.h>
#include <signal.h>
#include <stdatomic.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <time.h>
#include <unistd.h>

#define CHECK(call)                                                           \
    do {                                                                      \
        ze_result_t r_ = (call);                                              \
        if (r_ != ZE_RESULT_SUCCESS) {                                        \
            fprintf(stderr, "%s failed: 0x%x\n", #call, (unsigned)r_);        \
            exit(2);                                                          \
        }                                                                     \
    } while (0)

static char *guard;
static long pagesz;
static int nthreads;
static atomic_int arrived, inside, max_inside, leader, released;
static pthread_barrier_t barrier;

static void app_handler(int sig, siginfo_t *si, void *ctx) {
    (void)sig; (void)ctx;
    char *a = (char *)si->si_addr;
    if (a < guard || a >= guard + pagesz) {
        static const char m[] = "app handler: SIGSEGV outside the guard page\n";
        write(2, m, sizeof m - 1);
        _exit(3);
    }
    atomic_fetch_add(&arrived, 1);
    int now = atomic_fetch_add(&inside, 1) + 1;
    for (int m = atomic_load(&max_inside); now > m &&
         !atomic_compare_exchange_weak(&max_inside, &m, now);)
        ;
    // "safepoint": block inside the handler until every thread has arrived (max ~2 s)
    struct timespec ms = {0, 1000000};
    for (int i = 0; i < 2000 && atomic_load(&arrived) < nthreads; i++)
        nanosleep(&ms, NULL);
    if (!atomic_exchange(&leader, 1)) {
        mprotect(guard, pagesz, PROT_READ | PROT_WRITE);
        atomic_store(&released, 1);
    }
    while (!atomic_load(&released))
        nanosleep(&ms, NULL);
    atomic_fetch_sub(&inside, 1);
}

static void print_owner(const char *when) {
    struct sigaction sa;
    sigaction(SIGSEGV, NULL, &sa);
    Dl_info info = {0};
    void *h = (sa.sa_flags & SA_SIGINFO) ? (void *)sa.sa_sigaction : (void *)sa.sa_handler;
    dladdr(h, &info);
    printf("%-16s SIGSEGV handler %p in %s, flags 0x%x%s\n", when, h,
           info.dli_fname ? info.dli_fname : "?", (unsigned)sa.sa_flags,
           (sa.sa_flags & SA_ONSTACK) ? "" : " (no SA_ONSTACK)");
    fflush(stdout);
}

static void *worker(void *arg) {
    (void)arg;
    pthread_barrier_wait(&barrier);
    (void)*(volatile char *)guard;  // every thread faults on the guard page at once
    return NULL;
}

int main(int argc, char **argv) {
    nthreads = argc > 1 ? atoi(argv[1]) : 4;
    const char *mode = argc > 2 ? argv[2] : "init";
    pagesz = sysconf(_SC_PAGESIZE);
    guard = mmap(NULL, pagesz, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

    struct sigaction sa;
    memset(&sa, 0, sizeof sa);
    sa.sa_sigaction = app_handler;
    sa.sa_flags = SA_SIGINFO | SA_ONSTACK;
    sigemptyset(&sa.sa_mask);
    sigaction(SIGSEGV, &sa, NULL);
    printf("threads=%d mode=%s\n", nthreads, mode);
    print_owner("app installed:");

    if (strcmp(mode, "none") != 0) {
        CHECK(zeInit(ZE_INIT_FLAG_GPU_ONLY));
        uint32_t n = 1;
        ze_driver_handle_t drv;
        CHECK(zeDriverGet(&n, &drv));
        n = 1;
        ze_device_handle_t dev;
        CHECK(zeDeviceGet(drv, &n, &dev));
        ze_device_properties_t props = {.stype = ZE_STRUCTURE_TYPE_DEVICE_PROPERTIES};
        CHECK(zeDeviceGetProperties(dev, &props));
        printf("device: %s\n", props.name);
        ze_context_desc_t cdesc = {.stype = ZE_STRUCTURE_TYPE_CONTEXT_DESC};
        ze_context_handle_t ctx;
        CHECK(zeContextCreate(drv, &cdesc, &ctx));
        print_owner("after init:");
        ze_device_mem_alloc_desc_t ddesc = {.stype = ZE_STRUCTURE_TYPE_DEVICE_MEM_ALLOC_DESC};
        void *dptr;
        CHECK(zeMemAllocDevice(ctx, &ddesc, 4096, 64, dev, &dptr));
        print_owner("after dev alloc:");
        if (strcmp(mode, "shared") == 0) {
            ze_host_mem_alloc_desc_t hdesc = {.stype = ZE_STRUCTURE_TYPE_HOST_MEM_ALLOC_DESC};
            void *sptr;
            CHECK(zeMemAllocShared(ctx, &ddesc, &hdesc, 4096, 64, dev, &sptr));
            print_owner("after shared:");
        }
    }

    pthread_barrier_init(&barrier, NULL, nthreads);
    pthread_t t[256];
    for (int i = 0; i < nthreads; i++)
        pthread_create(&t[i], NULL, worker, NULL);
    for (int i = 0; i < nthreads; i++)
        pthread_join(t[i], NULL);
    printf("OK: %d faults handled, max %d threads inside the app handler at once\n",
           atomic_load(&arrived), atomic_load(&max_inside));
    return 0;
}

Suggested fix

handlerIndex is the depth of the handler chain, and a chain always runs on a single thread, so the counter should be per-thread. Only one PageFaultManagerLinux is active at a time (activePageFaultManager), so a static thread_local member works. The patch below also fixes the bounds check and adds SA_ONSTACK. With SA_ONSTACK, threads that have no alternate stack still run the handler on their normal stack. With this change, the GPU-free copy above passes 5/5 at 1, 2, 4, 8 and 32 threads. I have not built NEO with it or run NEO's unit tests.

Patch against master 4cdaa40

--- a/shared/source/page_fault_manager/linux/cpu_page_fault_manager_linux.h	2026-09-25 14:29:13.000000000 +0000
+++ b/shared/source/page_fault_manager/linux/cpu_page_fault_manager_linux.h	2026-09-25 14:29:14.000000000 +0000
@@ -38,7 +38,10 @@
 
     std::vector<struct sigaction> previousPageFaultHandlers;
 
-    int handlerIndex = 0;
+    // Depth of the handler chain on the calling thread. Must be per-thread: SIGSEGV can be
+    // delivered to several threads at the same time, and a chained handler may block (e.g. a
+    // managed runtime's GC safepoint handler waits inside the signal handler).
+    static thread_local int handlerIndex;
 };
 
 class CpuPageFaultManagerLinux final : public PageFaultManagerLinux {};
--- a/shared/source/page_fault_manager/linux/cpu_page_fault_manager_linux.cpp	2026-09-25 14:29:13.000000000 +0000
+++ b/shared/source/page_fault_manager/linux/cpu_page_fault_manager_linux.cpp	2026-09-25 14:29:14.000000000 +0000
@@ -30,6 +30,7 @@
 }
 
 constinit std::atomic<PageFaultManagerLinux *> PageFaultManagerLinux::activePageFaultManager{nullptr};
+thread_local int PageFaultManagerLinux::handlerIndex = 0;
 
 PageFaultManagerLinux::PageFaultManagerLinux() {
     PageFaultManagerLinux::registerFaultHandler();
@@ -77,7 +78,7 @@
     activePageFaultManager.store(this);
 
     struct sigaction pageFaultManagerHandler = {};
-    pageFaultManagerHandler.sa_flags = SA_SIGINFO;
+    pageFaultManagerHandler.sa_flags = SA_SIGINFO | SA_ONSTACK;
     pageFaultManagerHandler.sa_sigaction = pageFaultHandlerWrapper;
 
     retVal = sigaction(SIGSEGV, &pageFaultManagerHandler, &previousPageFaultHandler);
@@ -114,7 +115,7 @@
 
 void PageFaultManagerLinux::callPreviousHandler(int signal, siginfo_t *info, void *context) {
     handlerIndex++;
-    UNRECOVERABLE_IF(handlerIndex < 0 && handlerIndex >= static_cast<int>(previousPageFaultHandlers.size()));
+    UNRECOVERABLE_IF(handlerIndex <= 0 || handlerIndex > static_cast<int>(previousPageFaultHandlers.size()));
     auto previousPageFaultHandler = previousPageFaultHandlers[previousPageFaultHandlers.size() - handlerIndex];
     if (previousPageFaultHandler.sa_flags & SA_SIGINFO) {
         previousPageFaultHandler.sa_sigaction(signal, info, context);