Skip to content

[BUG]: VirtualMemoryResource host_numa allocations always fail #2694

Open
Open
[BUG]: VirtualMemoryResource host_numa allocations always fail#2694
Assignees
Andy-Jostsomtri
Labels
P1Medium priority - Should do
Milestone
cuda.core 1.3.0

Description

@fedonman
fedonman
opened on Aug 25, 2026

Is this a duplicate?

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

Every VirtualMemoryResource allocation with location_type="host_numa" fails with CUDA_ERROR_INVALID_VALUE, on a system that reports host_numa_id 0 and host_numa_virtual_memory_management_supported True. VirtualMemoryResource.allocate passes prop.location.id = -1 whenever the resource has no device, which __init__ sets for every host location type, and cuMemCreate rejects CU_MEM_LOCATION_TYPE_HOST_NUMA with that id. cuda_core/cuda/core/typing.py documents HOST_NUMA as "host memory pinned to a specific NUMA node", and VirtualMemoryResourceOptions has no field that carries a node id.

location_type="host_numa_current" also fails, for a separate reason: cuMemCreate rejects CU_MEM_LOCATION_TYPE_HOST_NUMA_CURRENT for every node id I passed it, including 0.

test_vmm_host_location_types_report_host_accessible in cuda_core/tests/test_memory.py parametrizes over host, host_numa and host_numa_current, but it only checks mr.device and mr.is_host_accessible, so it stays green without ever calling allocate.

How to Reproduce

from cuda.core import Device, VirtualMemoryResource, VirtualMemoryResourceOptions

dev = Device()
dev.set_current()
print("host_numa_id =", dev.properties.host_numa_id)
for location_type in ("host", "host_numa"):
    opts = VirtualMemoryResourceOptions(location_type=location_type, handle_type=None)
    buf = VirtualMemoryResource(dev, config=opts).allocate(4096)
    print(location_type, "allocated", buf.size, "bytes")
    buf.close()
host_numa_id = 0
host allocated 2097152 bytes
Traceback (most recent call last):
  File "/tmp/vmm_host_numa.py", line 8, in <module>
    buf = VirtualMemoryResource(dev, config=opts).allocate(4096)
  File "/home/vyron-vasileiadis/projects/forks/cuda-python/cuda_core/cuda/core/_memory/_virtual_memory_resource.py", line 555, in allocate
    raise_if_driver_error(res)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "cuda/core/_utils/cuda_utils.pyx", line 134, in cuda.core._utils.cuda_utils._check_driver_error
  File "cuda/core/_utils/cuda_utils.pyx", line 145, in cuda.core._utils.cuda_utils._check_driver_error
cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.

Expected behavior

allocate should either return a buffer pinned to a NUMA node, the way location_type="host" and location_type="device" already do, or fail at construction with an error naming the missing node id. Driving cuMemCreate directly on this system, CU_MEM_LOCATION_TYPE_HOST_NUMA returns CUDA_SUCCESS at location.id = 0 and CUDA_ERROR_INVALID_VALUE at location.id = -1, so the location id is the only thing standing between the current behaviour and a working allocation.

Operating System

Ubuntu 26.04 LTS

nvidia-smi output

Tue Aug 25 11:22:23 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.84                 Driver Version: 595.84         CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3050 ...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   46C    P8              3W /   30W |      11MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Activity

  1. github-actions
    added
    triageNeeds the team's attention
    on Aug 25, 2026
  2. Andy-Jost
    added a parent issue on Sep 17, 2026

  3. somtri commented on Sep 19, 2026

    @somtri
    somtri
    on Sep 19, 2026

    I'd like to investigate the host-NUMA allocation failure and add a test that calls allocate.

  4. rparolin
    added this to the cuda.core 1.3.0 milestone on Sep 24, 2026
  5. rparolin
    added
    P1Medium priority - Should do
    and removed
    triageNeeds the team's attention
    on Sep 24, 2026
  6. rparolin
    assigned
    Andy-Jost
    on Sep 24, 2026

  7. Andy-Jost commented on Oct 5, 2026

    @Andy-Jost
    Andy-Jost
    on Oct 5, 2026
    Contributor

    @somtri Thanks, the issue is yours.

    What goes wrong: the resource drops its device for every host location type, and _fill_prop then writes prop.location.id = -1. cuMemCreate requires the NUMA node id for CU_MEM_LOCATION_TYPE_HOST_NUMA (0 on a system without NUMA) and rejects CU_MEM_LOCATION_TYPE_HOST_NUMA_CURRENT for physical allocations, so neither type can allocate today.

    The fix we expect:

    • Add a NUMA node field to VirtualMemoryResourceOptions, for example numa_id: int | None = None. Require it for location_type="host_numa", reject it for every other location type in _check_config, and write it to prop.location.id in _fill_prop. The access descriptors copy that location, and the driver ignores the id of a HOST_NUMA access entry, so _fill_access needs no change.
    • Reject location_type="host_numa_current" in _check_config with a ValueError that says the driver does not accept it for physical allocations. Keep the enum member.
    • Tests in cuda_core/tests/test_memory.py, next to test_vmm_host_location_types_report_host_accessible: one that allocates with host_numa and device.properties.host_numa_id, copies data through the buffer, and closes it without a CUDAWarning (skip when properties.host_numa_virtual_memory_management_supported is false or host_numa_id is negative); one that host_numa_current raises at construction; one that a missing or misplaced numa_id raises. VMM tests call the real driver: no monkeypatching of driver.* and no hand-built Buffer objects.
    • A bullet under "Fixes and enhancements" in cuda_core/docs/source/release/1.3.0-notes.rst.

    Keep the other host-location gaps (device_id ignored for host resources, peers on a host resource) out of this change; they are item 7 of #2910.

  8. Andy-Jost
    assigned
    somtri
    on Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1Medium priority - Should do

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions