Mountpoint pod scheduling fails during node group updates — no mechanism to reserve node capacity or taint nodes for CSI driver priority

[mashiofu-cafex]

During a node group update or rollover, the AWS Mountpoint S3 CSI driver fails to schedule its mountpoint pods (mp-*) when the target node has exhausted its pod/IP capacity. Any pod with an S3 PVC gets stuck in ContainerCreating indefinitely with a misleading error that doesn't point to the actual cause.

Error seen on the requesting pod:

MountVolume.SetUp failed for volume "s3-master": rpc error: code = Internal
desc = Could not mount "s3-csi-bucket-xxx" at "/var/lib/kubelet/pods/.../mount":
Failed to wait for Mountpoint Pod "mp-46cdk" to be ready:
mppod/watcher: mountpoint pod not found. Seems like Mountpoint Pod is not in 'Running' status.

Error seen on the mountpoint pod:

0/3 nodes are available: 1 Too many pods, 2 node(s) didn't satisfy
plugin(s) [NodeAffinity]. no new claims to deallocate,
preemption: not eligible due to preemptionPolicy=Never.

Environment

• CSI Driver version: v2.5.0
• Mountpoint version: 1.22.2
• Kubernetes version: v1.35.3-eks
• Installed as: EKS managed addon (aws-mountpoint-s3-csi-driver) via eksctl create addon --version latest
• Node instance type: r6a.xlarge
• Max pods per node (default VPC CNI): 58

Steps to Reproduce

1. Run an EKS cluster with nodes at or near their default VPC CNI pod limit (58 for r6a.xlarge)
2. Trigger a node group update/rollover — existing pods reschedule onto remaining nodes, filling them up
3. Create a pod with an S3 PVC on a node now at pod capacity
4. The CSI driver creates a mountpoint pod (mp-*) with a NodeAffinity pinning it to the same node
5. The mountpoint pod stays Pending — the node is full and no other node satisfies the NodeAffinity
6. The requesting pod stays in ContainerCreating with the misleading error above

Expected Behavior
One or more of the following:

1. Capacity pre-check — before creating a mountpoint pod, check whether the target node has available pod slots and return a clear actionable error if not (e.g. node has insufficient pod capacity, consider enabling VPC CNI prefix delegation)
2. Node taint/label support — document or automate a mechanism to taint new nodes during rollouts with a matching toleration on mountpoint pods, so the CSI driver can claim pod slots before general workloads fill the node
3. Better error propagation — when a mountpoint pod fails scheduling, surface the FailedScheduling reason in the PVC mount error so operators can identify the root cause without separately inspecting the mp-* pod

Workaround
Cordon the full node, delete the affected pod so it reschedules onto a node with capacity, freeing a slot for the mountpoint pod. This is disruptive and not viable during automated rollouts.

[vladem]

Hey @mashiofu-cafex, thanks for the detailed report. We're planning to address this issue with DaemonSet-based architecture. We'll update the issue once it is released.