- Dynamic Node Manager
A Python service that watches Kubernetes namespaces, estimates YuniKorn queue demand (running + pending pods), and dynamically converts Slurm batch nodes into Kubernetes nodes when demand exceeds configured queue headroom or when bin-pack deadlocks are detected. When nodes go idle long enough, it reverts them back to Slurm and reduces YuniKorn queue capacity accordingly.
This tool is designed for environments where:
- Slurm/batch nodes can be switched into a Kubernetes pool via an external
node-convertscript. - YuniKorn is used for queue management and its capacities are controlled through a
queues.yamlstored in a ConfigMap.
You configure a set of monitored_namespaces. For each namespace, you also configure a corresponding YuniKorn queue_path.
For each namespace’s queue path, it:
- Lists pods in that namespace.
- Filters pods to those assigned to the queue path (by annotation/label).
- Sums effective CPU/memory for:
- Running pods (counted as “used”)
- Pending pods (counted as “pending demand”)
- Applies default requests for Pending pods that declare no resources (so “invisible” pending work still triggers scaling).
Conversion is triggered when any of these is true:
- Total demand (running + pending) exceeds queue headroom for CPU or memory.
- Pending demand exceeds available capacity (after subtracting running usage).
- A “bin-pack blocked” condition is detected (see below).
When scaling out is required, the actuator:
- Runs
/usr/site/rcac/sbin/node-convert --set k8s ... --namespace <ns> - Waits for new node(s) to appear in the Kubernetes API
- Records converted nodes in a local JSON file
- Increases YuniKorn queue capacity by
(nodes_added * node_cpu_capacity, nodes_added * node_memory_capacity)only after the nodes actually join the cluster
Periodically checks converted nodes:
- If a converted node has no active pods in its namespace, and it has been idle longer than
reversion_idle_seconds, the tool:- runs
/usr/site/rcac/sbin/node-convert --set batch --node-name <node> - waits for node to disappear from the K8s API
- decreases YuniKorn queue capacity accordingly
- removes it from the converted node registry
- runs
This tool treats YuniKorn as the source of truth for “how much capacity a queue is allowed to consume”.
It:
- Reads
queues.yaml (ConfigMap: yunikorn-configs by default) - Locates the queue by
queue_path(dot-separated path likeroot.platform.services.dev) - Adjusts:
resources.max.{vcore,memory}resources.guaranteed.{vcore,memory}
- Ensures
guaranteed <= maxafter changes
If a pod is Pending and has no declared requests/limits, this tool assigns defaults:
- CPU:
default_cpu_request_m(milliCPU) - Memory:
default_mem_request(parsed viakubernetes.utils.quantity.parse_quantity)
This is critical to avoid false “no demand” conditions.
A common failure mode: you have available queue headroom on paper, but the cluster cannot schedule the next pending pod because it needs a fresh node (fragmentation/binpack).
This tool estimates:
max_pend_cpu,max_pend_mem: largest pending pod “shape”pods_per_node: how many of that shape fit on a noderunning_pods_est: rough estimate of how many pods are already consuming the queue
If it detects a “perfectly filled” pattern with pending work waiting, it triggers conversion for at least one node.
The service runs two daemon threads:
Runs every check_interval_seconds:
trigger()
Computes needs per namespace and enqueues conversion tasks into a global queue.check_converted_nodes()
Finds idle converted nodes and reverts them.
Continuously:
- Checks global conversion cap
- Enforces cooldown
- Pops tasks from the global queue
- Performs node conversion (1 node per task)
- Updates YuniKorn capacity after nodes join
kubernetesPython clientpyyaml
/etc/dynamic-node/dynamic_node_config.ini
Main configuration./etc/dynamic-node/namespace_queue_paths.json
Map of namespace → YuniKorn queue path.
/usr/site/rcac/sbin/node-convert
Used to convert nodes between Slurm batch and Kubernetes.
node-convert --list # every Slurm node
node-convert --list --node-type a # only 'a' nodes
node-convert --list --node-name a001 # one node
Read-only: it runs the exact same eligibility logic the conversion path uses and prints a verdict per node, so you can see why a node was rejected.
NODE STATE IDLE VERDICT REASON
a001 IDLE 12h40m OK -
a008 ALLOCATED 0s PREEMPT job 6000001 (x-ob/pur230002) running 1h0m
a009 ALLOCATED 0s PREEMPT job 6000001 (x-ob/pur230002) running 1h0m
a006 IDLE 4d14h SKIP reservation maint_sep
a007 IDLE 2d14h SKIP planned for pending job 5510003
a003 IDLE+DRAIN 1d5h SKIP state flag DRAIN
a002 IDLE+PLANNED 5h40m SKIP state flag PLANNED
a004 ALLOCATED 40m SKIP not idle (ALLOCATED)
a005 IDLE 30s SKIP idle only 30s (< 600s)
a010 ALLOCATED 0s SKIP not idle (ALLOCATED)
Viable now: 1 of 10 node(s) matching prefix 'a'
Preemptible: 2 node(s) held by x-ob/pur230002
OK = usable right now. PREEMPT = busy, but the job holding it is preemptible
backfill. SKIP = off limits, with the reason.
When fewer nodes are viable than requested, --allow-preempt lets node-convert
free some by preempting jobs that match both PREEMPT_USER and
PREEMPT_ACCOUNT (default x-ob / pur230002). Nothing else is ever touched,
and preemption is off unless explicitly enabled.
These jobs are single-node by construction, and the script enforces that rather
than assuming it: a matching job spanning more than one node is reported and
skipped, never preempted. Killing a 4-node job to satisfy a 1-node request would
destroy four nodes' worth of work. --list names the job and its node count so
the skip is visible.
node-convert --set k8s --node-type a --num-nodes 2 --allow-preempt
Order of operations, which matters:
- Pick the youngest matching jobs first, so the least compute is lost.
PREEMPT_MAX_RUNTIME_SECONDSadditionally protects jobs past a given age. - Skip jobs that span more than one node or whose node is outside the requested type, and ping-check the node before destroying any work.
- Add the nodes to the
k8sreservation first (withIGNORE_JOBS, which allows reserving a busy node). Without this, the scheduler can hand the node to another job in the gap between the job exiting andk8s-convertrunning. SIGTERMthe job, waitPREEMPT_GRACE_SECONDS, then requeue it (PREEMPT_MODE=requeue, the default) or cancel it (PREEMPT_MODE=cancel). A job withRequeue=0is skipped rather than killed, unless mode iscancel.- Wait for the node to leave
COMPLETINGand reachIDLE, then convert. - If the conversion aborts anyway (too few reachable nodes), release the preempted nodes from the reservation. Otherwise they sit reserved and unschedulable with nothing running on them.
If PROTECT_LARGEST_PENDING=1 trips, no preemption is attempted either: a
starved large job should not be made worse.
Consider whether Slurm's native preemption (a preemptible QOS or partition plus
PreemptMode=REQUEUE) fits better. It is the supported mechanism and it lets the
scheduler make the choice; this flag exists for the case where those jobs are a
convention rather than a configured QOS.
A Slurm node is only taken if all of these hold:
- base state is
IDLEwith noPLANNED,RESERVED,DRAIN*,DOWN,FAIL*,MAINT,REBOOT_*,POWER*,NOT_RESPONDING,INVALID_REGorCOMPLETINGflag LastBusyTimeis at leastMIN_IDLE_SECONDSago- the node is not listed in
SchedNodesof any pending job (i.e. the backfill scheduler has not already earmarked it for a job that is waiting to start) - the node is not in a reservation that is active or starts within
RES_LOOKAHEAD_SECONDS(thek8sreservation itself is excluded)
Candidates are sorted longest-idle first. After the node joins the k8s
reservation its state is re-checked, and reservation membership is rolled back
if Slurm allocated it in the meantime.
Environment overrides:
MIN_IDLE_SECONDS(default600)RES_LOOKAHEAD_SECONDS(default86400)PROTECT_LARGEST_PENDING(default0, disabled) — when set to1, refuse to convert if doing so would leave fewer eligible idle nodes than the largest pending job requests.PREEMPT_ENABLED(default0) — same as passing--allow-preempt.PREEMPT_USER/PREEMPT_ACCOUNT(defaultx-ob/pur230002) — a job must match both to be preemptible.PREEMPT_MODE(defaultrequeue) —requeueorcancel.PREEMPT_GRACE_SECONDS(default60) —SIGTERMfirst, then wait this long.PREEMPT_MAX_RUNTIME_SECONDS(default0, no limit) — never preempt a job that has run longer than this.PREEMPT_DRAIN_TIMEOUT(default300) — how long to wait for a preempted node to reachIDLE.PASSWORD(no default, required for--set batch) — the racadm password used to power cycle a node on its way back to Slurm.--set batchrefuses to start without it, before anything is drained.
Note: the PLANNED node state requires Slurm 22.05 or newer; on older versions
the SchedNodes check carries most of the weight.
Every external command (scontrol, squeue, kubectl, pdsh, ssh/racadm)
is checked, and the script is built so that a failure never leaves a node in
neither pool:
- If Slurm or Kubernetes cannot be queried, nothing is converted. A node is never judged viable on the strength of a query that failed — an unreadable pending queue would otherwise make backfill-planned nodes look idle.
- The
k8sreservation is never rewritten from a membership list that could not be read, and it is never deleted just because reading it failed. - If
pdshfails part way through, the affected nodes stay drained and in the reservation (and so out of Slurm's reach) and the script names them, rather than handing half-converted machines back to the batch scheduler. - Selected nodes are drained with a
Reason(MARK_REASON, defaultbatch node selected for k8s conversion) sosinfo -Rexplains why they left the batch pool — a reservation carries no per-node note of its own, and Slurm only keeps aReasonon a node that is out of service. Any rollback that releases the reservation also clears the drain — otherwise the node would be neither reserved nor schedulable, andDRAINwould keep it out of future selection too. - On the way back to batch, each step gates the next: a node that could not be drained is not deleted, and a node that could not be power cycled is not resumed in Slurm and keeps its reservation. In a multi-node revert the nodes that succeeded are released normally and the failures are listed; the exit status is non-zero if any node failed.
Expected keys (section [settings]):
kubeconfig_path(optional)
If set, loads kubeconfig from this path. Otherwise uses default kubeconfig loading behavior.monitored_namespaces
Comma-separated namespaces to watch.reversion_idle_seconds
How long a converted node must be idle (no active pods) before revert.check_interval_seconds
Evaluate loop interval.max_converted_nodes
Global cap on how many nodes may be converted at any time.allow_preempt(defaultfalse)
Passes--allow-preempttonode-convert, letting it preempt single-node backfill jobs when no node is idle. See thenode-convertsection above.yunikorn_cm_namespace(default:yunikorn)
Namespace containing the YuniKorn ConfigMap.yunikorn_cm_name(default:yunikorn-configs)
Name of the YuniKorn ConfigMap containingqueues.yaml.yunikorn_queue_annotation(default:yunikorn.apache.org/queue)
Key used to identify queue assignment on pods (annotation or label).node_cpu_capacity(default:128)
vcores to add/subtract per converted node.node_memory_capacity(default:256G)
memory to add/subtract per converted node.global_cooldown_seconds(default:30)
Cooldown between conversions (note: present but not currently wired to advance_next_conversion_allowed_at).default_cpu_request_m(default:1000)
Used only when pending pods have no resource requests/limits.default_mem_request(default:2Gi)
Used only when pending pods have no resource requests/limits.
A JSON object mapping each monitored namespace to a YuniKorn queue path:
{
"workloads-dev": "root.platform.services.dev",
"workloads-prod": "root.platform.services.prod"
}This service supports two modes:
- monitor: run the continuous evaluate + actuator loops (normal operation)
- test: create a synthetic load Deployment in a namespace/queue (validation)
- A valid kubeconfig accessible to the process (either:
- set
kubeconfig_pathin/etc/dynamic-node/dynamic_node_config.ini, or - rely on default kubeconfig loading via
KUBECONFIG/~/.kube/config)
- set
/etc/dynamic-node/dynamic_node_config.iniis present and readable/etc/dynamic-node/namespace_queue_paths.jsonis present and contains queue mappings for all monitored namespaces- The process identity can:
- list nodes and pods
- read/patch the YuniKorn ConfigMap containing
queues.yaml - (test mode) create/patch/delete Deployments in the target namespace
- The external converter exists and is executable:
/usr/site/rcac/sbin/node-convert
Monitor mode starts two background threads:
- Evaluate loop: periodically computes demand and enqueues conversions; checks for idle nodes to revert
- Actuator loop: drains the conversion queue and performs
node-convertactions
python3 dynamic-node-manager.py --mode monitorTest mode creates a Deployment that schedules pods annotated to a specific YuniKorn queue. Each pod runs a long sleep to hold resources.
python3 dynamic-node-manager.py \
--mode test \
--namespace workloads-dev \
--queue root.platform.services.dev \
--replicas 200- Verify the manager is logging demand decisions and conversion actions via syslog.
- Confirm YuniKorn
queues.yamlis being updated (capacity changes show asMAX/GUARbumps in logs). - Confirm nodes join/leave the Kubernetes API after
node-convertruns. - Confirm
/etc/dynamic-node/converted_nodes.jsonis being updated as nodes are converted and reverted.