[Shichun Feng (Sean) (KeyOfSpectator)]

Operators who know which microVM actors to watch can plot guest CPU and memory using the existing Prometheus scrape of atelet’s /metrics. The existing ate_actor_stats_* series still serve dashboards grouped by template and worker pool.

Set --debug-actor-guest-metrics-uids to expose the new series for up to 16 actor UIDs per atelet. An empty list disables them. The series reuse cached guest samples and are visible to readers of the scrape endpoint when enabled.

ate_debug_actor_guest_cpu_usage_cores{actor_atespace="demo",actor_name="runner-17",actor_uid="c7f3fb50-ef2d-4075-b59a-4a8e192e48a0"} 0.75
ate_debug_actor_guest_memory_usage_bytes{actor_atespace="demo",actor_name="runner-17",actor_uid="c7f3fb50-ef2d-4075-b59a-4a8e192e48a0"} 402653184
ate_debug_actor_guest_memory_working_set_bytes{actor_atespace="demo",actor_name="runner-17",actor_uid="c7f3fb50-ef2d-4075-b59a-4a8e192e48a0"} 335544320

CPU appears after two distinct samples in the same activation. For spikes that predate the allowlist, retained ate.actor.usage_sampled events can provide the history. Refs #3792.

Passed go test -race ./cmd/atelet ./internal/serverboot -count=3 -shuffle=on. The full local suite had failures in other packages. The registry check needs Weaver or Docker.

AI use: Codex helped prepare the patch, tests, and PR description.

• Docs updated

[Shichun Feng (Sean) (KeyOfSpectator)]

Tim Bai (@baizhenyu) Krisztian F (@krisztianfekete)
Hi, could you take a quick look at the design in this draft? It reuses the existing guest samples and exposes metrics for explicitly selected actors on a separate, opt-in debug endpoint. Does that fit the project's approach to metric cardinality?

In real-world use cases, we’d like to add actor metadata labels to guest CPU and memory metrics so users can tell which actor each time series belongs to.

[Shichun Feng (Sean) (KeyOfSpectator)]

Krisztian F (@krisztianfekete) Could a maintainer approve the pending workflows on this PR so the e2e checks can run? Thanks!

[Shichun Feng (Sean) (KeyOfSpectator)]

Thanks, Krisztian F (@krisztianfekete). I misunderstood the use case when I proposed a separate debug endpoint. I’ve updated the PR so our existing Prometheus scrape of atelet’s /metrics can collect a separate set of per-actor guest CPU and memory metrics. The new series are off by default and limited to 16 explicitly configured actor UIDs per atelet. The existing aggregate metrics continue to serve the current dashboards.

We want to look back at the CPU and memory history of individual actors during a workload run. For example, in reinforcement learning, we want to see how much guest CPU and memory an actor running a rollout worker used during a particular phase of training. We can join the resource series with our workload metadata using actor_atespace and actor_name to identify the workload and its role. actor_uid lets us distinguish actors recreated under the same name.

You’re right that the allowlist has to be configured before the run to build that history in Prometheus. For an actor we didn’t select beforehand, we can look back at retained ate.actor.usage_sampled events. Would the revised opt-in design fit the project’s cardinality policy?