feat(compute): add Lambda MicroVM P3 approval lifecycle

[isadeks]

P3 lets an agent wait for human approval while its MicroVM sleeps or retires with a verified checkpoint. A later decision wakes the original worker or admits one replacement with the saved conversation and workspace. Unanswered approvals remain available by default; explicit deadlines remain supported.

Linear users can reply approve or deny under the approval comment. The service verifies the actual comment through Linear’s API, rejects bots and the saved OAuth token’s own identity, and applies the task owner’s decision to that request. CLI decisions work on all backends; sleep and replacement are MicroVM-specific.

Area

• cdk — infrastructure, handlers and permissions
• agent — runtime and image
• cli — task and approval settings
• docs — guides and design sources
• tooling — contracts, packaging and CI

Related

ADR-021 P3 for #645, including remaining P2 prerequisites. Related follow-ups: #817, #841 and #857. AgentCore remains the default compute backend.

Changes

• Add approval-aware suspend/resume supervision, guest activity barriers, explicit lifecycle connection closure, credential renewal and correlated failure diagnostics.
• Preserve unanswered requests, with cancellation feedback, a ten-minute default sleep delay and task/deployment controls.
• Verify versioned conversation/workspace checkpoints before retirement, confirm worker shutdown before releasing capacity, and restore the pending action on one admitted replacement.
• Move approval creation and non-human timeout writes behind a trusted IAM API. The worker’s session tag restricts its API path to its own task. Workers can read approval records but cannot overwrite human decisions, notification markers or retention fields. MicroVM writes also require the active worker lease.
• Require an explicit MicroVM layout until migration is verified: true for new/already-nested installations, false for existing flat installations. Omission fails synthesis. Group stateless scheduled maintenance in a separate nested stack while retaining existing tables and MicroVM identities. Both layouts are covered across all image, gateway and vault configurations by the 490-resource budget.
• Strengthen terminal cleanup, transaction-conflict diagnostics, SDK-version checks, bootstrap permissions and real DynamoDB transaction tests in CI. Preserve stored serialization bytes.
• Reuse the existing Linear AgentCore Identity vault integration for MicroVM workers.

Latest review fixes (bf7c271d)

• Configure both CLI/API decision functions with the concurrency table and the shared admission limit; Linear's separate wiring had hidden this gap.
• Clean terminal, fenced replacements that never obtained a launch receipt, checking matching continuation/slot/lease identities and no worker ID. DynamoDB Local tests drive failure through cleanup and reject a worker ID injected before the conditional write. Unknown launch outcomes retain their existing lifetime protection; valid pending checkpoints do not expire.
• Reproduce project-hook denial on pinned SDK 0.2.110 / CLI 2.1.191 using a synthetic loopback model. Retire only the tool ID in the CLI error result; do not clear all registrations at Stop.
• Preserve valid credentials on advisory refresh failures while keeping forced wake refresh mandatory. Default MicroVM uv installs to copies after reproducing Linux cache hardlinks. Confirm termination conflicts with GetMicrovm before treating cleanup as successful.
• Correct remaining timeout/stranded-request documentation, document all three live harness commands, and cover heartbeat grace/staleness boundaries. The checkpoint-code table and approval API's non-semantic validation wording were already present.

Local full build passed with DynamoDB Local and real SDK continuation probes enabled: 5,758 CDK tests (261 suites), 2,254 agent tests (86.84% coverage), and 1,005 CLI tests, plus lint, type checks, synthesis and documentation. The final agent diagnostic change passed another complete 2,254-test quality run. Final layout compatibility and heartbeat checks passed 25 tests, with CDK lint/compile passing. The documentation build and internal links passed across 82 pages; all three live-harness inspection commands ran without AWS calls.

Deployed bf7c271d to the retained nested fixture in us-east-1: CloudFormation UPDATE_COMPLETE, MicroVM image 3.0, AgentCore runtime 4, and coordinator version 5. Four deployed Lambda code hashes match the reviewed assets; approve/deny functions have the concurrency table and shared limit. Verified 26 baseline stateful resource identities were preserved. Latest-head GitHub checks passed.

Both live Cognito-authenticated API decision cases passed after a saved checkpoint, sleep, source-worker loss and PARKED: approval and denial started replacement workers in 11.7s and 12.3s. Approval performed exactly one Read; denial performed none and acknowledged the decision. Both tasks completed, their workers terminated, and scheduled cleanup closed leases and returned capacity to zero. These are the API routes used by CLI decisions; no interactive CLI command was used. Tests used an explicit 30-second sleep delay with a test-only policy, leaving the ten-minute default unchanged. The fixture configuration was restored exactly; final checks found no active tasks, open leases or held slots. Cleanup follows the five-minute reconciliation schedule. This already-nested update does not verify flat-to-nested migration. Earlier results below retain their recorded revisions.

Scope and remaining tradeoffs

This PR includes the trusted approval writer, Linear replies, retained-request default, shared maintenance stack and MicroVM layout selection. Maintainer agreement on this expanded scope and the proposed ADRs remains open. The @aws-sdk/client-ssm dependency supports the runtime suspend setting; build-workflow changes make DynamoDB Local and the pinned SDK recovery probes required in CI. These are intentional dependency/CI changes, not incidental cleanup.

Nonblocking optimization remains: eager transcript append repeatedly copies/encodes history (a local synthetic 1,000-frame run took about 5.5 seconds cumulatively). Replacing checkpoint readback with HeadObject must preserve checksum/version validation and recovery after a lost upload reply. Broad type/handler refactors and restored-scope revalidation are deferred rather than mixed into lifecycle corrections. Known launch rejections still share the conservative unknown-outcome capacity hold; earlier release needs a durable, operation-specific rejection classification.

Validation

Review validation at 159576e6 (eight focused commits after the third review):

• Complete final-head GitHub build passed: 5,718 CDK tests, 2,231 agent tests (zero skips; 86.66% coverage), and 1,005 CLI tests. Includes real DynamoDB transaction checks and the real SDK approve/deny recovery probe, explicitly enabled in the build workflow. The full local build also passed before the final worker-token validation restoration, which passed its focused tests and this final full CI run.
• Cancellation-race tests exercise both actual pipeline paths. Restoring the false failure behavior produces two failing tests; bypassing both checked terminal writes produces three. Internal-whitespace/case replay tests reject an over-broad normalizer.
• Expanded flat/nested resource-budget, cdk-nag and bootstrap coverage remains in the full suite. Both layouts retain their original budget; widest configuration is below 490 resources.
• Final correction restores the original worker_attempt_id validator; 22 broker unit tests passed. Source tracing shows this field carries the platform launch/lease token, not AWS’s physical MicroVM identifier.
• A live probe using the worker's actual AgentCore Identity vault token successfully edited its own control comment, but Linear rejected editing a human-authored deny comment. The human body and author stayed unchanged. This verifies the current app-token installation; no historical approval threads were modified.
• The documentation build validates internal pages and anchors across 82 rendered pages, plus local existence of emitted GitHub repository-file targets. An injected missing repository target is rejected.
• Security/dependency/workflow, dead-code and PR-title checks passed at 159576e6. The silent-failure scan and its rule tests pass. Lint, type checks, contracts, documentation and infrastructure synth pass. The full build and all PR checks passed at that revision.
• Deployed 159576e6 successfully to the retained isolated fixture after pausing submissions and draining tasks, leases and capacity. Verified the new approval API, AgentCore runtime version 3, and MicroVM image 2.0 against the built image/artifact references. Seven live IAM simulation checks passed: own-task invocation/read allowed; other-task or untagged invocation and direct approval writes denied. The borrowed Linear vault and shared logging settings remain unchanged and functional.
• All six latest-image Linear coding-agent flows passed: AgentCore approve/deny, MicroVM same-worker wake approve/deny, and MicroVM replacement-worker recovery approve/deny. Verified actual owner-authored replies, matching decision sources and thread acknowledgments, approved reads exactly once, denied reads never succeeding, continued agent execution, terminated MicroVMs and scheduled capacity release. All four MicroVMs waited the full default 600 seconds before suspension; normal wake cases were observed asleep for over a minute. Replacement cases used controlled loss of a suspended worker with a verified checkpoint, then restored on different workers. No approval or continuation rows were manufactured. A separate automated wake probe measured 69.031 seconds between the suspend response and resume hook, inside the earlier 5–90 second failure window; resume returned HTTP 200 in 143 ms. That task also completed on the same worker. All seven test slots are released and active capacity is zero.

Live stress validation and capacity fix at 7384f9b0:

• A 12-task burst respected the 10-task cap, queued overflow, handled queued/startup cancellation and deliberate worker termination, and passed 50 concurrent status reads. Nine ordinary workloads completed without approval gates.
• Stress testing exposed simultaneous slot releases failing with DynamoDB TransactionConflict. Bounded retries preserve the exact transaction, idempotency token and worker-lease conditions. An isolated live reproduction went from 6/12 successful releases to 12/12; concurrent admission still accepted exactly 10/12. All 71 focused unit/transaction tests and the full local build passed: 5,723 CDK, 2,231 agent, 1,005 CLI tests.
• Deployed the fix and verified the actual Lambda packages. Two workers each passed 1 GiB memory, 512 MiB disk and 45 CPU-second benchmarks; a third was cancelled during its Python workload. All test workers terminated, leases closed and capacity returned to zero. The patched cleanup sweep reported no failures. Tests and reproducible harnesses remain private; no benchmark PRs were published.

The later main merge at 2a79c0e retains this fix; its CI is separate from the live deployment above.

Earlier live approval-boundary validation used the actual approval service and Python signing client: 34 AWS checks covered creation, field rejection, task/tag isolation, denied direct writes, timeout races and MicroVM lease fencing; nine IAM simulation checks also passed. These were isolated API tests with task records for all three backends, without coding-agent images.

Earlier coding-agent tests additionally covered explicit expiry/cancellation and AWS access after credential expiry. Those earlier configurations remain separate from the latest-image cases above. Retained Linear fixtures and reproducible integration harnesses stay outside this public PR.

Before marking ready

• Review-baseline GitHub checks complete (159576e6); full local build and live stress validation passed for capacity fix 7384f9b0.
• GitHub checks passed for the subsequent main merge (2a79c0e).
• GitHub checks passed for the nested-default change (35f05bbb).
• Complete GitHub checks for the latest review-fix head (bf7c271d).
• Obtain maintainer acceptance of expanded feat(compute): add AWS Lambda MicroVMs as a ComputeStrategy backend #645 scope, ADR-021/ADR-023 and contributor assignment.
• Verify API approve/deny decisions on PARKED tasks in the latest review-fix deployment (the same routes used by CLI).
• Deploy and verify the trusted approval service with matching worker images after draining existing tasks.
• Complete the six latest-image Linear coding-agent approval/recovery cases.
• Complete the compatibility review: retain version-1 checkpoint field names and the distinct migration/runtime hash encodings so saved data remains readable.
• Complete reusable flat-to-nested migration commands and independently verify upgrading a current-main deployment.

Review decisions and limits

ADR-023 records the approval writer, rollout and retained-request decision. No default approval expiry is deliberate; zero is the no-expiry sentinel and the positive timeout range starts at 30 seconds. Worker/resource lifetime remains separate. A public SigV4 endpoint is intentional, with task-path IAM authorization.

This protects approval-record writes, not complete isolation from compromised ambient compute credentials: those credentials can still select session tags, and legacy channel bundles still include the webhook signing key. Action descriptions and policy metadata are worker assertions. TIMED_OUT currently also includes worker polling failures and is not proof that a human deadline elapsed. These limits are documented rather than presented as solved by the writer API.

The normal pipeline propagates failed terminal writes, while a superseding cancellation is benign and does not produce a false failure reaction. Actual pipeline entry points and the outer runtime's unhealthy response after a failed backup write are tested. Pure transaction conflicts get bounded retries; failed ownership checks do not. The continuation scan cursor uses compare-and-set to preserve progress when old and new schedules overlap during upgrade. Existing orchestrator heartbeat/backend monitoring handles lost running workers; a second broad RUNNING reaper was not added. Checkpoint field renames or hash-encoding consolidation require a versioned migration, not a style-only edit.

Deployment notes

Re-bootstrap to bundle 1.9.0 or later. For the approval permission change, pause submissions and drain existing tasks, including suspended workers and retained continuations; deploy CDK and the matching rebuilt image together. Old workers lose direct approval-write access and fail closed at their next gate. See the upgrade procedure.

Automatic suspension remains behind the deployment compatibility gate, which defaults off. Once enabled, the default sleep delay is ten minutes. MicroVM deployments must now explicitly select microvm_nested_stack: use true for new/already-nested installations and retain false for existing flat installations. Omission fails synthesis. Explicitly selecting true does not migrate a flat deployment. Regenerate old cloud assemblies before deploying. Portable migration and independent upgrade acceptance remain open.

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution under the terms of the project license.

[ayushtr-aws]

Review at head 548bdf48 — Request changes

The engineering here is unusually rigorous, and the security boundary work (trusted approval writer, Linear consent read-back, payload bootstrap) survived an adversarial security pass with no findings above threshold. But the PR ships two verified correctness defects on the new wake path, one unbounded storage leak, an unsafe-by-default layout change with unfinished migration tooling, and a governance gap.

Vision alignment

• Tenet 1 / Tenet 5 — advanced. Approvals survive worker retirement, decisions stay owner-attributed, workers lose direct approval-write access, and the session role can only invoke /tasks/${aws:PrincipalTag/task_id}.
• Tenet 6 — traded. approval_timeout_s default 300 → 0 applies to all backends, so an unanswered gate on AgentCore/ECS bills to the worker lifetime ceiling. Documented only in ADR-023, which is proposed.
• Tenet 4 — traded. microvm_nested_stack defaults to true with an advisory warning (see B2).
• ADR-003 — violated. See B1. ADR-021 and ADR-023 both remain proposed and need maintainer acceptance as part of merging.

Blocking

B1 — Scope and governance (ADR-003). #645's acceptance criteria cover a MicroVM ComputeStrategy. This PR also delivers a new IAM-authenticated approval-writer API, native Linear approve/deny replies (~1,000 lines across 8 modules), a repo-wide change to the default approval expiry, a ConcurrencyMaintenance nested stack that affects every deployment including default AgentCore, and a nested-by-default MicroVM layout. None has an approved issue (#793 separately wants the ADR-023 number). #645 is assigned to @dreamorosi, not the author. Your own first self-review flagged this. Please file approved issues for the approval-writer API, Linear reply approvals, and the maintenance-stack split (or re-approve #645 with widened deliverables), and get assigned.

B2 — cdk/src/stacks/agent.ts:361 nests MicroVM by default while migration is unfinished. The checklist item "Complete reusable flat-to-nested migration commands and independently verify upgrading a current-main deployment" is unchecked. cdk/src/migration/permissions.ts and template.ts are referenced only by tests, with no script, mise task, or doc entry point. The synth warning at :364-372 does not block a cdk deploy that can delete payload/artifact buckets. Fix: when compute_type=lambda-microvm and microvm_nested_stack is unset, throw until migration tooling lands and is verified; then downgrade to a warning.

B3 — cdk/src/constructs/task-api.ts:1460-1471 never sets USER_CONCURRENCY_TABLE_NAME on the approve/deny Lambdas. enableMicrovmContinuations adds only CONTINUATION_BUCKET_NAME and ORCHESTRATOR_FUNCTION_ARN, but approve-task.ts:267 / deny-task.ts:241 → wakeMicrovmAfterApproval → dispatchMicrovmContinuation:63 → admitContinuation, which reads process.env.USER_CONCURRENCY_TABLE_NAME! (microvm-continuation-start.ts:32). A CLI/API decision on a PARKED task runs a TransactWrite with TableName: undefined, is caught as wake-reconciliation-failed, and the replacement launches only when the 5-minute reconciler sweep reaches the row. linear-integration.ts:330 sets the variable, which is why the six Linear live flows passed. Fix: set USER_CONCURRENCY_TABLE_NAME and MAX_CONCURRENT_TASKS_PER_USER in enableMicrovmContinuations, and add a CDK assertion on both decision functions' environment.

B4 — cdk/src/handlers/reconcile-microvm-continuations.ts:72-85 throws forever for a replacement that failed before launching. admitContinuation writes the lease ACTIVE (microvm-continuation-start.ts:101); the runner can throw MICROVM_CONTINUATION_VERSION_CHANGED (microvm-continuation-runner.ts:181) before any start receipt exists; the catch at :262-269 calls failContinuationAttempt, which moves the lease to FENCED (:85-92). For a terminal task with no microvm_start, the reconciler gate accepts only PARKED/CLOSED or ACTIVE-with-slot-held, so a FENCED lease throws MICROVM_CONTINUATION_LEASE_INVALID every sweep and deleteClosedTaskContinuations at :108 never runs. continuation-bucket.ts:39-41 has no object expiration, so up to 1 GiB per affected task is retained indefinitely behind a "will retry" warning. Fix: treat FENCED with no lease_microvm_id as never-launched (or have failContinuationAttempt write CLOSED when no handle exists), and add a lifecycle expiration to the bucket as defence in depth.

B5 — agent/src/hooks.py:1978-1979 leaks tool registrations when a second PreToolUse denier fires. _pre calls lifecycle.tool_finished only on the non-allow path. Repo tasks load setting_sources including project (runner.py:428), so a target repo's .claude/settings.json can register a PreToolUse hook or deny rule that blocks a tool the Cedar hook allowed. If the CLI then returns an error tool result without emitting PostToolUse/PostToolUseFailure, the id stays in MicrovmLifecycle._tools, and suspend() (microvm_lifecycle.py:355) / continuation_checkpoint() (:262) refuse for the rest of the task. Confidence note: the code path is verified; the CLI behaviour (no Post* event when a later hook denies) was checked by an automated pass against CLI 2.1.258 but not reproduced by me. If the CLI does emit a failure event there, this drops to a hardening nit. Either way, evicting stale ids at the Stop hook plus a test for the "no Post* hook arrives" case is cheap.

Non-blocking

Correctness / operations

• task-concurrency.ts:172-178 + reconcile-microvm-continuations.ts:63-67: a known RunMicrovm 4xx rejection after receipt and lease exist holds the seat for the full 8-hour lifetime with no log. Replacement launches only. Record the rejection on the receipt so release is immediate.
• request-approval.ts:95-99: the lease ConditionCheck is unconditional for lambda-microvm; a task RUNNING before this deploy has no lease row and fails closed at its first gate. Mitigated by the documented drain step; say so in the error string.
• aws_session.py:388-389: advisory refresh_needed() escalates to _protected_refresh(is_mandatory=True) on a 1-attempt/2 s STS client, so a transient STS stall becomes a broker 503 while cached keys are still valid.
• continuation_workspace.py:222: rejecting st_nlink != 1 will fail every checkpoint if uv hardlinks site-packages into the workspace. Set UV_LINK_MODE=copy in the image or relax the check.
• lambda-microvm-strategy.ts:918-943: ConflictException on Terminate is now unconfirmed and emits ERROR + a task event on the routine cancel/finalize race; confirm via GetMicrovm first.
• continuation_runtime.py:239-242: approval_scopes are seeded from the worker-written manifest without cross-checking APPROVED rows. Not a new boundary crossing, but validate against the approvals table on restore.
• server.py:536 still discards the write_terminal outcome on the outermost crash path (noting your decline for the record).

Efficiency

• continuation_session.py:258-270: eager append deep-copies and re-encodes the whole conversation per frame — O(n²) on the SDK loop.
• continuation_storage.py:405: re-downloads the tar it just uploaded to learn VersionId; use HeadObject with ChecksumMode=ENABLED, as the coordinator already does at microvm-continuation-storage.ts:229.

Design / clarity

• microvm-start.ts:36-42 StartReceipt is private and re-declared inline in three modules; TaskRecord lacks microvm_start. Export and share.
• microvm-continuation-types.ts:42-53 ContinuationRecord has state-dependent optionals; make it a discriminated union like ApprovalRecord in this same PR.
• approve-task.ts:231-290 and deny-task.ts:208-265 are a ~60-line duplicate.
• contracts/constants.json approval_timeout_s has default: 0 outside min: 30, and the sync script was relaxed to allow it. A no_deadline_sentinel field would make the schema self-describing.
• approval-request-service.ts:89 cdk-nag reason still reads as semantic validation.
• New dependency @aws-sdk/client-ssm and edits to .github/workflows/build.yml are "ask first" items per AGENTS.md; both look justified but should be called out in the body.

Documentation

• Starlight mirror regenerated locally with mise //docs:sync: zero drift. ✅
• Bootstrap 1.6.0 → 1.9.0 with rationale; artifacts regenerated locally with zero drift; template body 46,725 bytes. DEPLOYMENT_ROLES.md golden baseline updated. ✅
• Stale: docs/guides/QUICK_START.mdx:483 still says "300 s by default". CEDAR_HITL_GATES.md:199-213, :485, :2129-2133, :2515 still describe built-in rules carrying 300/600 s timeouts that builtin-policies.ts:141-179 no longer declares. types.ts:1394-1395 says the stranded reconciler leaves approval rows PENDING, but reconcile-stranded-tasks.ts:297 cancels them.
• Missing: the six checkpoint code values; a documented runner for cdk/test/live/*.live.ts; any entry point for cdk/src/migration/.

Tests & CI

• All four checks green at head. test/bootstrap run locally: 140 tests pass; synth-coverage covers both MicroVM layouts and asserts AWS::SSM::Parameter and AWS::Lambda::MicrovmImage. Types/constants sync scripts pass; the only relaxation is the deliberate default === 0 sentinel.
• #366: no bundling-stacks re-enable; large synths in beforeAll. Env-gated DynamoDB Local / SDK probe tests are fail-closed under CI=true and wired in build.yml. ✅
• R1/R2 from your third pass are closed: _persist_finished_task raises only on FAILED, and test_terminal_persistence_through_task_entry_point drives run_task end to end.
• Gaps: agent-heartbeat.ts has zero direct test references for its boundary arithmetic; approval-request-service.test.ts has no negative case; nothing asserts approve/deny Lambda environment (would have caught B3); nothing covers the B5 "no Post* hook" path.

Human heuristics

• Proportionality — concern. Core lifecycle work is proportionate. cdk/src/migration/* (test-only) and the approve/deny duplication add surface without payoff yet.
• Coherence — concern. Two vocabularies for one lifecycle (lease_state vs ContinuationRecord.state), dual attempt_id meanings, TaskStatus literals in microvm-worker-lease.ts:35-36. Acknowledged where wire-format bound.
• Clarity — concern. B4's perpetual "will retry" warning and the silent seat hold both hide a permanent state behind a transient-looking message.
• Appropriateness — pass. Integration code verified against real AWS and Linear; real-DynamoDB tests in CI; ADR-023 names residual risks honestly.

Suggested order

B3 and B4 are small, mechanical fixes. B5 is a Stop-hook eviction plus one test. B2 is a one-line throw. B1 is the maintainers' call and should be settled before re-review.