fix(ssh): persist sandbox host identities

[quocanh261997]

Summary

Sandbox SSH host keys currently change when the supervisor restarts, so clients cannot reliably recognize the same sandbox. Give each sandbox ID one Ed25519 host key and expose its public SHA256 fingerprint. Keep the private key in the configured gateway credential store and deliver it only to the supervisor.

Related Issue

Closes #3835

Replaces #4027, which the vouch check closed before contributor approval. The contributor is now vouched.

Changes

• Merge current main at d1e8f44a0 and resolve the supervisor startup conflict. Keep the selected backend setup and startup tracing from main, preserve the assigned SSH host key, and update the two new supervisor test inputs for the host-key field.
• Wait for confirmed sandbox deletion before reusing its name in the SSH identity E2E test. Check the delete command succeeds and retain a bounded failure if cleanup stalls.
• Merge current main into the branch, resolve the compute and schema conflicts, and update the SSH handshake test for russh 0.63. Preserve provisioning ownership and cancellation-safe failed-create cleanup. Use the database record version when deleting sandbox-owned credentials and the parent record.
• Route failed SSH-key preparation and rejected sandbox creation through the shared cleanup path. Remove the sandbox and key when cleanup succeeds; retain a durable Deleting record when cleanup needs a retry. Keep cleanup running if the caller disconnects, including while waiting for the cleanup lock.
• Persist a sandbox-owned credential handle and fingerprint. Keep identity across stop/start, automatic restarts, and gateway recovery; remove credentials during deletion. Retain cleanup ownership when credential deletion fails and finish staging if a request is interrupted.
• Require the managed supervisor to use its assigned host key. Reject missing, invalid, or mismatched key material instead of replacing an established identity.
• Return the fingerprint on Sandbox and SSH-session responses, sandbox JSON output, and Rust, Python, Go, and TypeScript sandbox references.
• Document identity lifetime, authenticated fingerprint lookup, workload isolation, and matching runtime releases. CLI/TUI automatic verification remains the optional follow-on from the issue.

Testing

Verification after merging main at d1e8f44a0:

• Supervisor unit tests — 153 passed, including the new backend startup and trace tests.
• cargo test -p openshell-server --features test-support — 1,994 unit tests passed, 1 ignored; integration tests passed.
• Supervisor process, TUI, and VFIO tests — 102, 92, and 59 passed respectively.
• Content-guard example — all 16 tests passed.
• Python — 190 tests passed; TypeScript — 142 passed; Go checks passed.
• Rust lint and compile checks passed. The mise run pre-commit hook also passed.
• Live Docker conformance — all six scenarios passed.
• Live SSH identity E2E — one passed, no failures or skips. Get/List matched the presented fingerprint. Pinned OpenSSH connections worked after stop/start and managed gateway restart. The workload could not read supervisor authentication. Delete/recreate under the same name produced a new fingerprint.

mise run ci is not green locally: proxy::tests::mediated_connect_keeps_workload_bytes_read_with_the_synthesized_header again timed out waiting for a local TCP request. This PR does not change that test. The gateway and remaining SSH-related tests were then run separately and passed. The timeout's root cause remains unresolved.

Rust checks used LIBRARY_PATH=/opt/homebrew/lib for Homebrew Z3 on macOS. The live Docker checks used an ignored copy of the current launcher with host.docker.internal as the supervisor endpoint and the temporary mTLS gateway bound on all interfaces. The copy retains the runtime-image setting added on main and is excluded from the PR. Rootless Podman and Kubernetes were not exercised locally in this update.

GitHub checks must run against the new commit. Earlier verification remains in the PR comments.

Checklist

• Follows Conventional Commits.
• Commits are signed off (DCO).
• Architecture docs updated where applicable; public behavior and related operating skills are documented.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[quocanh261997]

Local verification evidence

The saved local run passed all six Docker CLI conformance scenarios and the SSH identity E2E test. This report covers the working-tree implementation subsequently committed as e5de0a892813408bf08a36d93ab33b66a9929d18, the PR’s initial implementation commit. Updated verification for the cleanup fix is recorded in this later comment.

Environment: macOS on ARM64, Docker Desktop. Live test run: October 1, 2026. The excerpts below come from the saved test output.

What the live SSH test verified

Check	Result
Sandbox Get and List expose the same public fingerprint	Passed
ssh-keygen reports the same fingerprint for the key actually presented over SSH	Passed
The workload cannot read /.openshell/supervisor/auth.json	Passed
Stop/start preserves the fingerprint and an OpenSSH connection with StrictHostKeyChecking=yes succeeds using the previously saved host key	Passed
Managed gateway restart preserves the fingerprint and the same pinned SSH connection succeeds	Passed
Delete/recreate under the same name produces a different fingerprint	Passed

The test source at this commit contains these assertions. The gateway restart branch ran; its skip message is absent from the saved output.

Actual SSH test output:

running 1 test
test ssh_host_identity_survives_restarts_and_changes_after_recreation ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.25s

Supporting checks

• Docker CLI conformance: smoke, sandbox-lifecycle, file-transfer, mechanistic-proposal, new-hostname-proposal, and policy-local each returned "passed": true; the overall report also returned "passed": true.
• mise run ci completed successfully. Its gateway test summary was 1882 passed; 0 failed; 8 ignored. The seven new SSH credential tests below all passed.
• mise run pre-commit completed successfully before publishing the PR.

Actual SSH credential test output

test ssh_identity::tests::cancelled_prepare_finishes_before_deletion_and_cannot_recreate_identity ... ok
test ssh_identity::tests::concurrent_candidates_keep_one_resolvable_identity ... ok
test ssh_identity::tests::deletion_retries_before_releasing_key_ownership ... ok
test ssh_identity::tests::identity_survives_reload_and_is_never_public ... ok
test ssh_identity::tests::losing_candidate_cleanup_failure_remains_owned_for_deletion ... ok
test ssh_identity::tests::default_credential_store_persists_key_across_runtime_reconstruction ... ok
test ssh_identity::tests::published_identity_is_never_replaced_when_storage_is_inconsistent ... ok

Credential test source

Reproduction and scope

The standard focused Docker command is:

OPENSHELL_E2E_DOCKER_TEST=ssh_host_identity mise run e2e:docker

On this Mac, the stock launcher's container-side 127.0.0.1 endpoint could not reach the host gateway. The successful run used an ignored copy of e2e/with-docker-gateway.sh, with just this Darwin-specific addition after the existing container-network check:

if [ "$(uname -s)" = "Darwin" ]; then
  GATEWAY_BIND_IP="0.0.0.0"
  SUPERVISOR_GATEWAY_HOST="host.docker.internal"
fi

The gateway still used mTLS. The successful local invocation was:

LIBRARY_PATH=/opt/homebrew/lib \
OPENSHELL_E2E_DOCKER_TEST=ssh_host_identity \
OPENSHELL_CONFORMANCE_BIN="$PWD/target/debug/openshell-conformance" \
mise exec -- bash plans/3835-with-docker-gateway-macos.sh \
  bash e2e/rust/e2e-docker.sh __openshell_run_docker_e2e

The launcher adjustment is excluded from the PR. This evidence covers local Docker verification; Kubernetes pod rescheduling and other driver E2E lanes were not exercised. GitHub runner validation is still pending.

[drew]

gator-agent

PR Review Status

Thanks @quocanh261997 for documenting the current-head Docker and SSH verification. I checked the full patch and the creation/cleanup paths those tests exercise; one PR-owned failure path can leave a sandbox record and its host-key credential behind after create reports an error.

Action required: make every post-commit identity failure either fully remove the sandbox and key or durably leave the sandbox in a retryable deletion state.

Blocking findings:

• GATOR-e5de0a89-01: see the inline Warning on the post-commit identity preparation call.

Carried findings:

• None

Gator metadata

• Validation: project-valid through linked, validated issue #3835
• Docs: sandbox identity behavior is documented under docs/; related operating skills are updated
• Checks: DCO and vouch gates are green; required branch workflows have not been dispatched for this head
• E2E: test:e2e is required for sandbox lifecycle and credential-flow changes, but dispatch waits until blocking review feedback is resolved
• Head SHA: e5de0a892813408bf08a36d93ab33b66a9929d18
• Base SHA: 76cfd0e31d5e1633db7ccd86ad9023ef7a2461b2
• Merge base SHA: 021400be8af471f8669369e679de3e18cf0bd672
• Patch ID: accf65c51eed8be52b1848efb2eb9bfbfe089f99
• Gator payload: 10
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[quocanh261997]

🏗️ build-from-issue-agent

Local verification after the failed-create fix

Tested the working tree now committed as a645d9fd5d61993ed8c8f3d848955e38f0e81be5 on October 2, 2026, using macOS ARM64 and Docker Desktop. This covers the fix for the failed-create cleanup finding.

The six new regression tests all passed. They inject failures to check that:

• A failed SSH credential save leaves no sandbox or key, and the name can be reused.
• If key deletion fails after the driver rejects creation (AlreadyExists, FailedPrecondition, or another error), the durable sandbox stays Deleting. Reconciliation then removes the row and key.
• Failed fingerprint publication removes the staged key and sandbox.
• A backend cleanup error does not restore the failed sandbox to Provisioning.
• Cleanup finishes after the caller is canceled, both during driver deletion and while waiting for its lock.

test compute::tests::ssh_credential_delete_failure_keeps_failed_create_deleting_until_reconciliation ... ok
test compute::tests::ssh_credential_store_failure_compensates_create_and_releases_name ... ok
test compute::tests::ssh_failed_create_backend_cleanup_error_does_not_restore_provisioning ... ok
test compute::tests::ssh_failed_create_compensation_survives_cancellation_while_waiting_for_lock ... ok
test compute::tests::ssh_failed_create_compensation_survives_request_cancellation ... ok
test compute::tests::ssh_fingerprint_persistence_failure_compensates_staged_identity ... ok

Full LIBRARY_PATH=/opt/homebrew/lib mise run ci passed, including workspace Rust tests, gateway tests (1888 passed; 0 failed; 8 ignored), Python (258 passed), TypeScript (142 passed), Go checks, and formatting/lint/compile checks. mise run pre-commit passed. The eight ignored gateway tests are existing suite exclusions; all six added regressions ran.

E2E Test Attestation

Gateway mode: Docker. Every live test/scenario executed by the focused launcher passed:

Test / scenario	Result
CLI conformance smoke	Passed
CLI conformance sandbox-lifecycle	Passed
CLI conformance file-transfer	Passed
CLI conformance mechanistic-proposal	Passed
CLI conformance new-hostname-proposal	Passed
CLI conformance policy-local	Passed
ssh_host_identity_survives_restarts_and_changes_after_recreation	Passed

test ssh_host_identity_survives_restarts_and_changes_after_recreation ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 22.33s

The SSH test checks the published fingerprint against the key actually presented, workload read restrictions, a pinned OpenSSH connection after stop/start and a managed gateway restart, and a different key after delete/recreate. The gateway restart branch ran; no SSH test was skipped.

Command used:

LIBRARY_PATH=/opt/homebrew/lib \
OPENSHELL_E2E_DOCKER_TEST=ssh_host_identity \
OPENSHELL_CONFORMANCE_BIN="$PWD/target/debug/openshell-conformance" \
mise exec -- bash plans/3835-with-docker-gateway-macos.sh \
  bash e2e/rust/e2e-docker.sh __openshell_run_docker_e2e

As in the earlier evidence, the ignored launcher copy uses host.docker.internal for the supervisor endpoint and binds the temporary mTLS gateway on all interfaces so Docker Desktop containers can reach the macOS host. This launcher adjustment is excluded from the PR. The launcher's optional static-linkage check was skipped because readelf is unavailable on this Mac. Kubernetes rescheduling and other driver E2E lanes were not run locally.

[github-actions]

Label test:e2e applied, but pull-request/4094 does not exist yet. A maintainer needs to comment /ok to test a645d9fd5d61993ed8c8f3d848955e38f0e81be5 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[drew]

/ok to test a645d9f

[drew]

gator-agent

PR Review Status

Thanks @quocanh261997. I checked your cleanup update against the prior failed-create finding: post-commit identity failures and driver-create rejections now share compensation that removes the sandbox and key when possible, retains a durable Deleting record when cleanup must retry, and continues cleanup after caller cancellation. The added regressions cover the requested credential-store and key-deletion failures. No blocking code-review findings remain.

Blocking findings:

• No blocking findings remain

Carried findings:

• GATOR-e5de0a89-01: resolved by this head; the Gator-owned thread has been closed

Required test dispatch is not complete yet. Gator applied test:e2e and posted /ok to test for the current head; the contributor mirror must be created before the E2E label can be re-applied and the required workflows confirmed queued.

Gator metadata

• Validation: project-valid through linked, validated issue #3835
• Docs: sandbox identity behavior is documented under docs/; no additional docs change is required for this cleanup-only delta
• Checks: DCO is green; required current-head branch workflows are awaiting contributor-mirror dispatch
• E2E: test:e2e is required; mirror creation was requested with /ok to test a645d9fd5d61993ed8c8f3d848955e38f0e81be5 and workflow queue confirmation remains pending
• Head SHA: a645d9fd5d61993ed8c8f3d848955e38f0e81be5
• Base SHA: 76cfd0e31d5e1633db7ccd86ad9023ef7a2461b2
• Merge base SHA: 021400be8af471f8669369e679de3e18cf0bd672
• Patch ID: 3b6a5b4b21488f22e5a5a68154f7db415e0e635c
• Gator payload: 10
• Review mode: follow_up
• Previous reviewed SHA: e5de0a892813408bf08a36d93ab33b66a9929d18
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[drew]

/ok to test a645d9f

[quocanh261997]

Fixed the rootless Podman SSH test in 7800f2f3fd34deccbd7dab62533e48779acbcd19.

The failed job reached delete/recreate, then failed with sandbox already exists. Deletion can be accepted before removal finishes, while the original name remains reserved. The test now checks the delete command succeeds and waits until the gateway no longer lists the sandbox before reusing its name. The wait has a two-minute limit and fails if cleanup does not complete. Host-key comparisons remain intact.

Verification:

• Podman-feature lint of ssh_host_identity: passed.
• mise run pre-commit, including the commit hook: passed.
• Live Docker conformance: all six scenarios passed.
• Live SSH identity E2E: one passed, no failures or skips. Covered fingerprint matching, pinned SSH after stop/start and managed gateway restart, workload access restrictions, and a different fingerprint after same-name recreation.

This update changes only the E2E test. Rootless Podman execution of the updated test still needs GitHub CI; it was not run locally on macOS. GitHub Branch Checks passed on the preceding commit. The PR description now separates the latest verification from the earlier results.

GitHub currently reports Waiting for /ok to test mirror for this head. A maintainer needs to approve the new commit with:

/ok to test 7800f2f3fd34deccbd7dab62533e48779acbcd19

[drew]

gator-agent

PR Review Status

Thanks @quocanh261997. I checked your October 4 deletion-polling update against the prior cleanup obligation and the test harness behavior. Waiting for the original sandbox to disappear before same-name recreation fixes the observed rootless Podman race, the earlier failed-create finding remains resolved, and this test-only delta introduces no new Critical defect.

Blocking findings:

• No blocking findings remain

Carried findings:

• GATOR-e5de0a89-01: remains resolved; this delta does not alter the compensation or retryable deletion implementation

Required test dispatch is not complete for this head. test:e2e remains required, and Gator will request the current-head contributor mirror before entering pipeline watch.

Gator metadata

• Validation: project-valid through linked, validated issue #3835
• Docs: the stable host-identity contract is documented under docs/how-it-works/sandboxes/overview.mdx; no additional docs change is required for this test-only delta
• Checks: DCO is green; current-head Branch Checks, Helm, Trivy, and E2E workflows are awaiting contributor-mirror dispatch
• E2E: test:e2e is required; current-head mirror creation and workflow queue confirmation are pending
• Head SHA: 7800f2f3fd34deccbd7dab62533e48779acbcd19
• Base SHA: 71c3cd957abef062eb7f37010056717cd49f2ed3
• Merge base SHA: 71c3cd957abef062eb7f37010056717cd49f2ed3
• Patch ID: 2cc204fdaa30f3122895ef858cb7524e04f99c26
• Gator payload: 10
• Review mode: critical_only
• Previous reviewed SHA: 6ec9c0a975e3334bc97e646b1ee6df69bb8fd9c2
• Review budget exhausted: yes
• Maintainer decision required: no
• Next state: gator:in-review

[drew]

/ok to test 7800f2f

[purp]

LGTM 🚢

[quocanh261997]

Merged main at d1e8f44a0 into this branch and pushed 2e66dea643b7777817bbb40ee60df7ee7d36b889.

The supervisor startup conflict is resolved. The update keeps the backend setup and startup tracing from main, preserves the assigned SSH host key, and adds the host-key field to the two new test inputs.

Verification on this update:

• All six live Docker conformance scenarios passed.
• The live SSH identity test passed with no skips: the fingerprint matched the SSH key, pinned connections survived sandbox and gateway restarts, the workload could not read supervisor authentication, and same-name recreation received a new key.
• Gateway tests: 1,994 unit tests passed, one ignored; integration tests passed. Supervisor tests: 153 passed. Process, TUI, VFIO, content-guard, Python, TypeScript, and Go checks also passed.
• Rust lint, compile checks, and the pre-commit hook passed.

The full local mise run ci still failed at proxy::tests::mediated_connect_keeps_workload_bytes_read_with_the_synthesized_header, which timed out waiting for a TCP request. This PR has no changes to that file compared with main. The cause remains unresolved; the remaining gateway and SSH-related tests passed when run separately. The PR body contains the local test setup and limits.

GitHub reports that the new commit is waiting for the test mirror. A trusted maintainer can start the checks with:

/ok to test 2e66dea643b7777817bbb40ee60df7ee7d36b889

[drew]

/ok to test 2e66dea

[drew]

/ok to test 2e66dea