Skip to content

Commit 6361caf

Browse files
authored
Add orphan cleanup for cloud resources (#4401)
* Prototype OrphanCleanupPipeline * Clean up * Fix orphan classification order * Batch owner record lookups in orphan cleanup * Log orphan resources of stuck pending instances * Emit events for orphaned resources * Add migrations for orphan cleanup * Recommend separate migrations for concurrent indexes * Clarify resource listing contract * Register OrphanCleanupPipeline * Pass provider_ref to backends * Match orphaned resources of failed job provisioning by job * Add provider_ref naming helpers * Implement orphan resource listing for AWS * Move AWS Elastic IP helpers to resources.py * Enroll backends in orphan cleanup * Mention orphan cleanup when instance termination gives up * Document orphan cleanup settings * Reword resource listing contract * List AWS instances without Elastic IP permissions * Detect orphan cleanup support of registered backends * Report partial resource listings in orphan cleanup * Ignore unsupported owner types in orphan cleanup * Test OrphanCleanupPipeline fetching and processing * Document orphan cleanup support for backends * Fix AWS resource listing edge cases * Simplify orphan cleanup result handling and logging * Tidy orphan cleanup docstrings and tests * Check orphan cleanup activity in worker instead of fetcher * Skip orphan groups of owners changed around listing and drop backoff for skips * Report orphan owner resources whose deletion is requested * Load orphan owner jobs before instances * Keep superseded job submissions until orphan cleanup * Bound how long superseded submissions wait for orphan cleanup * Clarify orphan cleanup filter docstring * Fix compute group provider_ref TODO * Replace em dash in release_eip docstring * Derive superseded submission max wait from orphan cleanup interval * Document orphan cleanup in contributing * Record issued provider refs in a ledger Placeholder instances carry the provider_ref of run_job() instead of jobs, and ProviderRefModel keeps refs after owners are deleted, so superseded job submissions no longer need to wait for orphan cleanup. * Wrap AWS errors when deleting orphaned resources
1 parent e59a6ca commit 6361caf

37 files changed

Lines changed: 3272 additions & 139 deletions

‎AGENTS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22

33
Before touching a subsystem, read the relevant notes in `contributing/`: `ARCHITECTURE.md`,
44
`PIPELINES.md`, `LOCKING.md`, `DATABASE.md`, `MIGRATIONS.md`, `RUNS-AND-JOBS.md`, `AUTOSCALING.md`,
5-
`BACKENDS.md`, `GPUHUNT.md`, `PROXY.md`, `RUNNER-AND-SHIM.md`, `FRONTEND.md`, `DOCS.md`,
6-
`DEVELOPMENT.md`, `RELEASE.md`.
5+
`BACKENDS.md`, `ORPHAN-CLEANUP.md`, `GPUHUNT.md`, `PROXY.md`, `RUNNER-AND-SHIM.md`, `FRONTEND.md`,
6+
`DOCS.md`, `DEVELOPMENT.md`, `RELEASE.md`.
77

88
## Project Structure & Module Organization
99
- Core Python package lives in `src/dstack`; internal modules (including server) sit under `_internal`, API surfaces under `api`, and plugin integrations under `plugins`.

‎contributing/BACKENDS.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,7 @@ Refer to examples:
144144

145145
Go to `compute.py` and implement `Compute` methods.
146146
Optionally, extend and implement `ComputeWith*` classes to support additional features such as fleets, volumes, gateways, placement groups, etc. For example, extend `ComputeWithCreateInstanceSupport` to support fleets.
147+
To support the cleanup of orphaned resources, extend `ComputeWithResourceListingSupport` as described in [Orphan cleanup support](#32-orphan-cleanup-support).
147148

148149
Refer to examples:
149150
[verda](https://github.com/dstackai/dstack/blob/master/src/dstack/_internal/core/backends/verda/compute.py),
@@ -228,3 +229,36 @@ To support a container-based backend, `dstack` expects the following:
228229
- A way to override the container user to root (as in `docker run --user root ...`)
229230

230231
Examples of container-based backends include: `kubernetes`, `vastai`, `runpod`.
232+
233+
### 3.2. Orphan cleanup support
234+
235+
`dstack` can find and delete cloud resources it created but no longer tracks, e.g. if the server
236+
crashed after the provider created an instance but before it was recorded, or if instance
237+
termination failed. The `OrphanCleanupPipeline` lists resources of backends that extend
238+
`ComputeWithResourceListingSupport` and matches them with instances by `provider_ref`.
239+
See [ORPHAN-CLEANUP.md](ORPHAN-CLEANUP.md) for the server side.
240+
241+
To support it in a backend:
242+
243+
1. Set `provider_ref` on every resource created for an instance or a job. It's
244+
`InstanceConfiguration.provider_ref` in `create_instance()` and `get_job_provider_ref(job)` in
245+
`run_job()`. Internal resources that outlive the instance if left behind (disks, keys, IPs)
246+
need it too, while resources deleted together with the instance don't.
247+
* If the provider supports tags, set the `PROVIDER_REF_TAG_KEY` tag and keep instance names as is.
248+
If the provider does not allow this key, use another one.
249+
* Otherwise, encode `provider_ref` in names with `generate_unique_instance_name(..., with_provider_ref=True)`
250+
and parse it with `parse_provider_ref_from_name()`.
251+
* Do not set it on resources of `run_jobs()` since orphan cleanup does not support compute groups yet.
252+
2. Implement `list_resource_groups()`. It returns a `ListedResourceGroup` per provisioning attempt: the instance
253+
(`primary_id`) together with internal resources created for it. Skip resources without `provider_ref`.
254+
List resources of each region all or nothing: if any listing in a region fails, return no groups
255+
of the region and report it in `ResourceListing.failures`. Otherwise, a tracked instance's resources
256+
may be listed without the instance and deleted. Skip instances whose deletion is in progress
257+
(e.g. shutting down), but list their internal resources.
258+
3. Implement `delete_resource_group()`. Delete everything in the group in the order the provider requires.
259+
The method must be idempotent and never delete resources that don't carry the group's `provider_ref`,
260+
such as attached volumes. If the deletion takes several steps, delete the instance first and raise
261+
`NotYetTerminated` instead of waiting for the next step, e.g. for the instance to be gone before its
262+
disk can be deleted. Next cleanups list the remaining internal resources and call the method again.
263+
264+
Refer to the [aws](https://github.com/dstackai/dstack/blob/master/src/dstack/_internal/core/backends/aws/compute.py) example.

‎contributing/MIGRATIONS.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,3 +73,24 @@ with op.get_context().autocommit_block():
7373
postgresql_concurrently=True,
7474
)
7575
```
76+
77+
Keep concurrent index creation in a separate migration. Do not combine `autocommit_block()`
78+
with other schema changes such as `add_column`/`batch_alter_table` in the same migration.
79+
`autocommit_block()` commits the migration's transaction before running the index statements,
80+
so a failure inside the block leaves the earlier changes committed while the migration is not
81+
marked as applied. A retry then re-runs those changes and fails. For example:
82+
83+
```python
84+
def upgrade() -> None:
85+
with op.batch_alter_table("table", schema=None) as batch_op:
86+
batch_op.add_column(sa.Column("col", sa.String(length=12), nullable=True))
87+
with op.get_context().autocommit_block():
88+
# If this fails, "col" is already committed. A retry fails on add_column
89+
# with "column already exists".
90+
op.create_index("ix_table_col", "table", ["col"], postgresql_concurrently=True)
91+
```
92+
93+
Instead, add the column in one migration and create the index in the next one.
94+
95+
Indexes on a table created in the same migration do not need `CONCURRENTLY`, since the table is
96+
empty. Create them without `autocommit_block()` so that the whole migration stays transactional.

‎contributing/ORPHAN-CLEANUP.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Orphan cleanup
2+
3+
`dstack` can leave cloud resources it no longer tracks: the server crashes or loses its lock between a provider call that creates resources and the commit that records them, the provider returns an error after creating them, or termination gives up after retrying. Orphan cleanup finds such resources by listing them in the provider and matching them with the DB, then reports or deletes them.
4+
5+
How a backend supports it is described in [BACKENDS.md](BACKENDS.md#32-orphan-cleanup-support). This doc covers the server side.
6+
7+
## `provider_ref`
8+
9+
Each record that provisions resources gets a `provider_ref` (12-char string). Backends set it on the resources they create (as a tag or in names) and report it back when listing.
10+
11+
- Only instances own `provider_ref`s (`InstanceModel.provider_ref`). Fleet instances get it on creation, for `create_instance()`. Placeholder instances, created for job submissions before `run_job()`, get it on creation too, pass it to `run_job()` (`JobSubmission.provider_ref`), and keep it when promoted.
12+
- Every issued `provider_ref` is also recorded in `ProviderRefModel`, committed with its owner before any provider call. Records are not deleted with owners, so resources of a deleted owner are still recognized. This matters for placeholder instances, which are deleted when a `run_job()` fails, possibly after creating resources.
13+
- Compute groups (`run_jobs()`) are not supported: their instances get no `provider_ref`, and backends must not set it on compute group resources.
14+
15+
## Cleanup unit and enrollment
16+
17+
An `OrphanCleanupModel` row exists per project backend (`BackendModel`) that supports resource listing. A scheduled task periodically creates rows in batches for backends of projects that have issued `provider_ref`s. Enrolling periodically instead of in a migration covers backends created by any replica, including older replicas during rolling deployments.
18+
19+
## `OrphanCleanupPipeline`
20+
21+
The pipeline processes each row at most once per cleanup interval (`DSTACK_SERVER_ORPHAN_CLEANUP_INTERVAL_SECONDS`). Failed cleanups are retried with an increasing delay.
22+
23+
Processing a row:
24+
25+
- STEP 1: Check whether the backend may have new orphans: it was never cleaned up, the cleanup mode changed, or the project had instance or job activity since the last clean cleanup (`clean_at`). If not, the row is only marked processed. Checking activity here rather than in the fetch query keeps fetching cheap regardless of project sizes, at the cost of a small query and row update per enrolled backend per interval.
26+
- STEP 2: List the backend's resource groups (`list_resource_groups()`). A resource group is one provisioning attempt as found in the provider: the owner resource (`primary_id`, the instance) and internal resources created with it.
27+
- STEP 3: Load the project's `ProviderRefModel` records and their instances (including soft-deleted ones) by `provider_ref`, in one statement. An instance is committed with its record, so it can't be missed while the record is seen.
28+
- STEP 4: Classify each group:
29+
- Instance found:
30+
- `PENDING`, `TERMINATING`: skip, since the resources may be in flight. An error is logged if a pending instance's resources look stuck.
31+
- `TERMINATED`: orphan.
32+
- `PROVISIONING`, `IDLE`, `BUSY`: tracked if the base backend and `primary_id` match the instance's provisioning data, orphan otherwise.
33+
- Without provisioning data: ignored with an error log.
34+
- Record found without an instance: orphan. The owner was deleted, e.g. a placeholder instance after a failed `run_job()`, so nothing can be creating its resources anymore.
35+
- No record: ignored. Resources of other servers and projects are never touched.
36+
- STEP 5: Owners that were provisioned or terminated around the listing are skipped instead of classified as orphans, since the listing may predate the change. The margin covers clock skew between replicas, the lag between setting timestamps and committing them, and providers listing new resources with a delay.
37+
- STEP 6: Handle orphans according to `DSTACK_SERVER_ORPHAN_CLEANUP_MODE`:
38+
- `report` (default): log, send to Sentry, and emit an event targeting the owner instance. If the instance was deleted, the event still targets it by the id and name kept in `ProviderRefModel`, so it shows next to the instance's earlier events.
39+
- `delete`: call `delete_resource_group()` and emit an event. If the backend raises `NotYetTerminated` for a group with an owner resource, the owner's deletion was requested and the event says the resources are being deleted. The remaining internal resources are listed and deleted by next cleanups.
40+
- `off`: the pipeline and enrollment are disabled.
41+
- STEP 7: Set the outcome:
42+
- Clean (everything listed, nothing skipped, all orphans handled): `clean_at` advances to the listing time minus the margin, and the retry delay is reset.
43+
- Unsettled (a group skipped, a deletion in progress, or some regions in `ResourceListing.failures`): `clean_at` stays, the retry delay is reset, and the backend is retried after the interval. Listed groups are still handled.
44+
- Failed (listing raised or a deletion failed): `clean_at` stays and the retry is backed off.
45+
46+
Events are emitted in the lock-guarded apply, so a worker that lost the lock emits nothing.
47+
48+
## Termination give-up
49+
50+
When instance termination gives up after retrying, the log says that orphan cleanup will retry deleting the instance if the mode is `delete` and the backend supports listing. Otherwise, it asks to terminate the instance manually.
51+
52+
## Limitations
53+
54+
- Resources are not cleaned up if the backend credentials change or the backend is deleted.
55+
- Compute groups, placement groups, volumes and gateways are not covered. `OwnerType` only has `INSTANCE`.
56+
- `ProviderRefModel` records are never deleted yet. There is one per fleet instance and per placeholder instance, i.e. per job submission that provisions new capacity.
57+
- On dstack Sky, each project's `dstack` backend lists the shared base backend accounts, so listings repeat per project. Cleaning up each shared account once is a possible optimization.

‎mkdocs/docs/reference/env.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,8 @@ For more details on the options below, refer to the [server deployment](../guide
151151
- `DSTACK_SERVER_GATEWAY_MAX_REPLICAS`{ #DSTACK_SERVER_GATEWAY_MAX_REPLICAS } - Maximum number of replicas allowed for a gateway. Defaults to `9`.
152152
- `DSTACK_SERVER_INSTANCE_HEALTH_MIN_COLLECT_INTERVAL_SECONDS`{ #DSTACK_SERVER_INSTANCE_HEALTH_MIN_COLLECT_INTERVAL_SECONDS } – Minimum time interval between consecutive health checks of the same instance.
153153
- `DSTACK_SERVER_EVENTS_TTL_SECONDS`{ #DSTACK_SERVER_EVENTS_TTL_SECONDS } - Maximum age of event records. Set to `0` to disable event storage. Defaults to 30 days.
154+
- `DSTACK_SERVER_ORPHAN_CLEANUP_MODE`{ #DSTACK_SERVER_ORPHAN_CLEANUP_MODE } – Controls the cleanup of cloud resources that `dstack` created but no longer tracks, e.g. after the server crashed during provisioning or failed to terminate an instance. `report` emits events about such resources, `delete` also deletes them, and `off` disables the cleanup. Defaults to `report`. Currently supported for the `aws` backend.
155+
- `DSTACK_SERVER_ORPHAN_CLEANUP_INTERVAL_SECONDS`{ #DSTACK_SERVER_ORPHAN_CLEANUP_INTERVAL_SECONDS } – Minimum interval between orphan cleanups of the same backend. A backend is cleaned up only after instances were provisioned or terminated in its project. Defaults to `3600`.
154156
- `DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY`{ #DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY } – A default Docker registry to use for job images that do not specify an explicit registry. E.g., if set to `registry.example`, then `image: ubuntu` becomes equivalent to `image: registry.example/ubuntu`. **Note**: This setting should only be used for configuring registries that act as a pull-through cache for Docker Hub. The default `dstack` images are also pulled from the configured registry.
155157
- `DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY_USERNAME`{ #DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY_USERNAME } – Username for authenticating with the default Docker registry. See `DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY_PASSWORD`.
156158
- `DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY_PASSWORD`{ #DSTACK_SERVER_DEFAULT_DOCKER_REGISTRY_PASSWORD } – Password for authenticating with the default Docker registry. Applied only when the image has no explicit registry and the run configuration does not specify `registry_auth`. **Note**: The value may be visible to anyone who can SSH into instances managed by `dstack`, which usually includes all users of that `dstack` server.

0 commit comments

Comments
 (0)