Conversation
When a dynamic channel pool scales in or drains a channel, read/write
transactions must not keep draining channels alive indefinitely across retries,
nor should retries remain pinned to a draining channel.
Previously, `TransactionRunner::run` created a single `TransactionAffinity`
instance outside its retry loop and reused it across all attempts without
releasing its read/write channel guard. This caused two problems:
1. If attempt 1 aborted and the channel was marked as draining, retry
attempt 2 remained pinned to that draining channel instead of selecting
a healthy active channel.
2. The channel's `active_rw_transactions` guard remained held during backoff
sleeps and across attempts, preventing the scaler from closing and sweeping
drained channels.
This change resolves both issues:
- Creates a fresh `TransactionAffinity` per attempt inside the retry loop,
matching the behavior of `WriteOnlyTransaction` and `PartitionedDmlTransaction`.
If a channel started draining during attempt 1, attempt 2 routes to a healthy
active channel.
- Explicitly calls `affinity.release_rw_guard()` as soon as the attempt future
completes, ensuring the channel's active R/W counter is decremented before
entering aborted backoff sleep.
- Introduces `RwGuardState { Active, Released }` in `TransactionAffinity` so
guard release is terminal for that attempt, preventing stragglers or late
affinity resolutions from resurrecting the guard.
- Adds comprehensive unit tests verifying that retries route to fresh channels,
guards are released prior to backoff, draining channels sweep to closed,
and dropping the runner future mid-attempt cleans up guards via RAII.
There was a problem hiding this comment.
Code Review
This pull request updates the Spanner transaction runner to instantiate a fresh TransactionAffinity handle for each transaction attempt instead of reusing a single handle across retries. It introduces a terminal RwGuardState::Released state to prevent late or concurrent operations from re-acquiring a guard on a draining channel. Additionally, the spanner field in DatabaseClient is exposed as pub(crate), and comprehensive tests are added to validate guard release behavior during retries, cancellations, and channel scale-ins. I have no further feedback to provide as no review comments were submitted.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7002 +/- ##
========================================
Coverage 97.11% 97.12%
========================================
Files 326 326
Lines 114380 114880 +500
========================================
+ Hits 111085 111575 +490
- Misses 3295 3305 +10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
When a dynamic channel pool scales in or drains a channel, read/write transactions must not keep draining channels alive indefinitely across retries, nor should retries remain pinned to a draining channel.
Previously,
TransactionRunner::runcreated a singleTransactionAffinityinstance outside its retry loop and reused it across all attempts without releasing its read/write channel guard. This caused two problems:active_rw_transactionsguard remained held during backoff sleeps and across attempts, preventing the scaler from closing and sweeping drained channels.This change resolves both issues:
TransactionAffinityper attempt inside the retry loop, matching the behavior ofWriteOnlyTransactionandPartitionedDmlTransaction. If a channel started draining during attempt 1, attempt 2 routes to a healthy active channel.affinity.release_rw_guard()as soon as the attempt future completes, ensuring the channel's active R/W counter is decremented before entering aborted backoff sleep.RwGuardState { Active, Released }inTransactionAffinityso guard release is terminal for that attempt, preventing stragglers or late affinity resolutions from resurrecting the guard.