Skip to content

fix(spanner): renew channel affinity across retries - #7002

Open
olavloite wants to merge 1 commit into
googleapis:mainfrom
olavloite:spanner-renew-channel-affinity-on-retry
Open

olavloite wants to merge 1 commit into
googleapis:mainfrom
olavloite:spanner-renew-channel-affinity-on-retry

Conversation

@olavloite

Copy link
Copy Markdown
Contributor

When a dynamic channel pool scales in or drains a channel, read/write transactions must not keep draining channels alive indefinitely across retries, nor should retries remain pinned to a draining channel.

Previously, TransactionRunner::run created a single TransactionAffinity instance outside its retry loop and reused it across all attempts without releasing its read/write channel guard. This caused two problems:

  1. If attempt 1 aborted and the channel was marked as draining, retry attempt 2 remained pinned to that draining channel instead of selecting a healthy active channel.
  2. The channel's active_rw_transactions guard remained held during backoff sleeps and across attempts, preventing the scaler from closing and sweeping drained channels.

This change resolves both issues:

  • Creates a fresh TransactionAffinity per attempt inside the retry loop, matching the behavior of WriteOnlyTransaction and PartitionedDmlTransaction. If a channel started draining during attempt 1, attempt 2 routes to a healthy active channel.
  • Explicitly calls affinity.release_rw_guard() as soon as the attempt future completes, ensuring the channel's active R/W counter is decremented before entering aborted backoff sleep.
  • Introduces RwGuardState { Active, Released } in TransactionAffinity so guard release is terminal for that attempt, preventing stragglers or late affinity resolutions from resurrecting the guard.
  • Adds comprehensive unit tests verifying that retries route to fresh channels, guards are released prior to backoff, draining channels sweep to closed, and dropping the runner future mid-attempt cleans up guards via RAII.

When a dynamic channel pool scales in or drains a channel, read/write
transactions must not keep draining channels alive indefinitely across retries,
nor should retries remain pinned to a draining channel.

Previously, `TransactionRunner::run` created a single `TransactionAffinity`
instance outside its retry loop and reused it across all attempts without
releasing its read/write channel guard. This caused two problems:
1. If attempt 1 aborted and the channel was marked as draining, retry
   attempt 2 remained pinned to that draining channel instead of selecting
   a healthy active channel.
2. The channel's `active_rw_transactions` guard remained held during backoff
   sleeps and across attempts, preventing the scaler from closing and sweeping
   drained channels.

This change resolves both issues:
- Creates a fresh `TransactionAffinity` per attempt inside the retry loop,
  matching the behavior of `WriteOnlyTransaction` and `PartitionedDmlTransaction`.
  If a channel started draining during attempt 1, attempt 2 routes to a healthy
  active channel.
- Explicitly calls `affinity.release_rw_guard()` as soon as the attempt future
  completes, ensuring the channel's active R/W counter is decremented before
  entering aborted backoff sleep.
- Introduces `RwGuardState { Active, Released }` in `TransactionAffinity` so
  guard release is terminal for that attempt, preventing stragglers or late
  affinity resolutions from resurrecting the guard.
- Adds comprehensive unit tests verifying that retries route to fresh channels,
  guards are released prior to backoff, draining channels sweep to closed,
  and dropping the runner future mid-attempt cleans up guards via RAII.
@olavloite
olavloite requested review from a team as code owners September 28, 2026 09:29
@product-auto-label product-auto-label Bot added the api: spanner Issues related to the Spanner API. label Sep 28, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Spanner transaction runner to instantiate a fresh TransactionAffinity handle for each transaction attempt instead of reusing a single handle across retries. It introduces a terminal RwGuardState::Released state to prevent late or concurrent operations from re-acquiring a guard on a draining channel. Additionally, the spanner field in DatabaseClient is exposed as pub(crate), and comprehensive tests are added to validate guard release behavior during retries, cancellations, and channel scale-ins. I have no further feedback to provide as no review comments were submitted.

@olavloite
olavloite requested a review from rahul2393 September 28, 2026 09:32
@codecov

codecov Bot commented Sep 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.44853% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.12%. Comparing base (6ec4f47) to head (be64082).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
src/spanner/src/transaction_runner.rs 99.37% 3 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main    #7002    +/-   ##
========================================
  Coverage   97.11%   97.12%            
========================================
  Files         326      326            
  Lines      114380   114880   +500     
========================================
+ Hits       111085   111575   +490     
- Misses       3295     3305    +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: spanner Issues related to the Spanner API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant