Skip to content

test: assert blue/green switchover calls resume and are routed - #1370

Draft
JuanLeee wants to merge 1 commit into
mainfrom
test/bg-1352-integration
Draft

JuanLeee wants to merge 1 commit into
mainfrom
test/bg-1352-integration

Conversation

@JuanLeee

Copy link
Copy Markdown
Contributor

Description

The Blue/Green switchover integration tests on main assert nothing about the wrapper's own calls or about where new connections go, so neither #1352 defect (a suspended sync call that never returns after a switchover, substitute routings that never match) can fail them. Their fixed waits also outlast the harness's 600 s pytest timeout, so they can't finish. On a real switchover the inherited monitors never see it complete on MySQL, a monitor can die before the switchover is requested, and the async twin looks for the sync plugin class and records unhandled exceptions on every run. The harness's teardown chooses the cluster to delete from a status it read before the switchover settled and stops on a failed switchover, and the harness can't create an Aurora PostgreSQL Blue/Green cluster on its default version, 17.9.

This PR makes test_switchover and test_switchover_async check what #1352 broke, inside the one switchover each run makes, and fixes what keeps the tests and the harness from finishing. It changes no wrapper code; #1360, the fix it covers, is on main.

  1. Probes now start on the phase the plugin publishes, which an observer thread reads from the storage service every 20 ms. An execute on an open blue connection and a new connect to a green instance, both started in IN_PROGRESS, must return within 60 s, after the phase has moved on, and use at most 1.0 s of their thread's CPU. The 60 s is twice bg_connect_timeout_ms, pinned at its 30 s default, and matches the bound in chore: assert B/G plugin releases suspended traffic promptly aws-advanced-jdbc-wrapper#2098. A call still in flight after 60 s gets a 2 s CPU sample from /proc and fails as "Blue/Green plugin: call hangs after switchover; IP substitution never applied #1352 spin - sync execute suspended in IN_PROGRESS did not return within 60 s" or as blocked, so a hung wrapper fails with a named text and the suite still ends. Every failure text, "check not run" and "check inconclusive" included, fails the test through pytest.fail.
  2. A new connection to the blue endpoint opened in POST (and in PREPARATION, when that phase is sampled) must reach the substitute host the plugin's status lists for it. The probe reads the listed routing from the status and never calls is_match, the code under test, and compares it with the host the driver reached (mysql-connector's server_host, psycopg's info.host). A connection that reached the dialled name fails with "Blue/Green plugin: call hangs after switchover; IP substitution never applied #1352 routing - connection opened in POST went to the dialled host, not the listed substitute".
  3. The old test waited up to 6 minutes on its finish latch, slept 12 and joined each thread for 30 s, so since 4bb55cfd (chore: add timeout to tests and print stacktrace #1062, 2025-12-19) put a 600 s pytest timeout on every test it couldn't finish. It now waits until the probes settle and the switchover completes, at most 240 s from the request (150 s when none was made), then polls the RDS API for SWITCHOVER_COMPLETED or SWITCHOVER_FAILED for up to 300 s. Every thread is a daemon, the joins share one 20 s deadline, and both tests carry @pytest.mark.timeout(SWITCHOVER_TEST_TIMEOUT_SEC) (900 s). A Blue/Green plugin: call hangs after switchover; IP substitution never applied #1352 failure is reported before the unhandled-exceptions check and assert_test, and the switchover metrics print first, so an error while printing them can't hide it.
  4. The plugin's logger is held at INFO during the test, and after it while any test thread is still alive, because an unfixed wrapper's suspended calls log two debug records per loop pass. In one unfixed Aurora PostgreSQL run an inherited monitor's records reached about 650 MB in 25 s and the Gradle test worker ran out of heap. The switch interval lowered for the CPU sample is restored under the same rule.
  5. The direct topology monitors polled the status table without autocommit, so on MySQL every poll repeated its first REPEATABLE READ snapshot and assert_test failed with "BG switchover hasn't completed." on the fixed wrapper. They now connect with autocommit=True, as the plugin's own status connections do. Both tests also build the services container on the main thread before any thread starts, because conftest releases it before each test and its lazy re-initialisation takes no lock: on Aurora MySQL the writer's executing monitor died with "'NoneType' object has no attribute 'register_monitor_type'" and the switchover was never requested. That failure now names the wait result and the unhandled exceptions.
  6. Since ecc898e8 (feat: asyncio support (async wrapper, plugins, and SQLAlchemy dialects) #1257, 2026-07-29) the async test unwrapped the sync BlueGreenPlugin from async connections, and AsyncBlueGreenPlugin doesn't subclass it, so its monitors recorded "Unable to find blue/green plugin" as an unhandled exception on every run. They now unwrap AsyncBlueGreenPlugin. The async test's own probe is an execute on the blue cluster endpoint, through the connection that owns the status provider, because each async connection keeps its own status store and the async status lists no instance endpoints. The probe takes its provider out of the shutdown hooks every async worker drains when it ends, stops it itself, and prints the role and execute routing the plugin had for its host.
  7. TestEnvironment.deleteBlueGreenDeployment read the deployment's status once and chose the Aurora cluster to delete from that read, so a teardown that started during a switchover took the not-completed branch even when the switchover finished meanwhile. It now waits for the deployment to settle and re-reads it, so a deployment still switching can hold teardown for up to 15 + 15 minutes before it throws (from reading; not observed). AuroraTestUtility.deleteBlueGreenDeployment also accepts switchover_failed; before, a failed switchover waited 15 minutes and threw out of close(), which skipped the cluster and parameter group deletion. Both came in with 9a1a4d9c (feat: add support for blue/green deployments #911, 2025-07-25).
  8. AuroraTestUtility.getAuroraParameterGroupFamily returned aurora-postgresql16 for every PostgreSQL version, so with the harness on aurora-postgresql 17.9 RDS refused the cluster ("Please use a Parameter Group with DBParameterGroupFamily aurora-postgresql17"). It now asks RDS (describeDBEngineVersions) for the family of the engine version, as the JDBC harness does, and keeps the old mapping as the fallback. When a cluster create fails, the custom cluster parameter group is now deleted too; before, close() never ran for that environment and the group was left in the account.

Not in scope

  1. Two product defects outside Blue/Green plugin: call hangs after switchover; IP substitution never applied #1352, both on main before fix: select blue/green routings from the latest status and match them by host and port #1360, aren't fixed here and have no issue yet. The services container's lazy initialisation takes no lock (item 5 works around it). In aio/blue_green_plugin.py the status monitor calls the zero-argument get_async_host_list_provider_supplier with two arguments and logs the TypeError at DEBUG, so the async status never learns its instance endpoints or publishes COMPLETED: an async execute on the writer instance endpoint ran through IN_PROGRESS in 0.06 s, never held.
  2. For that reason the async test has no connect or routing probe.
  3. The RDS Multi-AZ instance teardown branch keeps its single status read, and the parameter group fallback still returns aurora-postgresql16 when RDS can't be asked.
  4. CI doesn't run these tests: exclude-bg defaults to true and no workflow calls a test-bgd-* task. Each switchover test needs its own deployment and pytest session, so a run selects one test with FILTER.

Tests

No test is added; test_switchover and test_switchover_async change, and the verdict logic in the new tests/integration/container/utils/blue_green_checks.py has no unit tests. The tests ran on real Aurora Blue/Green switchovers through the Gradle harness (test-bgd-mysql-aurora and test-bgd-pg-aurora, Python 3.12), one test per deployment: FILTER="test_switchover and not async" with -Dexclude-async-drivers=true, and FILTER="test_switchover_async and MYSQL_ASYNC" with -Dexclude-sync-drivers=true. With the wrapper before #1360 (fdce8d9), Aurora MySQL and Aurora PostgreSQL failed with "#1352 spin - sync execute suspended in IN_PROGRESS did not return within 60 s", the matching connect text and the POST routing text, with no pytest timeout. With #1360's fix (ad72ac6, whose Blue/Green plugin code matches main apart from a constant rename in aio/blue_green_plugin.py), the sync test passed on Aurora MySQL and Aurora PostgreSQL and the async test passed on Aurora MySQL, its execute held 5.22 s from IN_PROGRESS into POST. On Aurora PostgreSQL the server can drop a stuck call before 60 s, and one unfixed run reported "#1352 busy loop - a suspended call used more than 1.0 s of its thread CPU" instead. Each run took about an hour, mostly cluster and deployment setup and teardown. The RDS Multi-AZ instance tasks, the async test on Aurora PostgreSQL and Python versions other than 3.12 weren't run. On this commit mypy, flake8, isort --check-only and pytest tests/unit pass on Python 3.12, and the harness compiles with --release 8. No CHANGELOG entry, message key or docs change.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

The Blue/Green switchover tests now check what #1352 broke, inside the
one switchover each lane runs. Probes start on the phase the plugin
publishes. An execute on an open blue connection and a new connect to a
green instance, both started in IN_PROGRESS, must return within 60 s,
after the phase moved on, and use at most 1.0 s of their thread's CPU.
The 60 s is twice bg_connect_timeout_ms, which the tests pin at its 30 s
default. A new connection to the blue endpoint opened in POST (and in
PREPARATION, when sampled) must reach the substitute host the plugin's
status lists for it. A call still in flight after 60 s is reported as a
spin or as blocked from a thread CPU sample, so a hung wrapper fails the
test instead of hanging the suite. The plugin's logger is held at INFO
during the test, and after it while any test thread is still running,
so a spinning call can't flood the report with debug records. The
switchover metrics are printed before the verdict, and an error while
printing them can't hide it.

The old test asserted nothing about the wrapper's calls or where new
connections went, left hung threads running, and could not finish under
the harness's 600 s pytest timeout: it waited 6 minutes and then slept
12. It now waits until the switchover completes, bounded from the
request, then for the RDS API to report completion. Every thread is a
daemon and the joins share one deadline. The services container is built
on the main thread before the monitor threads start: conftest releases
it and its lazy re-initialisation isn't locked, so their first connects
could find no monitor service. If the switchover is never requested, the
failure names the wait result and any unhandled exceptions. The direct
topology monitors now poll with autocommit on, as the plugin's own
status connections do: without it MySQL's REPEATABLE READ kept returning
the first snapshot, so they never saw the switchover complete. The async
twin gets the same changes, unwraps AsyncBlueGreenPlugin instead of the
sync class, and adds an execute probe on the blue cluster endpoint,
through the connection that owns the status provider (the async status
doesn't list instance endpoints). That probe takes the provider out of
the shutdown hooks every async worker drains when it ends, and stops it
itself, and its summary line shows the role and execute routing the
plugin had for its host. The harness waits for an in-flight switchover
to settle before it picks the cluster to delete, and also deletes a
deployment whose switchover failed. It asks RDS for the cluster
parameter group family of the engine version instead of assuming
aurora-postgresql16, so the PostgreSQL task works on newer default
versions, and it deletes that group when the cluster create fails.

Covers #1352.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant