Repository navigation
Conversation
The Blue/Green switchover tests now check what #1352 broke, inside the one switchover each lane runs. Probes start on the phase the plugin publishes. An execute on an open blue connection and a new connect to a green instance, both started in IN_PROGRESS, must return within 60 s, after the phase moved on, and use at most 1.0 s of their thread's CPU. The 60 s is twice bg_connect_timeout_ms, which the tests pin at its 30 s default. A new connection to the blue endpoint opened in POST (and in PREPARATION, when sampled) must reach the substitute host the plugin's status lists for it. A call still in flight after 60 s is reported as a spin or as blocked from a thread CPU sample, so a hung wrapper fails the test instead of hanging the suite. The plugin's logger is held at INFO during the test, and after it while any test thread is still running, so a spinning call can't flood the report with debug records. The switchover metrics are printed before the verdict, and an error while printing them can't hide it. The old test asserted nothing about the wrapper's calls or where new connections went, left hung threads running, and could not finish under the harness's 600 s pytest timeout: it waited 6 minutes and then slept 12. It now waits until the switchover completes, bounded from the request, then for the RDS API to report completion. Every thread is a daemon and the joins share one deadline. The services container is built on the main thread before the monitor threads start: conftest releases it and its lazy re-initialisation isn't locked, so their first connects could find no monitor service. If the switchover is never requested, the failure names the wait result and any unhandled exceptions. The direct topology monitors now poll with autocommit on, as the plugin's own status connections do: without it MySQL's REPEATABLE READ kept returning the first snapshot, so they never saw the switchover complete. The async twin gets the same changes, unwraps AsyncBlueGreenPlugin instead of the sync class, and adds an execute probe on the blue cluster endpoint, through the connection that owns the status provider (the async status doesn't list instance endpoints). That probe takes the provider out of the shutdown hooks every async worker drains when it ends, and stops it itself, and its summary line shows the role and execute routing the plugin had for its host. The harness waits for an in-flight switchover to settle before it picks the cluster to delete, and also deletes a deployment whose switchover failed. It asks RDS for the cluster parameter group family of the engine version instead of assuming aurora-postgresql16, so the PostgreSQL task works on newer default versions, and it deletes that group when the cluster create fails. Covers #1352.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The Blue/Green switchover integration tests on
mainassert nothing about the wrapper's own calls or about where new connections go, so neither #1352 defect (a suspended sync call that never returns after a switchover, substitute routings that never match) can fail them. Their fixed waits also outlast the harness's 600 s pytest timeout, so they can't finish. On a real switchover the inherited monitors never see it complete on MySQL, a monitor can die before the switchover is requested, and the async twin looks for the sync plugin class and records unhandled exceptions on every run. The harness's teardown chooses the cluster to delete from a status it read before the switchover settled and stops on a failed switchover, and the harness can't create an Aurora PostgreSQL Blue/Green cluster on its default version, 17.9.This PR makes
test_switchoverandtest_switchover_asynccheck what #1352 broke, inside the one switchover each run makes, and fixes what keeps the tests and the harness from finishing. It changes no wrapper code; #1360, the fix it covers, is onmain.IN_PROGRESS, must return within 60 s, after the phase has moved on, and use at most 1.0 s of their thread's CPU. The 60 s is twicebg_connect_timeout_ms, pinned at its 30 s default, and matches the bound in chore: assert B/G plugin releases suspended traffic promptly aws-advanced-jdbc-wrapper#2098. A call still in flight after 60 s gets a 2 s CPU sample from/procand fails as "Blue/Green plugin: call hangs after switchover; IP substitution never applied #1352 spin - sync execute suspended in IN_PROGRESS did not return within 60 s" or as blocked, so a hung wrapper fails with a named text and the suite still ends. Every failure text, "check not run" and "check inconclusive" included, fails the test throughpytest.fail.POST(and inPREPARATION, when that phase is sampled) must reach the substitute host the plugin's status lists for it. The probe reads the listed routing from the status and never callsis_match, the code under test, and compares it with the host the driver reached (mysql-connector'sserver_host, psycopg'sinfo.host). A connection that reached the dialled name fails with "Blue/Green plugin: call hangs after switchover; IP substitution never applied #1352 routing - connection opened in POST went to the dialled host, not the listed substitute".4bb55cfd(chore: add timeout to tests and print stacktrace #1062, 2025-12-19) put a 600 s pytest timeout on every test it couldn't finish. It now waits until the probes settle and the switchover completes, at most 240 s from the request (150 s when none was made), then polls the RDS API forSWITCHOVER_COMPLETEDorSWITCHOVER_FAILEDfor up to 300 s. Every thread is a daemon, the joins share one 20 s deadline, and both tests carry@pytest.mark.timeout(SWITCHOVER_TEST_TIMEOUT_SEC)(900 s). A Blue/Green plugin: call hangs after switchover; IP substitution never applied #1352 failure is reported before the unhandled-exceptions check andassert_test, and the switchover metrics print first, so an error while printing them can't hide it.assert_testfailed with "BG switchover hasn't completed." on the fixed wrapper. They now connect withautocommit=True, as the plugin's own status connections do. Both tests also build the services container on the main thread before any thread starts, because conftest releases it before each test and its lazy re-initialisation takes no lock: on Aurora MySQL the writer's executing monitor died with "'NoneType' object has no attribute 'register_monitor_type'" and the switchover was never requested. That failure now names the wait result and the unhandled exceptions.ecc898e8(feat: asyncio support (async wrapper, plugins, and SQLAlchemy dialects) #1257, 2026-07-29) the async test unwrapped the syncBlueGreenPluginfrom async connections, andAsyncBlueGreenPlugindoesn't subclass it, so its monitors recorded "Unable to find blue/green plugin" as an unhandled exception on every run. They now unwrapAsyncBlueGreenPlugin. The async test's own probe is an execute on the blue cluster endpoint, through the connection that owns the status provider, because each async connection keeps its own status store and the async status lists no instance endpoints. The probe takes its provider out of the shutdown hooks every async worker drains when it ends, stops it itself, and prints the role and execute routing the plugin had for its host.TestEnvironment.deleteBlueGreenDeploymentread the deployment's status once and chose the Aurora cluster to delete from that read, so a teardown that started during a switchover took the not-completed branch even when the switchover finished meanwhile. It now waits for the deployment to settle and re-reads it, so a deployment still switching can hold teardown for up to 15 + 15 minutes before it throws (from reading; not observed).AuroraTestUtility.deleteBlueGreenDeploymentalso acceptsswitchover_failed; before, a failed switchover waited 15 minutes and threw out ofclose(), which skipped the cluster and parameter group deletion. Both came in with9a1a4d9c(feat: add support for blue/green deployments #911, 2025-07-25).AuroraTestUtility.getAuroraParameterGroupFamilyreturnedaurora-postgresql16for every PostgreSQL version, so with the harness on aurora-postgresql 17.9 RDS refused the cluster ("Please use a Parameter Group with DBParameterGroupFamily aurora-postgresql17"). It now asks RDS (describeDBEngineVersions) for the family of the engine version, as the JDBC harness does, and keeps the old mapping as the fallback. When a cluster create fails, the custom cluster parameter group is now deleted too; before,close()never ran for that environment and the group was left in the account.Not in scope
mainbefore fix: select blue/green routings from the latest status and match them by host and port #1360, aren't fixed here and have no issue yet. The services container's lazy initialisation takes no lock (item 5 works around it). Inaio/blue_green_plugin.pythe status monitor calls the zero-argumentget_async_host_list_provider_supplierwith two arguments and logs the TypeError at DEBUG, so the async status never learns its instance endpoints or publishesCOMPLETED: an async execute on the writer instance endpoint ran throughIN_PROGRESSin 0.06 s, never held.aurora-postgresql16when RDS can't be asked.exclude-bgdefaults totrueand no workflow calls atest-bgd-*task. Each switchover test needs its own deployment and pytest session, so a run selects one test withFILTER.Tests
No test is added;
test_switchoverandtest_switchover_asyncchange, and the verdict logic in the newtests/integration/container/utils/blue_green_checks.pyhas no unit tests. The tests ran on real Aurora Blue/Green switchovers through the Gradle harness (test-bgd-mysql-auroraandtest-bgd-pg-aurora, Python 3.12), one test per deployment:FILTER="test_switchover and not async"with-Dexclude-async-drivers=true, andFILTER="test_switchover_async and MYSQL_ASYNC"with-Dexclude-sync-drivers=true. With the wrapper before #1360 (fdce8d9), Aurora MySQL and Aurora PostgreSQL failed with "#1352 spin - sync execute suspended in IN_PROGRESS did not return within 60 s", the matching connect text and the POST routing text, with no pytest timeout. With #1360's fix (ad72ac6, whose Blue/Green plugin code matchesmainapart from a constant rename inaio/blue_green_plugin.py), the sync test passed on Aurora MySQL and Aurora PostgreSQL and the async test passed on Aurora MySQL, its execute held 5.22 s fromIN_PROGRESSintoPOST. On Aurora PostgreSQL the server can drop a stuck call before 60 s, and one unfixed run reported "#1352 busy loop - a suspended call used more than 1.0 s of its thread CPU" instead. Each run took about an hour, mostly cluster and deployment setup and teardown. The RDS Multi-AZ instance tasks, the async test on Aurora PostgreSQL and Python versions other than 3.12 weren't run. On this commitmypy,flake8,isort --check-onlyandpytest tests/unitpass on Python 3.12, and the harness compiles with--release 8. No CHANGELOG entry, message key or docs change.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.