Repository navigation
Conversation
Add container tests for the KMS Encryption plugin on PostgreSQL and MySQL, including SQLAlchemy and Django, against LocalStack's KMS, a real KMS key and Aurora. The harness gets the RUN_KMS_ENCRYPTION_TESTS_ONLY and KMS_LOCALSTACK features, three Gradle tasks, a LocalStack container pinned by digest, PostgreSQL statement logging and a JUnit XML report. No product code changes. Add a compose lane, tests/integration/kms_local/run.sh, that runs the LocalStack modules without Gradle. It replaces tests/integration/test_kms_encryption_live.py. A new workflow runs the LocalStack job on pull requests, the real-key job on pushes to main and on dispatch, and the Aurora job on dispatch only. Each job fails when no KMS test ran. Two strict xfails track product defects: fetchmany() without a size returns no rows on mysql-connector's C extension, and on Aurora PostgreSQL the plugin's metadata lookup after a writer failover stops failover_v2 from running. The real-key jobs need one-time administrator setup: a symmetric key, an OIDC role trusted only for main and limited to that key, a Decrypt-only role, and the KMS_KEY_ARN, KMS_REGION, KMS_IT_ROLE_ARN and KMS_DECRYPT_ONLY_ROLE_ARN repository variables.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds the KMS Encryption plugin's integration tests, on PostgreSQL and MySQL servers with LocalStack's KMS, a real KMS key, or Aurora.
It changes no product code.
It is the fifth PR in the KMS stack, after #1330, #1300, #1317 and #1331 (the plugin). Its base is #1331's branch, so its diff is the tests alone, and it stays a draft until the PRs below it merge.
tests/integration/container/test_kms_encryption_*.py,sqlalchemy/test_sqlalchemy_kms_encryption.pyanddjango/test_django_kms_encryption.py. They cover 66 planned cases: 57 on LocalStack, 6 that need a real key and 3 that need Aurora.RUN_KMS_ENCRYPTION_TESTS_ONLYandKMS_LOCALSTACKfeatures, three Gradle tasks, a LocalStack container pinned by digest with no published port, PostgreSQL statement logging in KMS environments, and a JUnit XML report beside each HTML report. Outside a KMS environment the KMS modules skip, and inside one everything else does.tests/integration/kms_local/, that runs the LocalStack modules on a laptop without Gradle, replacing the deletedtests/integration/test_kms_encryption_live.py.Lanes
tests/integration/kms_local/run.sh up,test PG,test MYSQL,down../gradlew test-kms-encryption-docker.run.sh real PGorreal MYSQL, or./gradlew test-kms-encryption-docker-kms../gradlew test-kms-encryption-aurora.KMS_MANUAL_TESTS=1adds three manual tests, which CI never runs.What CI runs
The new
kms_encryption_integration_tests.ymlruns three jobs on PostgreSQL and MySQL:kms-localstack: on pull requests that touch the wrapper, the integration tests,pyproject.toml,poetry.lockor the workflow, on pushes tomain, and on dispatch. It needs no secrets, so forks run it too.kms-real-key: on pushes tomainand on dispatch, through OIDC to a role limited to the test key.kms-aurora: on dispatch only, on two-instance clusters it deletes at the end.Each job fails when no KMS test ran, as an all-skipped pytest run exits 0.
Results
test_mysql_cursor_kinds_and_binary_prefixreran and failed as intended.run.sh realand the harness (on Finch, with two local workarounds). The three manual tests passed on PostgreSQL;test_via_service_condition_makes_every_statement_fail_closedhasn't rerun since a review fix changed it. A forced failure printed neither the account id nor the key id.test-kms-encryption-docker. Finch couldn't create its LocalStack container and Docker Desktop wasn't available, so this PR'skms-localstackjob is its first run.Expected failures
Both are strict xfails: a fix fails them until the mark goes.
test_fetchmany_without_a_size_reads_arraysize_rows[mysql-cext]:AwsWrapperCursor.fetchmany()defaultssizeto 0 and passes it on, so mysql-connector's C extension returns no rows.test_failover_and_read_write_splitting_keep_encryptingon Aurora PostgreSQL: after a writer failover, the plugin's metadata lookup connects to the old writer and raisesMetadataErrorabovefailover_v2, so failover never starts. mysql-connector fails earlier, atcursor(), whichfailover_v2handles.Administrator setup, once
subequal torepo:aws/aws-advanced-python-wrapper:ref:refs/heads/main(StringEquals) andaudequal tosts.amazonaws.com. It may callkms:GenerateDataKey,kms:Decryptandkms:DescribeKeyon the key andsts:AssumeRoleon the Decrypt-only role, with an explicit deny on key and alias changes.kms:Decrypton the key.main.KMS_KEY_ARN(a full key ARN, not an alias),KMS_REGION,KMS_IT_ROLE_ARNandKMS_DECRYPT_ONLY_ROLE_ARN.What to check
kms-localstackrun on both engines, including its "KMS tests run" count.test-dockerwithFILTER='test_direct_connection'passed).By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.