Conversation
The root namespace worker factory was shut down by its destroy method without waiting, and the service stubs were closed right after it, so pollers and in-flight activity completions failed with UNAVAILABLE: Channel shutdown invoked. Non-root namespace factories were shut down only when another context closed. Stop every worker factory from a SmartLifecycle that waits for termination within spring.lifecycle.timeout-per-shutdown-phase, before the beans are destroyed.
A shut down WorkerFactory cannot be started again. Spring Framework 7 pauses a cached test context when another one is started and restarts it when a test reuses it, so the stopped factory left the reused context without pollers and its workflows never progressed.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
WorkerFactoryLifecycle, aSmartLifecyclethat shuts down a startedWorkerFactorywhen the context stops and waits for its workers to terminate. The wait is bounded by the standardspring.lifecycle.timeout-per-shutdown-phase.isPauseable()returnsfalse, Spring Framework 7): a shut down factory cannot be started again, so a paused context, such as a cached Spring TestContext one, keeps its workers running.temporalWorkerFactoryLifecyclebean, and each non-root namespace registers its own one next to itsWorkerFactory.ContextClosedEventbranch fromNonRootNamespaceEventListener. It shut down the non-root factories only when some other context closed, and did not wait.GracefulShutdownTest: an activity is still running when the context is closed, and the test checks that it has completed and the factory has terminated onceclose()returns.Why?
The root
WorkerFactoryis shut down by itsdestroyMethodwithout waiting, andtemporalWorkflowServiceStubsis destroyed right after it. Pollers and in-flight activity completions then fail withUNAVAILABLE: Channel shutdown invoked, and the running activity is cut off. Stopping the factory in the lifecycle phase lets in-flight tasks finish before any bean is destroyed.Closes #2026. It uses the existing Spring Boot shutdown timeout property instead of adding a new one.
Breaking changes
None. Context close now waits for in-flight tasks, up to
spring.lifecycle.timeout-per-shutdown-phase(30s by default).Server PR
N/A.
Testing
GracefulShutdownTestfails on currentmain(the activity has not completed whenclose()returns) and passes with the change. The full module passes on Spring Boot 2, 3 and 4.testWorkersKeepRunningWhenContextIsPausedAndRestartedpauses and restarts the context and then runs a workflow. On Spring Boot 4 it times out withoutisPauseable()(the restarted context has no pollers) and passes with it; on Spring Boot 2 and 3 it is skipped, as context pausing comes with Spring Framework 7.GracefulShutdownTestwas run on Spring Boot 2 and 4 for this commit.