Repository navigation
IGNITE-29084 Fix frozen U.currentTimeMillis() in tests after a leaked node is stopped - #13639
rezataroosheh wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
The synchronized clock handoff addresses the reported failure, regression coverage is included, and no blocking issues were identified.
0 open findings
What changed in this PR
Fixes frozen test time after a leaked node stops by separating external clock ownership from the real-node count.
Changes:
- Adds synchronized external-clock ownership and internal-timer shutdown.
- Adds regression coverage for leaked nodes and continued clock updates.
- Removes the obsolete planner-test node-count guard.
| File | Description |
|---|---|
| modules/core/src/test/java/org/apache/ignite/testsuites/IgniteUtilSelfTestSuite.java | Registers clock regression tests. |
| modules/core/src/test/java/org/apache/ignite/internal/util/IgniteClockTimerTest.java | Tests clock ownership and leaked-node behavior. |
| modules/core/src/test/java/org/apache/ignite/internal/util/GridTestClockTimer.java | Uses external-clock ownership instead of a fake node count. |
| modules/commons/src/main/java/org/apache/ignite/internal/util/CommonUtils.java | Separates clock ownership from node lifecycle tracking. |
| modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/planner/PlannerTimeoutTest.java | Removes the obsolete node-count assertion. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
@rezataroosheh Sorry but I don`t like an idea with additional flag, we need to understand how can we obtain a situation when gridCnt != 0 but there is no working : |
|
Thank you @zstan for review and feedback.
I've reworked the fix without the flag, and CommonUtils is no longer changed:
So there's always exactly one clock, and it's the test one. IgniteClockTimerTest emulates the leaked-node case. |
| assertFalse(internalClockRunning()); | ||
| } | ||
| finally { | ||
| // Restores the test clock counter and stops the internal clock timer if the test failed in between. |
There was a problem hiding this comment.
I miss - why we need to restore it ? Initially it initialized statically from GridAbstractTest, thus if no inheritance and active instances - no need to restore ? or i`m wrong ? Clarify plz ?
There was a problem hiding this comment.
I think fix need to be close to IgniteUtils.gridCnt logic, some inconsistences may occured there: gridCnt != 0 while timer = new IgniteThread(igniteInstanceName, "ignite-clock" is stopped.
There was a problem hiding this comment.
@zstan I checked whether CommonUtils itself can end up with gridCnt != 0 and the ignite-clock stopped. Both are changed only under mux: the clock starts when gridCnt goes from 0 -> 1 and stops when it drops to 0, and the clock loop exits only on interrupt. I also tried a node that fails early (in DiagnosticProcessor, before U.onGridStart()) while another node is running; gridCnt and the clock stay correct there. So I couldn't find a way to reach that state in production code.
In tests, gridCnt != 0 without ignite-clock is the intended state: GridTestClockTimer has always taken a slot (gridCnt++) so ignite-clock doesn't start, and the test clock runs instead. The freeze came from the hand-off between ignite-clock and the test clock, which is test code, so I kept the fix there.
About restoring the counter in the test: line 69 sets gridCnt = 0 on purpose, and the GridAbstractTest static initializer runs only once per JVM, so nothing else would fix it. If the test fails before line 81, the next tests in IgniteUtilSelfTestSuite would start ignite-clock alongside the test clock, and it would override the time mocked via timeSupplier(). The finally block only covers that failure case. new GridTestClockTimer() doesn't start a thread; it only fixes the counter.
If you have a scenario where gridCnt and ignite-clock get out of sync in CommonUtils, could you share it? I'll look into it.
There was a problem hiding this comment.
I attach time_repro (plz apply it to master branch) into an issue, Just run IgniteCalciteTestSuite2 and you will see that PlannerTimeoutTest will fail, I comment :
@AfterAll
public static void tearDown() {
//IgnitionEx.stopAll(true, null);
}
just to emulate a situation when someone forgot to stop a node.
There was a problem hiding this comment.
@zstan Thanks for the repro! I ran IgniteCalciteTestSuite2 with time_repro.patch:
- master: PlannerTimeoutTest fails in init(): expected:<1> but was:<2>.
- this PR: PlannerTimeoutTest passes.
On master, the failure comes from the gridCnt check, not from the clock. When PlannerTimeoutTest starts, the 2 leaked ScriptTestSuite nodes are still running, so gridCnt is 2, and their ignite-clock keeps time moving. The freeze comes one step later: PlannerTimeoutTest's @afterall calls stopAllGrids(), which stops all nodes in the JVM, including the leaked ones. On master, gridCnt then drops to 0 and ignite-clock stops. The test clock was never started, so subsequent tests see a frozen U.currentTimeMillis(). That's the case IgniteClockTimerTest.testLeakedNodeDoesNotFreezeClock covers.
You're right that you should check gridCnt. The old check assertEquals(1, gridCnt) assumes no other nodes are running, so a forgotten node breaks it even when the counter is correct. I restored it in a form that always holds: G.allGrids().size() + 1 (each running node's slot plus the test clock's slot). With your repro:
- this PR: gridCnt = 3, nodes + 1 = 3, passes.
- master: gridCnt = 2, nodes + 1 = 3, fails with expected:<3> but was:<2>, because the test clock's slot is missing.
So the check now catches the real problem instead of the leaked node itself. Pushed in 9cccac9.
The other failures in the suite (TpchQueryPlannerTest plan diffs and the LIMIT 1.5 script in ScriptTestSuite) are the same on master and on this branch, so they're unrelated.
9cccac9 to
fe2d920
Compare
What
U.currentTimeMillis()could stop being updated in tests.GridAbstractTestnow always starts the test clock (GridTestClockTimer). If anignite-clocktimer is already running (started by a node that a previous test left running), the test clock stops it and takes its usualgridCntslot, so exactly one clock runs: the test one.CommonUtilsis not changed.Why
GridTestClockTimerused to suppress the internalignite-clocktimer by adding a fake node togridCnt, and it started only ifgridCnt == 0. If a previous test left a node running, the test clock was never started. Once the leaked node stopped,gridCntdropped to 0, the internal timer stopped, and time stayed frozen for every later test. This showed up as aPlannerTimeoutTestfailure after the JUnit 5 migration changed the test execution order (see the discussion in #13602).The test clock now always starts and stops the internal timer of any leaked node. The internal timer is never started while the test clock is active.
Tests
IgniteClockTimerTest, added toIgniteUtilSelfTestSuite, which also reproduces the leaked-node scenario.PlannerTimeoutTest: removed thegridCntguard added in IGNITE-29064 Change calcite module dependency for junit framework from 4 to 5 #13602, which is no longer needed.The Contribution Checklist
The description explains WHAT and WHY was made instead of HOW.
The following pattern must be used:
IGNITE-XXXX Change summarywhereXXXX- number of JIRA issue.(see the Maintainers list)
the
green visaattached to the JIRA ticket (see tabPR Checkat TC.Bot - Instance 1 or TC.Bot - Instance 2)Notes
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com #ignite channel.