Skip to content

IGNITE-29084 Fix frozen U.currentTimeMillis() in tests after a leaked node is stopped - #13639

Open
rezataroosheh wants to merge 4 commits into
apache:masterfrom
rezataroosheh:IGNITE-29084
Open

rezataroosheh wants to merge 4 commits into
apache:masterfrom
rezataroosheh:IGNITE-29084

Conversation

@rezataroosheh

@rezataroosheh rezataroosheh commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

What

U.currentTimeMillis() could stop being updated in tests. GridAbstractTest now always starts the test clock (GridTestClockTimer). If an ignite-clock timer is already running (started by a node that a previous test left running), the test clock stops it and takes its usual gridCnt slot, so exactly one clock runs: the test one. CommonUtils is not changed.

Why

GridTestClockTimer used to suppress the internal ignite-clock timer by adding a fake node to gridCnt, and it started only if gridCnt == 0. If a previous test left a node running, the test clock was never started. Once the leaked node stopped, gridCnt dropped to 0, the internal timer stopped, and time stayed frozen for every later test. This showed up as a PlannerTimeoutTest failure after the JUnit 5 migration changed the test execution order (see the discussion in #13602).

The test clock now always starts and stops the internal timer of any leaked node. The internal timer is never started while the test clock is active.

Tests

The Contribution Checklist

  • There is a single JIRA ticket related to the pull request.
  • The web-link to the pull request is attached to the JIRA ticket.
  • The JIRA ticket has the Patch Available state.
  • The pull request body describes changes that have been made.
    The description explains WHAT and WHY was made instead of HOW.
  • The pull request title is treated as the final commit message.
    The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
  • A reviewer has been mentioned through the JIRA comments
    (see the Maintainers list)
  • The pull request has been checked by the Teamcity Bot and
    the green visa attached to the JIRA ticket (see tab PR Check at TC.Bot - Instance 1 or TC.Bot - Instance 2)

Notes

If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com #ignite channel.

@rezataroosheh rezataroosheh changed the title IGNITE-29084 Fix frozen U.currentTimeMillis() in tests after a leaked… IGNITE-29084 Fix frozen U.currentTimeMillis() in tests after a leaked node is stopped Oct 1, 2026
@rezataroosheh
rezataroosheh marked this pull request as ready for review October 2, 2026 20:13
@zstan
zstan requested a balanced review from Copilot October 16, 2026 07:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The synchronized clock handoff addresses the reported failure, regression coverage is included, and no blocking issues were identified.

0 open findings

What changed in this PR

Fixes frozen test time after a leaked node stops by separating external clock ownership from the real-node count.

Changes:

  • Adds synchronized external-clock ownership and internal-timer shutdown.
  • Adds regression coverage for leaked nodes and continued clock updates.
  • Removes the obsolete planner-test node-count guard.
File Description
modules/​core/​src/​test/​java/​org/​apache/​ignite/​testsuites/​IgniteUtilSelfTestSuite.java Registers clock regression tests.
modules/​core/​src/​test/​java/​org/​apache/​ignite/​internal/​util/​IgniteClockTimerTest.java Tests clock ownership and leaked-node behavior.
modules/​core/​src/​test/​java/​org/​apache/​ignite/​internal/​util/​GridTestClockTimer.java Uses external-clock ownership instead of a fake node count.
modules/​commons/​src/​main/​java/​org/​apache/​ignite/​internal/​util/​CommonUtils.java Separates clock ownership from node lifecycle tracking.
modules/​calcite/​src/​test/​java/​org/​apache/​ignite/​internal/​processors/​query/​calcite/​planner/​PlannerTimeoutTest.java Removes the obsolete node-count assertion.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@zstan

zstan commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

@rezataroosheh Sorry but I don`t like an idea with additional flag, we need to understand how can we obtain a situation when gridCnt != 0 but there is no working : timer = new IgniteThread(igniteInstanceName, "ignite-clock" thread.

@rezataroosheh

Copy link
Copy Markdown
Contributor Author

Thank you @zstan for review and feedback.
You're right, gridCnt != 0 without a working ignite-clock thread doesn't happen. The frozen state is actually gridCnt == 0 with no clock at all:

  1. A test leaves nodes running (e.g. ScriptTestSuite).
  2. GridAbstractTest's static initializer runs while those nodes are still alive. Since gridCnt != 0, the test clock is not started; only their ignite-clock is running.
  3. stopAllGrids() in @afterall of GridCommonAbstractWrapperTest stops all nodes in the JVM, including the leaked ones. gridCnt drops to 0 and ignite-clock stops.
  4. Now nothing updates the time, so U.currentTimeMillis() is frozen.

I've reworked the fix without the flag, and CommonUtils is no longer changed:

  • GridAbstractTest always starts the test clock.
  • When the test clock starts, it stops an ignite-clock that is already running and takes its usual gridCnt slot, as before.

So there's always exactly one clock, and it's the test one. IgniteClockTimerTest emulates the leaked-node case.

assertFalse(internalClockRunning());
}
finally {
// Restores the test clock counter and stops the internal clock timer if the test failed in between.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I miss - why we need to restore it ? Initially it initialized statically from GridAbstractTest, thus if no inheritance and active instances - no need to restore ? or i`m wrong ? Clarify plz ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think fix need to be close to IgniteUtils.gridCnt logic, some inconsistences may occured there: gridCnt != 0 while timer = new IgniteThread(igniteInstanceName, "ignite-clock" is stopped.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zstan I checked whether CommonUtils itself can end up with gridCnt != 0 and the ignite-clock stopped. Both are changed only under mux: the clock starts when gridCnt goes from 0 -> 1 and stops when it drops to 0, and the clock loop exits only on interrupt. I also tried a node that fails early (in DiagnosticProcessor, before U.onGridStart()) while another node is running; gridCnt and the clock stay correct there. So I couldn't find a way to reach that state in production code.

In tests, gridCnt != 0 without ignite-clock is the intended state: GridTestClockTimer has always taken a slot (gridCnt++) so ignite-clock doesn't start, and the test clock runs instead. The freeze came from the hand-off between ignite-clock and the test clock, which is test code, so I kept the fix there.

About restoring the counter in the test: line 69 sets gridCnt = 0 on purpose, and the GridAbstractTest static initializer runs only once per JVM, so nothing else would fix it. If the test fails before line 81, the next tests in IgniteUtilSelfTestSuite would start ignite-clock alongside the test clock, and it would override the time mocked via timeSupplier(). The finally block only covers that failure case. new GridTestClockTimer() doesn't start a thread; it only fixes the counter.

If you have a scenario where gridCnt and ignite-clock get out of sync in CommonUtils, could you share it? I'll look into it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I attach time_repro (plz apply it to master branch) into an issue, Just run IgniteCalciteTestSuite2 and you will see that PlannerTimeoutTest will fail, I comment :

@AfterAll
public static void tearDown() {
    //IgnitionEx.stopAll(true, null);
}

just to emulate a situation when someone forgot to stop a node.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zstan Thanks for the repro! I ran IgniteCalciteTestSuite2 with time_repro.patch:

  • master: PlannerTimeoutTest fails in init(): expected:<1> but was:<2>.
  • this PR: PlannerTimeoutTest passes.

On master, the failure comes from the gridCnt check, not from the clock. When PlannerTimeoutTest starts, the 2 leaked ScriptTestSuite nodes are still running, so gridCnt is 2, and their ignite-clock keeps time moving. The freeze comes one step later: PlannerTimeoutTest's @afterall calls stopAllGrids(), which stops all nodes in the JVM, including the leaked ones. On master, gridCnt then drops to 0 and ignite-clock stops. The test clock was never started, so subsequent tests see a frozen U.currentTimeMillis(). That's the case IgniteClockTimerTest.testLeakedNodeDoesNotFreezeClock covers.

You're right that you should check gridCnt. The old check assertEquals(1, gridCnt) assumes no other nodes are running, so a forgotten node breaks it even when the counter is correct. I restored it in a form that always holds: G.allGrids().size() + 1 (each running node's slot plus the test clock's slot). With your repro:

  • this PR: gridCnt = 3, nodes + 1 = 3, passes.
  • master: gridCnt = 2, nodes + 1 = 3, fails with expected:<3> but was:<2>, because the test clock's slot is missing.

So the check now catches the real problem instead of the leaked node itself. Pushed in 9cccac9.

The other failures in the suite (TpchQueryPlannerTest plan diffs and the LIMIT 1.5 script in ScriptTestSuite) are the same on master and on this branch, so they're unrelated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants