IGNITE-29177 Analyze and test the reliability of MDC historical rebalance mechanism by EgorBaranovEnjoysTyping · Pull Request #13714 · apache/ignite

EgorBaranovEnjoysTyping · 2026-10-20T12:34:40Z

…ance mechanism

Thank you for submitting the pull request to the Apache Ignite.

In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:

The Contribution Checklist

There is a single JIRA ticket related to the pull request.
The web-link to the pull request is attached to the JIRA ticket.
The JIRA ticket has the Patch Available state.
The pull request body describes changes that have been made.
The description explains WHAT and WHY was made instead of HOW.
The pull request title is treated as the final commit message.
The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
A reviewer has been mentioned through the JIRA comments
(see the Maintainers list)
The pull request has been checked by the Teamcity Bot and
the green visa attached to the JIRA ticket (see tab PR Check at TC.Bot - Instance 1 or TC.Bot - Instance 2)

Notes

If you need any help, please email dev@ignite.apache.org or ask any advice on https://the-asf.slack.com #ignite channel.

…ance mechanism

sergey-chugunov-1985 · 2026-10-22T12:59:23Z

+
+. A full rebalance is used if the node has no local data for the partition (update counter is zero).
+
+This applies both to a *new node joining for the first time* (it has no data on disk at all) and to a node that was *evicted from the baseline* and rejoined (its local partition data was invalidated). In both cases the full partition content must be downloaded from another node.


Suggested change

This applies both to a *new node joining for the first time* (it has no data on disk at all) and to a node that was *evicted from the baseline* and rejoined (its local partition data was invalidated). In both cases the full partition content must be downloaded from another node.

This applies both to a *new node joining for the first time* (it has no data on disk at all) and to a node that was *removed from the baseline* and than added back (its local partition data was invalidated). In both cases the full partition content must be downloaded from another node.

"Evicted" is the precise term in Ignite (cf. evictFromTopology). Also note the typo n the suggestion: "than" → "then".

sergey-chugunov-1985 · 2026-10-22T13:00:07Z

 In clusters with persistence, the baseline topology has to be changed manually (default behavior), or can be changed automatically when link:clustering/baseline-topology#baseline-topology-autoadjustment[automatic baseline adjustment] is enabled.
 ====

+== Rebalance Strategy [[rebalance-strategy]]


Suggested change

== Rebalance Strategy [[rebalance-strategy]]

== Rebalancing Strategy [[rebalance-strategy]]

sergey-chugunov-1985 · 2026-10-22T14:43:10Z

+
+Example: a partition contains 100 entries. While the node was absent, 500 writes hit this partition. Downloading 500 delta entries is more expensive than just downloading all 100 entries → full rebalance.
+
+. A full rebalance is used if the partition size is below the `historical.rebalance.threshold` (DMS property, default 500 entries), unless the `IGNITE_PREFER_WAL_REBALANCE` system property is set to `true`.


Suggested change

. A full rebalance is used if the partition size is below the `historical.rebalance.threshold` (DMS property, default 500 entries), unless the `IGNITE_PREFER_WAL_REBALANCE` system property is set to `true`.

. A full rebalance is used if the partition size is below the `historical.rebalance.threshold` (a property stored in Distributed MetaStorage, default 500 entries), unless the `IGNITE_PREFER_WAL_REBALANCE` system property is set to `true`.

It's the standard abbreviation used throughout the codebase (DistributedMetaStorage,
GridDistributedMetastorageManager). Expanding it to "Distributed MetaStorage" (two words) would actually be incorrect — the canonical name is
"Distributed Meta Storage" (three words).

sergey-chugunov-1985 · 2026-10-22T14:54:41Z

+
+[NOTE]
+====
+If both nodes report the same update counter for a partition, no rebalance is performed for that partition.


Suggested change

If both nodes report the same update counter for a partition, no rebalance is performed for that partition.

If both nodes report the same update counter for a partition, rebalance doesn't start for that partition.

The formal phrasing is preferred in documentation, and "performed" is more precise than "start"
(nothing starts, it simply doesn't happen).

sergey-chugunov-1985 · 2026-10-22T14:58:11Z

+
+[NOTE]
+====
+*Historical rebalance requires the delta to be present in the WAL of the source node.* If the WAL segments that contain the delta were already archived or deleted by the time the node rejoins — for example, the node was absent for a long time, or WAL history and archiving are configured with a small retention — the source node cannot read the delta. In that case Ignite does not lose data: the supplier detects the failure to iterate over the WAL, marks the last checkpoint as not applicable for WAL rebalance, marks the affected partitions as missed, and the node falls back to a *full rebalance*.


Suggested change

*Historical rebalance requires the delta to be present in the WAL of the source node.* If the WAL segments that contain the delta were already archived or deleted by the time the node rejoins — for example, the node was absent for a long time, or WAL history and archiving are configured with a small retention — the source node cannot read the delta. In that case Ignite does not lose data: the supplier detects the failure to iterate over the WAL, marks the last checkpoint as not applicable for WAL rebalance, marks the affected partitions as missed, and the node falls back to a *full rebalance*.

*Historical rebalance requires the delta to be present in the WAL of the source node.* If the WAL segments that contain the delta were already archived or deleted by the time node lagged behind rejoins — for example, the node was absent for a long time, or WAL history and archiving are configured with a small retention — the source node cannot supply the delta. In that case Ignite does not lose data: the supplier detects the failure to iterate over the WAL, marks the last checkpoint as not applicable for WAL rebalance, marks the affected partitions as missed, and the node falls back to a *full rebalance*.

keeping the original ("the node rejoins"); the context already makes it clear which node and that it was
behind. "cannot supply the delta" — done.

sergey-chugunov-1985 · 2026-10-22T14:59:21Z

+The `rebalanceDelay` (deprecated) setting also prevents a historical rebalance: when a delay is configured, the rebalance is scheduled outside of the partition map exchange context, and the historical path (which is tied to that context) is never taken — every partition is restored with a full rebalance regardless of the delta size and regardless of `IGNITE_PREFER_WAL_REBALANCE`.
+====
+
+In a link:clustering/multi-data-center[multi-data-center] setup, the preloader prefers a node from the same data center that holds a copy of the partition. If no such node is available, it falls back to a node from a remote data center. This behavior applies to both full and historical rebalance.


Suggested change

In a link:clustering/multi-data-center[multi-data-center] setup, the preloader prefers a node from the same data center that holds a copy of the partition. If no such node is available, it falls back to a node from a remote data center. This behavior applies to both full and historical rebalance.

In a link:clustering/multi-data-center[multi-data-center] setup, the node demanding the data prefers a node from the same data center that holds a copy of the partition. If no such node is available, it falls back to a node from a remote data center. This behavior applies to both full and historical rebalance.

"Preloader" is the actual component name in the codebase (GridDhtPreloader). The node doesn't "prefer"
anything; the preloader mechanism does.

sergey-chugunov-1985 · 2026-10-22T15:10:28Z

    public void testHistoricalRebalanceCrossDC() throws Exception {
        checkHistoricalRebalance(true);
    }

+    /**
+     * Verifies that no rebalance demand is sent when a node restarts without any


Suggested change

* Verifies that no rebalance demand is sent when a node restarts without any

* Verifies that no rebalance request is sent when a node restarts without any

This is the protocol-level term used throughout the codebase (GridDhtPartitionDemandMessage) and in
the documentation itself.

sergey-chugunov-1985 · 2026-10-22T15:13:31Z

A lot of duplicated code shared between tests: start few nodes, activate nodes, create cache config, create cache, populate cache. Is it possible to extract at least some of these scenarios into separate methods?

…ance mechanism

baranov.e.g added 3 commits October 20, 2026 15:32

IGNITE-29177 Analyze and test the reliability of MDC historical rebal…

036ed7d

…ance mechanism

IGNITE-29177 Analyze and test the reliability of MDC historical rebal…

d72822d

…ance mechanism

IGNITE-29177 Analyze and test the reliability of MDC historical rebal…

513bd50

…ance mechanism

sergey-chugunov-1985 reviewed Oct 22, 2026

View reviewed changes

IGNITE-29177 Analyze and test the reliability of MDC historical rebal…

3369535

…ance mechanism

	This applies both to a new node joining for the first time (it has no data on disk at all) and to a node that was evicted from the baseline and rejoined (its local partition data was invalidated). In both cases the full partition content must be downloaded from another node.
	This applies both to a new node joining for the first time (it has no data on disk at all) and to a node that was removed from the baseline and than added back (its local partition data was invalidated). In both cases the full partition content must be downloaded from another node.

	== Rebalance Strategy [[rebalance-strategy]]
	== Rebalancing Strategy [[rebalance-strategy]]

	. A full rebalance is used if the partition size is below the `historical.rebalance.threshold` (DMS property, default 500 entries), unless the `IGNITE_PREFER_WAL_REBALANCE` system property is set to `true`.
	. A full rebalance is used if the partition size is below the `historical.rebalance.threshold` (a property stored in Distributed MetaStorage, default 500 entries), unless the `IGNITE_PREFER_WAL_REBALANCE` system property is set to `true`.

	If both nodes report the same update counter for a partition, no rebalance is performed for that partition.
	If both nodes report the same update counter for a partition, rebalance doesn't start for that partition.

	In a link:clustering/multi-data-center[multi-data-center] setup, the preloader prefers a node from the same data center that holds a copy of the partition. If no such node is available, it falls back to a node from a remote data center. This behavior applies to both full and historical rebalance.
	In a link:clustering/multi-data-center[multi-data-center] setup, the node demanding the data prefers a node from the same data center that holds a copy of the partition. If no such node is available, it falls back to a node from a remote data center. This behavior applies to both full and historical rebalance.

	* Verifies that no rebalance demand is sent when a node restarts without any
	* Verifies that no rebalance request is sent when a node restarts without any

Conversation

EgorBaranovEnjoysTyping commented Oct 20, 2026

The Contribution Checklist

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants