Skip to content

test: add Pipe, metadata, LOAD, deletion, and write recovery ITs - #19051

Draft
Caideyipi wants to merge 34 commits into
masterfrom
test/pipe-p0-it-master
Draft

Caideyipi wants to merge 34 commits into
masterfrom
test/pipe-p0-it-master

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Add integration coverage for recovery windows that can lose replicated or loaded data, leave stale metadata, or retain staging resources indefinitely.

Pipe recovery

  • Exercise receiver outages, history/realtime handoff, multiple actual DataRegions and writers, rejected requests, lost successful responses, and queue/backpressure isolation.
  • Cover runtime STOP/START, failed and successful ALTER, DROP and same-name recreation, role revocation, interrupted TsFile transfer, schema changes, Region redistribution, processor restart, forwarding loops, and ConfigNode restart during DROP.

Metadata recovery

  • Cover live-node lease fencing/resynchronization, DDL procedure rollback or continuation across ConfigNode failover, cross-SchemaRegion deletion, asynchronous DROP and same-name recreation, delayed attribute batches, and SchemaRegion snapshot installation followed by log-tail recovery.

  • Retain schema-log tag/alias recovery, database TTL and view-cache recovery, and ConfigNode catalog snapshot plus subsequent DDL recovery.

  • Check Tree tag and alias indexes across DataNode restart and same-name recreation: stale tag keys and values must disappear, recreated aliases and data must resolve correctly, and SHOW/COUNT must preserve the exact retained series.

  • Schema-log recovery uses a nonempty durable mlog and a forced restart. Catalog recovery observes snapshot loading on the restarted peer, then requires that peer to become leader before checking the complete catalog, data, and follow-up DDL.

  • Hold an old remote-fetch response across an ATTRIBUTE update and verify that it cannot repopulate stale cache entries.

  • Strengthen DROP/MOD progress recovery with actual LAST-cache-hit results, retained JDBC/TableSession clients, and Tablet writes after same-name recreation.

LOAD TsFile recovery

  • P0-04: keep the original committed task's staged TsFile/mods/progress and WAL payload references across the IoTConsensus writer's process restart while a real replica is behind. Observe the actual cleaner scan and retain the same file set and bytes until durable peer checkpoints permit cleanup, without additional writes. Check physical data on every replica, per-file mods, duplicate timestamps, and complete query values/types.
  • P0-06: require original async Tree and Table pending handoffs to complete after a witnessed temporary failure is removed, including process restart, without resubmission. Check same-named Tree files and their distinct mods, source ownership, queue cleanup, explicit Table databases, null TAG/FIELD values, and complete ordered schema.
  • Keep the Tree schema conflict until an original identity handoff is rejected for missing WRITE_DATA and moved to failed; then release the other handoffs. Stop owned cluster processes before directory cleanup.

The LOAD additions are limited to three IT files and add no production hooks or interfaces. Existing fault-injection agents remain in test sources; production metadata hooks removed earlier are not restored. Compatible scenarios share class-scoped clusters.

Deletion integrity

  • Check measurement and device isolation within a physically shared TsFile across full deletion, rewrite, and forced restart.
  • Check that deleted points stay absent and later rewrites survive sequence/unsequence cross-space compaction and forced restart, while preserving other measurements and devices and allowing continued writes/deletes.
  • Keep ATTRIBUTE-filter deletion bound to the devices selected when DELETE completes, even after matching and nonmatching attributes change. Cover NULL versus empty TAGs and isolation from another table.
  • Use test-only helpers to inspect closed-file membership and physical chunk values before checking the complete SQL results.

Write recovery and admission

  • Recover a flushed baseline together with acknowledged unflushed Tree and Table writes, multiple sealed WAL segments, partition boundaries, sparse fields, and overwrites. Check complete values and TAG combinations before and after crash recovery and continued writes.
  • Preserve partial successes and fully rejected fields across forced restart, distinguish explicit all-NULL Table rows from rejected-only requests, and reject phantom rows or devices.
  • Route one multi-device batch through different single-replica DataRegions while one owner is READONLY; verify the successful and rejected subsets, retry only the failed target, and check values and NULLs after flush and restart.
  • Retain the original Session and size-one SessionPool across loss and recovery of an actually cached Ratis write leader. Use fixed-key retries, a success ledger, and connection-pool recovery assertions.
  • Trigger real write-memory admission rejection with bounded requests. Verify reservation rollback, recovery of the original writers, positive accounting for recovered writes in the rejected database, and release of both databases' accounting after flush.

The write additions are limited to six IT files and add no production hooks. Preserve the target branch's existing seed-fallback cases while adapting the new route-recovery fixture.

Validation

  • The tag/alias update at 4ef74f9d90ee adds IoTDBTagAlterIT#tagAliasIndexSurvivesRestartAndCanBeRebuilt with the reviewed negative-index and exact retained-path assertions. Spotless, whitespace, exact-content, and single-file commit-scope checks passed. Target execution was attempted but stopped during remote verification-harness preparation before compilation or IT execution; SimpleIT and ClusterIT Memory/PBTree variants remain unverified at this head. The IT selection check also did not complete on Windows because its shard-list subprocess exited 1. Earlier master results and the older isolated-checkout PBTree failure are historical evidence only.

  • The retained metadata additions passed 3 configurations on Linux / OpenJDK 17: 2 methods under TableClusterIT, plus DROP/MOD under TableSimpleIT, with 0 failures/errors/skips for those retained methods. The broader cluster command exited 1 because the SchemaRegion migration case failed at DELETE DEVICES with a null TDataNodeLocation; that entire new case was withdrawn from this update. The single-node Maven command exited 0.

  • The original complete metadata rerun reported 4 passing and 5 failing configurations. Failed propagation scenarios remain excluded. That historical rerun also had a TagAlter PBTree failure; the corrected, complete parameterized TagAlter case is now included, with its current-head verification reported separately above. Assertions and scenarios were not weakened.

  • Target metadata validation rebuilt the required IoTDB dependency closure from clean source in an isolated reactor, then reused those exact source-built artifacts under recorded provenance for the focused fixes. This was not a fresh uncached full build. The IT module compiled. The controller uses the target fixture's existing JVM options API; runtime ASM scope keeps cglib's ASM dependency in ordinary child-node templates.

  • Scoped metadata Spotless, source-selection, final source-hash and whitespace checks passed. Existing module Checkstyle excludes all inputs because its exclusion expression has an empty alternative; its successful exit is not counted as effective validation, and that configuration remains unchanged.

  • All newly added write scenarios were run locally before transfer. Five files match the independently reviewed source byte for byte. The Session route-recovery addition preserves the reviewed write logic while omitting an audit setup helper unavailable in the target fixture; existing target seed-fallback tests remain unchanged.

  • Spotless passed for the adapted Session file; exact-content, compatibility, whitespace, and six-file commit-scope checks passed. This transfer did not rerun executable tests on the target branch or validate the combined PR head. The adapted route-recovery method, complete write selection, Linux, and Chinese locale remain unverified at this head.

  • The three added deletion scenarios were run locally and passed before transfer. Their test bodies, shared setup, and supporting utility code match the independently reviewed source at 28c5028bb462 plus the working changes. This transfer did not rerun executable tests or validate the combined PR head.

  • Spotless passed for all four deletion files; whitespace, commit-scope, and exact-content checks passed. The deletion update changes test sources only.

  • All newly added LOAD scenarios have been run locally on Apache IoTDB / JDK 17 with ClusterIT and TableClusterIT. A combined selection passed 4 tests, 0 failures, 0 errors, 0 skipped. After strengthening the ordered Table schema assertion, the latest focused run passed 1 test, 0 failures, 0 errors, 0 skipped, with Maven exit code 0.

  • Those LOAD runs used source revision 28c5028bb462 plus the working changes. The exact reviewed three-file contents were transferred into this PR. Existing source-built dependencies and node templates were reused under recorded provenance; this was not a fresh uncached full build.

  • Earlier metadata P0 verification at 030e5c543f9 passed 17 tests, 0 failures, 0 errors, 0 skipped. The retained adapted schema-log and catalog recovery cases subsequently passed individually. These historical results are not a full-PR-head validation.

  • Spotless passed for the LOAD source changes; whitespace and exact file-content checks passed for this update. No executable test was rerun while transferring the unchanged files to the PR branch.

  • Existing Pipe and earlier metadata suites were not rerun in this transfer. The full IT suite, other editions/profiles, and Chinese locale remain unverified. The complete combined PR diff still requires a fresh independent review. Directory-cleanup failures were not injected.

  • Precise LOAD PIECE/PREPARE faults, lost COMMIT responses, and coordinator-orphan task recovery (P0-01/02/03/05) remain outside this addition. Ordinary restart/throttling and the empty cleaner scan probe are not counted as that coverage.

Review status

Keep this PR as a draft pending an independent review of the complete updated PR diff and context. The individual workspace reviews do not approve the complete combined PR diff. The write transfer includes a target-fixture adaptation and also requires independent review at the updated head.

The Pipe-focused workflow-dispatch input pipe_p0_only continues to select IoTDBPipeRecoveryIT under MultiClusterIT2DualTreeAutoEnhanced.

@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 48.11%. Comparing base (1b2158f) to head (986e23a).
⚠️ Report is 9 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19051      +/-   ##
============================================
- Coverage     48.12%   48.11%   -0.01%     
  Complexity      625      625              
============================================
  Files          5572     5572              
  Lines        411488   411557      +69     
  Branches      54166    54195      +29     
============================================
+ Hits         198010   198041      +31     
- Misses       213478   213516      +38     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Caideyipi

Caideyipi commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator Author

I reviewed the P1 metadata recovery test additions in commit c5ebc9b1c15d. I found no concrete correctness issues in the covered scenarios; the partial items below are coverage gaps, not code defects.

  • P1-1: Partial — concurrent same/conflicting auto-creation and ConfigNode leader failover are checked; SchemaRegion leader failover is not covered.
  • P1-2: Partial — template extension, interrupted deactivation, and recovered results are checked, but the missed-broadcast DataNode is stopped and restarted, so recovery of a still-running node with stale cache is not covered.
  • P1-3: Covered — schema-log recovery checks tag/alias indexes, complete SHOW/count results, and follow-up reads and writes.
  • P1-4: Partial — database TTL inheritance, an overridden table TTL, expired-read filtering, and the recovered-node write boundary are checked; dynamic per-table or Tree parent/child TTL changes are not covered.
  • P1-5: Partial — Table tree-to-table auto-sync is checked after rename/replace/disable and node recovery; Tree logical-view recovery is not covered.
  • P1-6: Partial — a real SchemaRegion migration is verified with metadata changes and post-migration reads/writes; other migration/rebuild failure stages are not covered.
  • P1-7: Covered — quota exhaustion, failed activation, deletion/restart, bounded quota refresh, retry, and subsequent writes are checked.
  • P1-8: Covered — catalog snapshot, subsequent DDL, ConfigNode recovery and leadership transfer, catalog state, reads/writes, and continued DDL are checked.

Validation: the IT-enabled reactor compilation passed across 37 modules in the source worktree. The Failsafe report confirms only IoTDBSchemaLogRecoveryIT executed (1 test passed, 0 failures); the other new ITs were not executed. The PR remains a draft.

@Caideyipi Caideyipi changed the title test(pipe): add P0 recovery integration tests test: add Pipe and metadata recovery integration tests Oct 30, 2026
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Metadata P0 review/fix round for 030e5c543f9efe3b378e6e01c77fde35a4b875bb.

This round found and fixed test-logic and fixture precondition issues:

  • Use supported equality predicates and complete device keys for cross-Region DELETE DEVICES; retain full metadata/data assertions and the unaffected-device checks.
  • Make snapshot installation real by reducing the SchemaRegion log purge gap only inside the test agent, checking actual log truncation and a newer installed snapshot, selecting an actual follower distinct from the SQL coordinator, and addressing leader-transfer RPCs to the intended new leader.
  • Give the PBTree fixture enough schema memory for its pinned Region roots; prepare template catalog extensions while all nodes are online; assert the alias query's actual column header.
  • Wait for all actual IoTConsensus DataRegion replicas before the DROP fixture's flush/stop. Recreate only after the exact SUCCESS / written=1 / remaining=0 result while the stale DataNode remains offline.
  • Wait for a direct RPC response from the restarted ConfigNode before reusing the shared cluster.

Final validation: Apache IoTDB on JDK 17, ClusterIT + TableClusterIT, 3 ConfigNodes/3 DataNodes. 17 tests passed, with 0 failures, 0 errors and 0 skips; Maven exited 0. Tree ran in Memory/PBTree and Table in Memory. The Failsafe class XML reports confirm the counts. Spotless and git diff --check passed. Required IoTDB dependencies were built from clean source before the provenance-matched retries.

The new P0 fault agent and controls stay under integration-test/src/test; no new production hook was added. Existing P1 production hooks are unchanged and remain a follow-up/full-PR review concern. The existing Pipe/P1 suites were not rerun in this round.

This round does not approve the full PR. It remains a draft pending a fresh independent review of the latest complete diff and context.

Verify IoTConsensus staged payload retention across writer restart until a slow replica reaches the durable checkpoint, with physical values, mods and duplicate checks. Complete Tree and Table async retry recovery from original pending handoffs, including stored identity, database routing and ordered schema assertions.
@Caideyipi Caideyipi changed the title test: add Pipe and metadata recovery integration tests test: add Pipe, metadata, and LOAD TsFile recovery integration tests Oct 30, 2026
Cover deletion isolation in a shared TsFile, rewrite survival through cross-space compaction and restart, and stable attribute-filter deletion scope after attribute updates. Add test-only helpers to assert the physical file preconditions.
@Caideyipi Caideyipi changed the title test: add Pipe, metadata, and LOAD TsFile recovery integration tests test: add Pipe, metadata, LOAD, and deletion recovery integration tests Nov 2, 2026
Cover mixed sealed-file and multi-segment WAL recovery, partial and rejected inserts, cross-Region batch failure and retry, cached Session routes after leader loss, and write admission rollback with post-recovery memory release.
@Caideyipi Caideyipi changed the title test: add Pipe, metadata, LOAD, and deletion recovery integration tests test: add Pipe, metadata, LOAD, deletion, and write recovery ITs Nov 2, 2026
IoTConsensus applies writes locally before asynchronously replicating them. Wait for the complete initial row at every endpoint before injecting metadata faults, preserving the victim schema cache precondition and all recovery assertions.
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

CI failure investigation for job 125682983290, run at f680cfa4de6abd506d80c2d02439d3b58042a074.

The reported failures come from test preconditions. Commit 8fdf070a2cc12653d2f1c3d7da4babe22c40f787 fixes them:

  • Wait for the initial row at every DataNode before injecting the live-node metadata fault. IoTConsensus replication is asynchronous; the original immediate assertion raced replication.
  • Check that existing tables retain their creation-time TTL after a database TTL change, then explicitly change the table TTL. The test still verifies expired-read filtering, the overridden table, and recovered-node write rejection after a missed broadcast.
  • Keep the auto-sync recovery reproducer with an explicit skip reason and reference to Support auto-sync of tree-to-table views: detect new measurements and add the matching columns #18874 and IoTDBTreeToTableViewAutoSyncIT. auto_sync_schema is currently absent from TABLE_ALLOWED_PROPERTIES, and the existing auto-sync cases are also ignored. This recovery scenario remains a coverage gap until that feature is enabled.

Validation on Apache IoTDB with JDK 17 and TableClusterIT:

  • Spotless, git diff --check, and the IT selection/shard check passed.
  • Required IoTDB dependencies were built from clean source. The P0 recovery class passed all 9 tests; a focused rerun of the changed P0 method also passed. The P0 code is identical in the final commit.
  • The final TTL method passed at 8fdf070a2cc12653d2f1c3d7da4babe22c40f787 (1 test, 0 failures/errors/skips; Maven exit 0). The auto-sync method is intentionally ignored; it was skipped in the earlier class run. Other PR suites were not rerun in this investigation.

An additional potential kernel issue surfaced while testing ALTER TABLE inherited SET PROPERTIES ttl=DEFAULT after changing the database TTL from seven days to one day: SHOW TABLES reported one day, while a query still returned two-day-old data. That reset/default propagation path needs separate investigation; production code is unchanged.

Reviewed PR head: 339f1074518b0a611f86f9a094250b62a8e86942 (after the original failed run). Final local commit and confirmed source-branch/PR head: 8fdf070a2cc12653d2f1c3d7da4babe22c40f787. The update is pushed. The PR remains a draft pending an independent review of the complete current diff.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review of the P1 SessionPool capacity recovery addition, reviewed at 339f1074518b0a611f86f9a094250b62a8e86942 against 24fd30e1ac6b43a32aaeca2066034b1ae81fbc44.

Two actionable findings were fixed in SessionPoolIT.java:

  • The failed-write path did not cover an interrupted capacity wait in getSession(). The existing scenario now holds the sole session, observes a real wait on the pool monitor, interrupts the borrower, returns the held session, and checks the preserved interrupt flag, worker completion, reuse of the same Session, exact data, capacity counts, and cleanup. It reuses the same pool and cluster.
  • The 1-second connection setting also limits RPC reads. A workstation run timed out on the first healthy write while initial schema/DataRegion/TsFile creation was still in progress. The RPC and borrow deadlines are now 30 seconds; write retries remain disabled.

Validation: Spotless, whitespace checks, and IT source selection passed. A local Windows/JDK 17 clean 37-module reactor built all prerequisites but reproduced the 1-second timeout error. A subsequent Maven attempt stopped before the IT because the shared TsFile cache contained a corrupt JAR. The final test source was independently compiled and run through a focused JUnit method request with the same Cluster1 properties, source-built IoTDB jars/node templates, and isolated valid copies of the packaged external dependencies: 1 test, 0 failures, 0 errors, 0 skipped; exit 0. The log confirms the InterruptedException branch was executed. This final result is a workstation JUnit result; it is not a passing Maven reactor or CI result. This round's remote preparation attempts did not execute the IT, and CI for the final head remains unchecked.

The concurrently published metadata-test commit 8fdf070a2cc12653d2f1c3d7da4babe22c40f787 was preserved by rebasing the unpublished fix. The P1 file is identical to tested snapshot 8b35204a1e9770ddbd02da240902f3ec4f2514bf. Final local, source branch, and PR heads were confirmed equal at 986e23a1d46a595896ed8168ba5cdf35eb3475a6. The PR remains draft pending a fresh independent P1 re-review.

@sonarqubecloud

sonarqubecloud Bot commented Nov 3, 2026

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@Caideyipi
Caideyipi marked this pull request as ready for review November 3, 2026 08:13
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent P1 re-review of the SessionPool failed-write and interrupted-borrow capacity recovery coverage. Reviewed 986e23a1d46a595896ed8168ba5cdf35eb3475a6 against base 24fd30e1ac6b43a32aaeca2066034b1ae81fbc44; exactly one independent round was performed.

No actionable issue was found in the complete P1 target diff or its necessary SessionPool interactions. The test verifies an actual capacity wait on the same pool monitor before interruption, return and reuse of the held Session, the preserved interrupt flag, propagated worker failures, exact data excluding the rejected write, capacity counts, bounded worker cleanup, and node/result cleanup. Other P0/P1 scenarios were outside this review's scope. No source or test changes were needed.

Validation:

  • This round: git diff --check passed; no new build or test run was needed because the reviewed code was unchanged.
  • Local workstation: the previous Windows/JDK 17 Cluster1 focused JUnit run covers the unchanged current SessionPoolIT source: 1 test, 0 failures, 0 errors, 0 skipped; exit 0. That is a focused JUnit result, not a passing final Maven reactor/Failsafe result. The attempted final Maven run was blocked before the test by a corrupt shared TsFile JAR.
  • Remote execution: the earlier 37-module reactor and 1/0/0/0 IT result covered 339f1074518b0a611f86f9a094250b62a8e86942, not the reviewed head. This round did not run remote verification.
  • CI at the reviewed head: the latest complete check-runs query returned 42 checks: 20 successful, 1 failed (SonarCloud Code Analysis), 1 skipped, 19 in progress, and 1 queued. No final CI pass is claimed; the SonarCloud failure was not investigated outside this P1 scope.

Final local, source-branch, and PR heads were confirmed equal at 986e23a1d46a595896ed8168ba5cdf35eb3475a6. The candidate worktree remains clean. After confirming that the head had not changed, the PR was marked ready for review and its non-draft status was verified. This approval is limited to the P1 scope above.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Scoped independent re-review of commit 8fdf070a2cc12653d2f1c3d7da4babe22c40f787 against parent 339f1074518b0a611f86f9a094250b62a8e86942, limited to that test repair and necessary context.

[P2] Fixed a remaining replication race in the TTL recovery test. Successful writes and reads through different SQL coordinators do not establish that every actual IoTConsensus data replica has caught up. Stopping a DataNode immediately after fixture setup, or asserting final rows immediately after recovery writes, could mistake replication lag for a TTL/recovery failure. Added the existing MetadataFaultTestUtils.awaitDataReplication(database) barrier at both boundaries in IoTDBTableMetadataRecoveryIT.java. Exact TTL, row and expired-write rejection assertions are retained. No new test methods or production changes.

Other scoped conclusions:

  • The P0 bounded wait retries complete row/header assertions before fault injection and is appropriate for its live-node scenario.
  • Database TTL supplies the creation-time table default; altering it does not update existing tables. Explicitly altering the table TTL correctly exercises missed-broadcast recovery while preserving the overridden table's TTL.
  • Keeping the auto-sync reproducer ignored has evidence: auto_sync_schema is absent from the current allowed-property validation, and the corresponding existing tests were already ignored in Support auto-sync of tree-to-table views: detect new measurements and add the matching columns #18874. This remains a coverage gap, not a verified recovery scenario. The previously recorded ttl=DEFAULT catalog/cache discrepancy remains a separate follow-up and was not changed here.

Final verification: Linux/JDK 17, TableClusterIT, 3 ConfigNodes/3 DataNodes; clean-source build of the required 37-module IoTDB reactor with the build cache disabled. Failsafe actually executed IoTDBTableMetadataRecoveryIT#testTableTtlChangeAfterMissedBroadcast: 1 test, 0 failures, 0 errors, 0 skipped, Maven exit 0. Spotless and diff checks passed. Earlier preparation/cache-enabled attempts are not counted as verification. The unchanged P0 case relies on its previously recorded focused result. Auto-sync remains ignored and was not executed in this run; other PR suites and remote CI are not certified by this review.

Repair commit, final local SHA, confirmed source-branch SHA and confirmed PR head: 906980acc9064627a01ac64dfb052c488086ea5b. The later 986e23a SessionPool change was preserved. Exactly one review round was performed; because it found a test-logic issue, a fresh scoped re-review is still required. The existing open/non-draft state was left unchanged as requested. This is not approval of the whole PR.

@Caideyipi

Caideyipi commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator Author

Scoped independent re-review of commit 906980acc9064627a01ac64dfb052c488086ea5b against parent 339f1074518b0a611f86f9a094250b62a8e86942, limited to the two table metadata recovery IT files and necessary context. No in-scope issue was found and no code changes were needed.

The P0 test's bounded pre-fault wait polls all three DataNode SQL endpoints. The TTL test's two replication barriers enumerate each actual DataRegion replica and wait for IoTConsensus synchronization, covering both the pre-shutdown fixture and final post-recovery writes. The TTL assertions correctly distinguish the creation-time database TTL copied by existing tables from the explicit table TTL change.

The auto-sync reproducer remains ignored because current property validation rejects auto_sync_schema; it is a documented coverage gap, not passing test coverage.

Verification: no tests were rerun in this round. The retained clean Linux/JDK 17 TableClusterIT, 3CN/3DN record shows the TTL method ran with 1 test, 0 failures, 0 errors, 0 skipped and Maven exit 0. P0 was unchanged and not rerun; its prior focused pass is historical evidence. Auto-sync remains skipped.

Reviewed SHA: 906980acc9064627a01ac64dfb052c488086ea5b. Final local SHA: 906980acc9064627a01ac64dfb052c488086ea5b. Source branch and PR head confirmed at the same SHA. PR Draft status was left unchanged.

@Caideyipi
Caideyipi marked this pull request as draft November 4, 2026 05:15
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Metadata IT selection and target validation

The single local re-review covered the requested metadata IT diff at 8d7dbf3ca6e94b1bfbebe9002ee7cf20ec5947d6 plus the working changes, including earlier repairs. The subsequent requested rerun and transfer validation found the following concrete target issues and fixed them:

  • P1 — integration-test/pom.xml:99: direct ASM test scope excluded cglib's ASM dependency from ordinary child-node templates, causing NoClassDefFoundError: org/objectweb/asm/Type. The instrumented agent supplied ASM itself and hid this startup issue. ASM now has runtime scope in the IT module.
  • P2 — MetadataFaultController.java:59: the source fixture's environment-variable APIs do not exist on this target branch, causing IT test compilation to fail. The controller now uses the existing MppJVMConfig JVM options API and restores the prior options. No production source or production test hook was added.

The transfer now retains only testOldRemoteFetchCannotRefillStaleAttributes and the strengthened testDropTableModFileProgress, with their test-only fault scaffolding. Four failing propagation scenarios and the complete parameterized TagAlter addition remain outside the PR update. The SchemaRegion migration method passed the source rerun but failed on the target at IoTDBTableTagSchemaMigrationIT.java:222, where DELETE DEVICES returned a null TDataNodeLocation error; its entire new class has been withdrawn from the PR. These failures were preserved without assigning an unproven product/IT cause or weakening assertions.

Validation on Linux / OpenJDK 17 / Maven 3.8.8:

  • TableClusterIT executed 3 methods: both retained methods passed; the now-excluded migration method errored. The broader Maven command exited 1.
  • TableSimpleIT executed DROP/MOD: 1 passed, 0 failures, 0 errors, 0 skipped, Maven exit 0. The final retained selection therefore has 3 passing configurations.
  • Required IoTDB modules were initially rebuilt from clean source in an isolated reactor/repository. Focused follow-ups reused the same source-built dependencies under recorded provenance; this was not a fresh uncached full build. All five final retained paths match the executed source hashes.
  • Scoped Spotless, source-selection and whitespace checks passed. Existing module Checkstyle is ineffective because its exclusion expression has an empty alternative and was left unchanged. The full IT suite and Chinese locale were not run.

Reviewed source SHA: 8d7dbf3ca6e94b1bfbebe9002ee7cf20ec5947d6 plus the working diff. Final local candidate SHA and confirmed remote branch/PR head: 325dd94bde3ea8aaf9cc47b28484df26a1f9aaa3. Publication and head confirmation are complete.

The PR remains Draft. A fresh session must independently review the full current PR diff and relevant context at the latest remote head; no second complete review was performed in this session.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review (metadata IT scope only)

Reviewed the requested three-commit net change from 906980acc9064627a01ac64dfb052c488086ea5b through 325dd94bde3ea8aaf9cc47b28484df26a1f9aaa3, plus the review fix at a4208745f6fb41dad91886f6efbf523cd39b6633, limited to the five metadata IT files. This result does not indicate that the full PR passed.

  • Finding fixed: After releasing the held old fetch, the follow-up SQL assertion could fetch and refill a missing cache entry, masking an incorrect cache state. The test now reads the target node's attribute cache directly first and requires d1.changing to be present with value current before issuing SQL assertions.
  • Verification: On Linux/OpenJDK 17, a clean TableClusterIT reactor run for IoTDBTableMetadataPropagationIT#testOldRemoteFetchCannotRefillStaleAttributes completed with BUILD SUCCESS; the report recorded 1 test, 0 failures, 0 errors, and 0 skipped. Scoped compile, Spotless, and whitespace checks also passed. A separate attempt ended before tests during Maven dependency resolution (Connection reset retrieving extra-enforcer-rules:1.12.1). The previously reported broader cluster run still exits 1 on the withdrawn SchemaRegion migration test; it is outside this review selection. Full IT, Chinese-locale build, and hosted CI were not verified.

The source branch and PR HEAD both match a4208745f6fb41dad91886f6efbf523cd39b6633; the PR remains Draft.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Scoped independent re-review of the complete net diff from 906980acc9064627a01ac64dfb052c488086ea5b through a4208745f6fb41dad91886f6efbf523cd39b6633, including earlier fixes, strictly limited to:

  • integration-test/pom.xml
  • integration-test/src/test/java/org/apache/iotdb/relational/it/schema/IoTDBDropTableModProgressIT.java
  • integration-test/src/test/java/org/apache/iotdb/relational/it/schema/IoTDBTableMetadataPropagationIT.java
  • integration-test/src/test/java/org/apache/iotdb/relational/it/schema/fault/MetadataFaultAgent.java
  • integration-test/src/test/java/org/apache/iotdb/relational/it/schema/fault/MetadataFaultController.java

No actionable defect was confirmed in this scoped round. Necessary production context was read without modification. Other PR changes and withdrawn scenarios were excluded.

A candidate concern about the post-recreation LAST warm-up was investigated and rejected. TableDeviceSchemaCache.updateLastCacheIfExists deliberately updates only existing cache entries; querying an empty table does not initialize LAST entries for a device that does not yet exist. Removing the original post-insert warm-up incorrectly required the first write to populate a cold cache. That experimental candidate ran one test with one assertion failure (CachedDeviceNumber: 0) and was discarded. The file was restored exactly to the reviewed head; all original assertions remain.

Validation:

  • The unchanged reviewed head passed the focused Windows/JDK 17 TableSimpleIT run of IoTDBDropTableModProgressIT#testDropTableModFileProgress: 1 test, 0 failures, 0 errors, 0 skips; Maven exit 0, confirmed from Failsafe XML. IT outputs were cleaned, and required reactor dependencies came from this session's matching-source build with build-cache reuse disabled.
  • Earlier remote preparation attempts stopped before Maven because of source-fetch TLS errors; no tests ran in those attempts. This round did not rerun the cluster profile. The previously reported Linux/JDK 17 focused cluster pass remains prior verification evidence.
  • IT compilation, scoped Spotless, and whitespace checks passed. All newly added tests had already been run locally in the supplied verification background. Full IT coverage, the Chinese locale, and hosted CI remain unverified. The existing ineffective IT Checkstyle exclusion was left unchanged and is not counted as validation.

No code changes, commits, or pushes were retained. Final local HEAD, confirmed apache/iotdb:test/pipe-p0-it-master HEAD, and PR HEAD are all a4208745f6fb41dad91886f6efbf523cd39b6633; no concurrent head update was observed. The PR remains Draft as requested. These conclusions apply only to the five-file scope above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant