Repository navigation
HDDS-16600. Reuse build ozone-repo on integration CI and harden RocksDB download - #11473
Closed
jojochuang wants to merge 8 commits into
Closed
jojochuang wants to merge 8 commits into
jojochuang wants to merge 8 commits into
Conversation
…DB download Wire needs-ozone-repo and OZONE_REPO_CACHED into integration matrix jobs, skip full-reactor clean when the build job repo artifact is restored, and exclude integration-test modules from the reactor like unit.sh to avoid MDEP-187 unpack failures. Harden hdds-rocks-native RocksDB source wget retries. Add a Develocity CI summary helper script. Generated-by: Cursor (Auto) Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I3934df0500227bb4e0466646c11c800e762f86aa
jojochuang
changed the title
jojochuang
force-pushed
the
HDDS-16749
branch
from
October 11, 2026 05:17
8567b40 to
3568e7f
Compare
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: Iafd5aa3bb2aba3eac59e6118a96ae380e63b9dd2
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The reactor selection skips integration suites, while the retry and Develocity configurations contain ineffective or inaccurate settings.
4 open findings
What changed in this PR
Optimizes integration CI by reusing build artifacts and hardening dependency downloads.
Changes:
- Reuses cached Ozone Maven artifacts in integration jobs.
- Adjusts Maven reactor and cleanup behavior.
- Adds RocksDB retries and a Develocity summary utility.
| File | Description |
|---|---|
.github/workflows/check.yml |
Exposes repository-cache state to checks. |
.github/workflows/ci.yml |
Enables artifact reuse for integration jobs. |
hadoop-ozone/dev-support/checks/junit.sh |
Adjusts cached integration execution. |
hadoop-hdds/rocks-native/pom.xml |
Configures download resilience. |
dev-support/scripts/develocity_ci_summary.py |
Adds CI build-scan reporting. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+62
to
+68
| INTEGRATION_REACTOR_PL=() | ||
| if [[ "${OZONE_REPO_CACHED}" == "true" ]] && [[ "${CHECK}" == "integration" ]]; then | ||
| args_str="$*" | ||
| if [[ "${args_str}" != *"-Ptest-filesystem"* ]] && [[ "${args_str}" != *"-Ptest-flaky"* ]] \ | ||
| && [[ "${args_str}" != *"-Dtest="* ]]; then | ||
| INTEGRATION_REACTOR_PL=(-pl '!:ozone-integration-test,!:ozone-integration-test-recon,!:ozone-integration-test-s3') | ||
| fi |
| ) -> List[Dict[str, Any]]: | ||
| params = { | ||
| "fromInstant": str(from_instant), | ||
| "reverse": "true", |
| print(f"Failed: {failed} ({100.0 * failed / total:.1f}%)" if total else "Failed: 0") | ||
| if durations: | ||
| durations.sort() | ||
| mid = durations[len(durations) // 2] |
| <url>https://github.com/facebook/rocksdb/archive/refs/tags/v${rocksdb.source.version}.tar.gz</url> | ||
| <outputFileName>rocksdb-v${rocksdb.source.version}.tar.gz</outputFileName> | ||
| <outputDirectory>${project.build.directory}/rocksdb</outputDirectory> | ||
| <retryCount>5</retryCount> |
2 of 3 tasks
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I3558f3366fec0202d8d1655d3dfcda44b5278769
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I13a27185f700fd46baba3e98693b4bfc93e660c3
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I44d2c7a10769175bd8d6ae5ea3a27a444cbdcf14
Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: Idb6c1390130e6dce6643ac6c532993783541ecee
…e wget retries Run integration splits against ozone-integration-test modules when the CI repo artifact is cached, use download-maven-plugin retries parameter, and query Develocity builds forward from the configured start instant. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I4ae2099799fcc29a72ae02e13f9cf9f6bc0583a1
…e-repo Scoped integration runs only build ozone-integration-test, so root generate-resources for libhadoop in target/native-lib was skipped and TestDomainSocketFactory failed on CI client. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: I7e0c90bd5c4d0e3578d6a343a424c5f8ccd3dcdf
Contributor
|
Please continue work in #11320 instead of opening a new PR. (Also, please don't ignore review comments.) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
clean verifywithout restoring the ozone-repo artifact from the build job, rebuilding OM/SCM paths that trigger MDEP-187.needs-ozone-repo/OZONE_REPO_CACHEDfor integration matrix legs, skipsmvn cleanwhen the repo is cached, excludes integration-test modules from the reactor (same asunit.sh), and increases RocksDB source download retries/timeouts.Approach
Align main CI integration with the pattern documented in
dev-support/ci/integration-timing-baseline.mdand already used for flaky-test-check.Test plan
omMaven total ~25m vs ~58m on recent apache/master success).dev-support/scripts/develocity_ci_summary.py)https://issues.apache.org/jira/browse/HDDS-16600
Generated-by: Cursor (Auto)