Skip to content

HDDS-16881. Fix intermittent failure in ozone-ha/test-haproxy-s3g.sh - #11475

Merged
smengcl merged 1 commit into
apache:masterfrom
smengcl:HDDS-16881-flaky
Oct 12, 2026
Merged

smengcl merged 1 commit into
apache:masterfrom
smengcl:HDDS-16881-flaky

Conversation

@smengcl

@smengcl smengcl commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Generated-by: Claude Code (Opus 5.5)

What changes were proposed in this pull request?

ozone-ha/test-haproxy-s3g.sh intermittently fails in acceptance (HA-unsecure): the first awss3.robot tests fail with 1 != 0 because the aws cli gets 503 from HAProxy.

HDDS-16360 made HAProxy gate S3 traffic on /health/ready and added wait_for_s3g_ready so tests start only after s3g1 is ready. Its guard is docker-compose config --services | grep -q "^s3g1$", and testlib.sh runs with pipefail. grep -q exits on the first match, so docker-compose can get SIGPIPE writing the remaining service names, the pipeline returns 141, and the wait is skipped. Passing runs log Waiting for S3 Gateway (s3g1) to report ready...; the failing runs do not.

This patch replaces grep -q with grep ... > /dev/null, so grep reads the whole output.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16881

How was this patch tested?

  • Reproduced the mechanism with a line by line writer under set -o pipefail: grep -q returns 141 when lines follow the match, grep > /dev/null returns 0 (and still 1 when the service is absent).
  • hadoop-ozone/dev-support/checks/bats.sh passes.
  • CI run on this branch: all jobs passed, and acceptance (HA-unsecure) logs Waiting for S3 Gateway (s3g1) to report ready... before the HAProxy tests. One green run cannot prove the race is gone, but the fix removes it by construction: grep no longer exits before docker-compose finishes writing.

🤖 Generated with Claude Code

wait_for_s3g_ready checks for the s3g1 service with
`docker-compose config --services | grep -q "^s3g1$"` under pipefail.
grep -q exits on the first match, so docker-compose can get SIGPIPE while
writing the remaining service names and the pipeline fails. The wait is then
skipped and S3 tests start while all HAProxy backends are still DOWN (503).

Let grep consume the whole output instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 11, 2026 06:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request. Check if the Files changed in this pull request are included in default exclusions.

@cchung100m cchung100m left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks to @smengcl

@smengcl
smengcl marked this pull request as ready for review October 11, 2026 07:39
@Russole
Russole self-requested a review October 11, 2026 22:41

@chungen0126 chungen0126 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

@smengcl
smengcl merged commit 025266b into apache:master Oct 12, 2026
58 checks passed
@smengcl
smengcl deleted the HDDS-16881-flaky branch October 12, 2026 06:47
@smengcl

smengcl commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @cchung100m @chungen0126 for the reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants