Skip to content

HDDS-16821. Speed up TestSstFileSetReader - #11469

Merged
smengcl merged 1 commit into
apache:masterfrom
F64116045:HDDS-16821
Oct 11, 2026
Merged

HDDS-16821. Speed up TestSstFileSetReader#11469
smengcl merged 1 commit into
apache:masterfrom
F64116045:HDDS-16821

Conversation

@F64116045

@F64116045 F64116045 commented Oct 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

TestSstFileSetReader creates 1,000 keys per SST file, but only ten of them are used to generate the read boundaries.
The other keys make the tests slower without adding more test cases.

This change reduces the number of keys per file to 10.
Local test time decreased from 131.9 seconds to 4.836 seconds.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16821

How was this patch tested?

Local

Before

[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 131.9 s -- in org.apache.hadoop.hdds.utils.db.TestSstFileSetReader
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[1] -- Time elapsed: 0.295 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[2] -- Time elapsed: 3.585 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[3] -- Time elapsed: 6.321 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[4] -- Time elapsed: 9.76 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[5] -- Time elapsed: 22.457 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[6] -- Time elapsed: 32.044 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[1] -- Time elapsed: 0.026 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[2] -- Time elapsed: 2.414 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[3] -- Time elapsed: 5.027 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[4] -- Time elapsed: 7.175 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[5] -- Time elapsed: 17.391 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[6] -- Time elapsed: 25.151 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testDuplicateKeyHandlingWithLatestFilePrecedence(int)[1] -- Time elapsed: 0.031 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testDuplicateKeyHandlingWithLatestFilePrecedence(int)[2] -- Time elapsed: 0.023 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testDuplicateKeyHandlingWithLatestFilePrecedence(int)[3] -- Time elapsed: 0.022 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testMinHeapWithOverlappingSstFiles(int)[1] -- Time elapsed: 0.013 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testMinHeapWithOverlappingSstFiles(int)[2] -- Time elapsed: 0.02 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testMinHeapWithOverlappingSstFiles(int)[3] -- Time elapsed: 0.031 s

After

[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.836 s -- in org.apache.hadoop.hdds.utils.db.TestSstFileSetReader
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[1] -- Time elapsed: 0.284 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[2] -- Time elapsed: 0.432 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[3] -- Time elapsed: 0.325 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[4] -- Time elapsed: 0.42 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[5] -- Time elapsed: 0.806 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStreamWithTombstone(int)[6] -- Time elapsed: 1.125 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[1] -- Time elapsed: 0.007 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[2] -- Time elapsed: 0.112 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[3] -- Time elapsed: 0.167 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[4] -- Time elapsed: 0.171 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[5] -- Time elapsed: 0.347 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testGetKeyStream(int)[6] -- Time elapsed: 0.389 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testDuplicateKeyHandlingWithLatestFilePrecedence(int)[1] -- Time elapsed: 0.021 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testDuplicateKeyHandlingWithLatestFilePrecedence(int)[2] -- Time elapsed: 0.019 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testDuplicateKeyHandlingWithLatestFilePrecedence(int)[3] -- Time elapsed: 0.02 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testMinHeapWithOverlappingSstFiles(int)[1] -- Time elapsed: 0.012 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testMinHeapWithOverlappingSstFiles(int)[2] -- Time elapsed: 0.013 s
org.apache.hadoop.hdds.utils.db.TestSstFileSetReader.testMinHeapWithOverlappingSstFiles(int)[3] -- Time elapsed: 0.02 s

CI: https://github.com/F64116045/ozone/actions/runs/38073734762

@Russole
Russole self-requested a review October 11, 2026 02:06
@F64116045
F64116045 marked this pull request as ready for review October 11, 2026 07:45
Russole
Russole approved these changes Oct 11, 2026
View reviewed changes

@Russole Russole left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me. LGTM!

@adoroszlai
adoroszlai requested a review from smengcl October 11, 2026 17:04
smengcl
smengcl approved these changes Oct 11, 2026
View reviewed changes

@smengcl smengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@smengcl
smengcl merged commit ded6c5b into apache:master Oct 11, 2026
37 of 40 checks passed
@smengcl

smengcl commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

Thanks @F64116045 for the patch. Thanks @Russole for the review.

@F64116045

F64116045 commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

thanks @Russole @smengcl !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@smengcl smengcl smengcl approved these changes
@Russole Russole Russole approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@F64116045 @smengcl @Russole