Skip to content

GH-55272: [C++][Parquet][Python] Default to V2 data pages - #55453

Open
Awesome06 wants to merge 6 commits into
apache:mainfrom
Awesome06:gh-55272-parquet-default-v2-pages
Open

Awesome06 wants to merge 6 commits into
apache:mainfrom
Awesome06:gh-55272-parquet-default-v2-pages

Conversation

@Awesome06

@Awesome06 Awesome06 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Parquet V2 data pages are widely supported across the ecosystem. They provide explicit page-level null counts (num_nulls) in page headers and enable per-page selective compression (min_compression_savings), allowing pages that do not benefit from compression to remain uncompressed.

Benchmarks across TPC-H, SSB, and TPC-DS (discussed on GH-55272) showed that V2 data pages yield comparable file sizes (0.0% to 1.2% delta depending on schema sparsity) while improving read throughput by 12% to 35% on scanned columns.

What changes are included in this PR?

  • C++ Core (properties.h):
    • Changed the default data_page_version in WriterProperties::Builder from ParquetDataPageVersion::V1 to ParquetDataPageVersion::V2.
    • Updated docstrings to indicate V2 default.
  • C++ Tests (properties_test.cc, column_writer_test.cc):
    • Updated TestWriterProperties.Basics to expect ParquetDataPageVersion::V2.
    • Updated MinCompressionSavingsRequiresDataPageV2 to verify default V2 allows min_compression_savings without throwing, while explicitly configured V1 raises an exception.
    • In WriteLargeDictEncodedPage, updated expected page type to PageType::DATA_PAGE_V2.
  • C++ Documentation (parquet.rst):
    • Documented that data_page_version defaults to V2, with V1 retained for older reader compatibility.
  • Python API (core.py, _dataset_parquet.pyx, _parquet.pyx):
    • Changed default data_page_version from '1.0' to '2.0' in ParquetWriter, write_table, and dataset write options.
    • Adjusted _parquet.pyx validation so min_compression_savings > 0.01 only rejects explicit data_page_version="1.0".
  • Python Tests (test_parquet_writer.py):
    • Updated test_min_compression_savings_rejects_v1 to explicitly specify data_page_version="1.0".
    • Added test_min_compression_savings_default_v2 to verify default write paths accept min_compression_savings without error.

Are these changes tested?

Yes. Updated existing C++ properties and writer unit tests, and added Python test coverage for default V2 writer behavior.

Are there any user-facing changes?

Yes. Files written with default writer properties in C++ and PyArrow will now use Parquet Data Page V2 instead of V1. Users requiring V1 for legacy compatibility can pass data_page_version="1.0" (Python) or .data_page_version(ParquetDataPageVersion::V1) (C++).

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

Switch the default Parquet data page version from V1 to V2 across C++ and Python (PyArrow).

V2 data pages enable per-page compression selectivity (via min_compression_savings) and include explicit page-level null count metadata (num_nulls) in the page header.

Closes apache#55272.
Copilot AI balanced review requested due to automatic review settings November 9, 2026 12:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #55272 has been automatically assigned in GitHub to PR creator.

… masking and synchronize V2 page defaults

Reorder MaybeReplaceValidity before update_stats in WriteArrowDictionary so writeable_indices reflects parent nulls prior to computing page stats, and guard dictionary reuse with chunk_indices->null_count() == 0.

Update TestSerialize for BooleanType in Parquet C++ to account for V2 selective compression keeping incompressible 7-byte boolean pages uncompressed.

Update test_parquet_metadata_api expected boolean encodings to {'RLE'} for V2 defaults.

Synchronize parquet.rst Sphinx doctest expected row group byte size for V2 headers.
Copilot AI balanced review requested due to automatic review settings November 9, 2026 14:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…ion in TestSerialize and synchronize Sphinx doctest sizes

Update TestSerialize in Parquet C++ to allow selective compression leaving small or incompressible pages uncompressed under V2 page defaults. Wrap comment to adhere to 90-char limit for clang-format.

Synchronize column chunk total_compressed_size and total_uncompressed_size in parquet.rst Sphinx doctest for V2 page defaults.
Copilot AI balanced review requested due to automatic review settings November 9, 2026 17:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…V2 data pages

Update TestSerialize in Parquet C++ to check positivity of compressed and
uncompressed sizes under V2 data pages rather than asserting strict ordering.
Under V2, selective compression can leave data pages uncompressed, while
unconditional compression of dictionary pages can expand small payloads,
making total_compressed_size less than, equal to, or greater than
total_byte_size.
Copilot AI balanced review requested due to automatic review settings November 9, 2026 19:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@adamreeve adamreeve left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @Awesome06! I have a couple of questions

Comment thread cpp/src/parquet/column_writer.cc Resolved
Comment thread cpp/src/parquet/column_writer.cc Resolved
Copilot AI balanced review requested due to automatic review settings November 11, 2026 02:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…atistics with nulls

Add TestArrowWriteDictionary.StatisticsWithNullsMatchingDictLength to verify unreferenced dictionary values are not included in statistics when null count causes unique count to equal dictionary length.

Add TestArrowWriteDictionary.NestedDictionaryParentNullStatistics to verify that nested dictionary columns under nullable parents accurately reflect ancestor nulls in page and chunk statistics across both V1 and V2 data pages.
@Awesome06
Awesome06 force-pushed the gh-55272-parquet-default-v2-pages branch from d61267d to fd23951 Compare November 11, 2026 02:45
Copilot AI balanced review requested due to automatic review settings November 11, 2026 02:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 11, 2026 04:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@pitrou

pitrou commented Nov 16, 2026

Copy link
Copy Markdown
Member

@Awesome06 Please rewrite your PR description as a human writing for humans.

@Awesome06

Copy link
Copy Markdown
Contributor Author

@pitrou Kindly let me know the changes you expect in the PR description. As I'm not quite sure what to reduce or reword in the current description

@pitrou

pitrou commented Nov 16, 2026

Copy link
Copy Markdown
Member

@Awesome06 I think my demand was quite self-descriptive. If you are a human, write the PR description yourself instead of pasting AI output.

If you are not a human, please say so and this PR will be closed.

@@ -2024,7 +2024,8 @@ Status TypedColumnWriterImpl<ParquetType>::WriteArrowDictionary(
::arrow::compute::Unique(*chunk_indices, &exec_ctx));

// On first run, we might be able to re-use the existing dictionary

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's update this comment? This isn't about a first run but whether all dict values are referenced.

Suggested change
// On first run, we might be able to re-use the existing dictionary
// Use the original dictionary if we can detect that all indices are referenced,
// otherwise call Take to find referenced dictionary values.

Comment on lines +2027 to +2028
if (chunk_indices->null_count() == 0 &&
referenced_indices.length() == dictionary->length()) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to refine the shortcut condition? Something like:

Suggested change
if (chunk_indices->null_count() == 0 &&
referenced_indices.length() == dictionary->length()) {
const bool all_indices_referenced =
(referenced_indices.null_count() == 0)
? referenced_indices.length() == dictionary->length()
: referenced_indices.length() == dictionary->length() + 1;
if (all_indices_referenced) {

} else {
// In Parquet V2, selective compression may keep small or incompressible data
// pages uncompressed (total_compressed_size == total_byte_size), while
// compressed dictionary pages or pages with savings may be larger or smaller

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They shouldn't be "larger or smaller", they should be smaller, precisely because of selective compression.

Comment on lines +1002 to +1003
Using a larger value with V1 or a number outside the allowed range
raises ``ValueError``.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Using a larger value with V1 or a number outside the allowed range
raises ``ValueError``.
Using a larger value with V1 data pages or a number outside
the allowed range raises ``ValueError``.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++][Parquet] Default to V2 data pages

6 participants