Repository navigation
GH-55272: [C++][Parquet][Python] Default to V2 data pages #55453
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
c4f6b74
2191157
a94ed27
be961ea
fd23951
c6c6475
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -2024,7 +2024,8 @@ Status TypedColumnWriterImpl<ParquetType>::WriteArrowDictionary( | |||||||||||||||
| ::arrow::compute::Unique(*chunk_indices, &exec_ctx)); | ||||||||||||||||
|
|
||||||||||||||||
| // On first run, we might be able to re-use the existing dictionary | ||||||||||||||||
| if (referenced_indices.length() == dictionary->length()) { | ||||||||||||||||
| if (chunk_indices->null_count() == 0 && | ||||||||||||||||
|
Awesome06 marked this conversation as resolved.
Resolved
|
||||||||||||||||
| referenced_indices.length() == dictionary->length()) { | ||||||||||||||||
|
Comment on lines
+2027
to
+2028
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Would it make sense to refine the shortcut condition? Something like:
Suggested change
|
||||||||||||||||
| referenced_dictionary = dictionary; | ||||||||||||||||
| } else { | ||||||||||||||||
| PARQUET_ASSIGN_OR_THROW( | ||||||||||||||||
|
|
@@ -2064,12 +2065,12 @@ Status TypedColumnWriterImpl<ParquetType>::WriteArrowDictionary( | |||||||||||||||
| AddIfNotNull(rep_levels, offset)); | ||||||||||||||||
| std::shared_ptr<Array> writeable_indices = | ||||||||||||||||
| indices->Slice(value_offset, batch_num_spaced_values); | ||||||||||||||||
| if (page_statistics_ || bloom_filter_writer_) { | ||||||||||||||||
| update_stats(/*num_chunk_levels=*/batch_size, writeable_indices); | ||||||||||||||||
| } | ||||||||||||||||
| PARQUET_ASSIGN_OR_THROW( | ||||||||||||||||
| writeable_indices, | ||||||||||||||||
| MaybeReplaceValidity(writeable_indices, null_count, ctx->memory_pool)); | ||||||||||||||||
| if (page_statistics_ || bloom_filter_writer_) { | ||||||||||||||||
| update_stats(/*num_chunk_levels=*/batch_size, writeable_indices); | ||||||||||||||||
| } | ||||||||||||||||
|
Awesome06 marked this conversation as resolved.
Resolved
|
||||||||||||||||
| dict_encoder->PutIndices(*writeable_indices); | ||||||||||||||||
| // Update unencoded byte array data size to size statistics | ||||||||||||||||
| UpdateUnencodedDataBytes(); | ||||||||||||||||
|
|
||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -148,8 +148,15 @@ class TestSerialize : public PrimitiveTypedTest<TestType> { | |
| const int64_t total_compressed_size = rg_metadata->total_compressed_size(); | ||
| if (expected_codec_type == Compression::UNCOMPRESSED) { | ||
| ASSERT_EQ(total_byte_size, total_compressed_size); | ||
| } else { | ||
| } else if (writer_properties->data_page_version() == ParquetDataPageVersion::V1) { | ||
| ASSERT_NE(total_byte_size, total_compressed_size); | ||
| } else { | ||
| // In Parquet V2, selective compression may keep small or incompressible data | ||
| // pages uncompressed (total_compressed_size == total_byte_size), while | ||
| // compressed dictionary pages or pages with savings may be larger or smaller | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. They shouldn't be "larger or smaller", they should be smaller, precisely because of selective compression. |
||
| // than uncompressed sizes. | ||
| ASSERT_GT(total_byte_size, 0); | ||
| ASSERT_GT(total_compressed_size, 0); | ||
| } | ||
|
|
||
| int64_t total_column_byte_size = 0; | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -840,9 +840,9 @@ def _sanitize_table(table, new_schema, flavor): | |||||||||
| Certain encodings are only compatible with certain data types. | ||||||||||
| Please refer to the encodings section of `Reading and writing Parquet | ||||||||||
| files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_. | ||||||||||
| data_page_version : {"1.0", "2.0"}, default "1.0" | ||||||||||
| data_page_version : {"1.0", "2.0"}, default "2.0" | ||||||||||
| The serialized Parquet data page format version to write, defaults to | ||||||||||
| 1.0. This does not impact the file schema logical types and Arrow to | ||||||||||
| 2.0. This does not impact the file schema logical types and Arrow to | ||||||||||
| Parquet type casting behavior; for that use the "version" option. | ||||||||||
| use_compliant_nested_type : bool, default True | ||||||||||
| Whether to write compliant Parquet nested type (lists) as defined | ||||||||||
|
|
@@ -999,8 +999,8 @@ def _sanitize_table(table, new_schema, flavor): | |||||||||
| Use a value from 0.0 to 1.0. For example, 0.1 requires at least 10% savings. | ||||||||||
| Savings are measured on encoded values, excluding repetition and definition | ||||||||||
| levels. The default is 0.01, so compression must save at least 1%. | ||||||||||
| Values above this default require ``data_page_version="2.0"``. Using | ||||||||||
| a larger value with V1 or a number outside the allowed range raises ``ValueError``. | ||||||||||
| Using a larger value with V1 or a number outside the allowed range | ||||||||||
| raises ``ValueError``. | ||||||||||
|
Comment on lines
+1002
to
+1003
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
| """ | ||||||||||
|
|
||||||||||
| _parquet_writer_example_doc = """\ | ||||||||||
|
|
@@ -1087,7 +1087,7 @@ def __init__(self, where, schema, filesystem=None, | |||||||||
| use_byte_stream_split=False, | ||||||||||
| column_encoding=None, | ||||||||||
| writer_engine_version=None, | ||||||||||
| data_page_version='1.0', | ||||||||||
| data_page_version='2.0', | ||||||||||
| use_compliant_nested_type=True, | ||||||||||
| encryption_properties=None, | ||||||||||
| write_batch_size=None, | ||||||||||
|
|
@@ -2050,7 +2050,7 @@ def write_table(table, where, row_group_size=None, version='2.6', | |||||||||
| compression_level=None, | ||||||||||
| use_byte_stream_split=False, | ||||||||||
| column_encoding=None, | ||||||||||
| data_page_version='1.0', | ||||||||||
| data_page_version='2.0', | ||||||||||
| use_compliant_nested_type=True, | ||||||||||
| encryption_properties=None, | ||||||||||
| write_batch_size=None, | ||||||||||
|
|
||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's update this comment? This isn't about a first run but whether all dict values are referenced.