Skip to content

GH-55571: [C++][Parquet] Honor configured column encoding on dictionary fallback - #55572

Closed
ryanvickr wants to merge 1 commit into
apache:mainfrom
ryanvickr:gh-55571-configurable-dict-fallback-encoding
Closed

ryanvickr wants to merge 1 commit into
apache:mainfrom
ryanvickr:gh-55571-configurable-dict-fallback-encoding

Conversation

@ryanvickr

Copy link
Copy Markdown

Rationale for this change

When dictionary encoding is enabled on a column (enable_dictionary(), the default), WriterProperties::Builder::encoding(...) rejects PLAIN_DICTIONARY and RLE_DICTIONARY with "Can't use dictionary encoding as fallback encoding", indicating that the configured encoding is intended as the fallback encoding. However:

  1. ColumnWriter::Make overwrote encoding with properties->dictionary_index_encoding() when use_dictionary was true without preserving the resolved non-dictionary encoding, and TypedColumnWriterImpl::FallbackToPlainEncoding() hardcoded Encoding::PLAIN (see also [Parquet] Make default fallback encoding choice smarter #40636). As a result, configuring e.g. DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY, DELTA_BINARY_PACKED, or BYTE_STREAM_SPLIT on a dictionary-enabled column was ignored whenever the column overflowed dictionary_pagesize_limit or fell back on varying arrow::DictionaryArray chunks.
  2. Incompatible (physical_type, encoding) pairs on dictionary-enabled columns were not validated upfront on ColumnWriter creation, only failing at runtime if the column happened to overflow its dictionary.
  3. CheckDictionarySizeLimit() was called after every batch in WriteBatchInternal, WriteBatchSpacedInternal, and TypedColumnWriterImpl<ByteArrayType>::WriteArrowDense even when check_page == false (mid-record on a repeated column when pages_change_on_record_boundaries() is active), which could flush buffered dictionary pages mid-record.

What changes are included in this PR?

  • Preserve fallback_encoding in ColumnWriter::Make before switching encoding to properties->dictionary_index_encoding(), and use fallback_encoding_ in TypedColumnWriterImpl::FallbackToPlainEncoding().
  • Validate fallback_encoding upfront in TypedColumnWriterImpl when use_dictionary is true so unsupported (physical_type, fallback_encoding) combinations throw ParquetException immediately upon ColumnWriter creation.
  • Gate CheckDictionarySizeLimit() on check_page || descr_->max_repetition_level() == 0 || rep_levels == nullptr in WriteBatchInternal, WriteBatchSpacedInternal, and TypedColumnWriterImpl<ByteArrayType>::WriteArrowDense so dictionary fallback on repeated columns only flushes at record boundaries when pages_change_on_record_boundaries() is true.
  • Update WriterProperties::Builder::encoding(...) docstrings in cpp/src/parquet/properties.h.

Are these changes tested?

Yes, added unit tests in cpp/src/parquet/column_writer_test.cc and cpp/src/parquet/properties_test.cc covering:

  • PARQUET_1_0 (PLAIN_DICTIONARY -> PLAIN) and PARQUET_2_6 (RLE_DICTIONARY -> configured fallback encoding) dictionary page and data page encodings.
  • Dictionary fallback to DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY, DELTA_BINARY_PACKED, and BYTE_STREAM_SPLIT across WriteTable (including varying DictionaryArray chunks and SizeStatistics), WriteBatch, and WriteBatchSpaced.
  • Repeated list<utf8> dictionary overflow across mid-record batch splits with enable_write_page_index() verifying record-aligned OffsetIndex page locations.
  • Upfront rejection of incompatible fallback encodings on dictionary-enabled columns.

Are there any user-facing changes?

Yes: columns with both dictionary encoding enabled and an explicit non-dictionary WriterProperties::Builder::encoding(...) now fall back to that configured encoding instead of Encoding::PLAIN when the dictionary page size limit is reached, and incompatible fallback encodings on dictionary-enabled columns now fail immediately when creating the ColumnWriter.


AI Transparency Statement

  • I have used AI tools to help generate this pull request.
  • I have reviewed, understood, and verified all AI-generated code and text in this pull request, and I take full responsibility for the contribution.
  • Tool(s) used: Gemini
  • How it was used: Used to help draft unit tests and the PR description.

…ctionary fallback

Preserve the resolved non-dictionary column encoding (`fallback_encoding_`) in `ColumnWriter::Make` when dictionary encoding is enabled, and instantiate `fallback_encoding_` instead of hardcoded `Encoding::PLAIN` in `TypedColumnWriterImpl::FallbackToPlainEncoding()`. Also validate fallback encodings upfront on `ColumnWriter` creation when dictionary encoding is enabled, and gate `CheckDictionarySizeLimit()` on `check_page` for repeated columns when pages change on record boundaries.
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55571 has been automatically assigned in GitHub to PR creator.

@pitrou

pitrou commented Nov 16, 2026

Copy link
Copy Markdown
Member

Closing as this didn't even bother using the standard PR template.

@pitrou pitrou closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants