[Python][C++][Dataset][Parquet] Row-group statistics drop NaN rows for != and negated comparisons on float columns

[singhpratech]

Describe the bug, including details regarding any error messages, version, and platform.

Filtering a Parquet dataset on a float or double column can drop rows that contain NaN. The row group is skipped based on its min/max statistics, but NaN is not part of min/max.

import pyarrow as pa, pyarrow.parquet as pq, pyarrow.dataset as ds, pyarrow.compute as pc

t = pa.table({"x": pa.array([5.0, float("nan"), 5.0, 5.0])})
pq.write_table(t, "a.parquet")                       # statistics: min 5.0, max 5.0
pc.filter(t, pc.not_equal(t["x"], 5.0))              # [nan]
pq.read_table("a.parquet", filters=[("x", "!=", 5.0)])   # [] -- row lost

pq.write_table(t, "b.parquet", write_statistics=False)
pq.read_table("b.parquet", filters=[("x", "!=", 5.0)])   # [nan]

u = pa.table({"x": pa.array([1.0, float("nan"), 10.0])})
pq.write_table(u, "c.parquet")                       # statistics: min 1.0, max 10.0
ds.dataset("c.parquet").to_table(filter=~(ds.field("x") <= 10.0))   # [] -- in memory: [nan]

The same happens for ~(x == 5), is_null(x, nan_is_null=True) and x.isin([nan]), and for pc.is_nan(x) when min == max and the column has no nulls. Filters that are false for NaN (<, <=, >, >=, ==) are not affected.

Cause: ParquetFileFragment::EvaluateStatisticsAsExpression (cpp/src/arrow/dataset/file_parquet.cc) turns the statistics into the guarantee x == min when min == max, or x >= min and x <= max, optionally or is_null(x). Writers do not include NaN in min/max, so the guarantee does not hold for NaN rows. SimplifyWithGuarantee then simplifies x != 5, invert(x <= 10) and the others to false and the row group is skipped. GH-28074 (#15125) handled NaN written as min or max, but not NaN values outside them.

Expected: the same rows as filtering in memory or reading without statistics. Row groups can still be skipped for filters that are false for NaN.

Environment: pyarrow 25.0.1, macOS 26.6.2 arm64, Python 3.13. Also reproduced on C++ main at 1f789ff.

Component(s)

C++, Parquet, Python

[singhpratech]

take