Skip to content

Skip missing values when center() and standardize() memorize their statistics - #271

Open
raashish1601 wants to merge 1 commit into
pydata:masterfrom
raashish1601:fix/208-nan-in-center-standardize
Open

raashish1601 wants to merge 1 commit into
pydata:masterfrom
raashish1601:fix/208-nan-in-center-standardize

Conversation

@raashish1601

Copy link
Copy Markdown

Fixes #208, and the missing-value case of #166.

center() and standardize() memorize their mean (and variance) over every row, including missing ones. A single NaN in the column makes the mean NaN, so the transformed column is NaN in every row, and the NA action then drops all rows:

df = pd.DataFrame({"x1": np.arange(5), "x2": [1, 2, np.nan, 5, 6]})
dmatrix("~ x1 + standardize(x2)", df)
# DesignMatrix with shape (0, 3)

With this change the statistics skip missing values column by column, as R's scale() does (colMeans(x, na.rm = TRUE)), and the transform leaves NaN where the input was missing. The NA action then drops only the rows that actually have missing values:

[[ 1.          0.         -1.21267813]
 [ 1.          1.         -0.72760688]
 [ 1.          3.          0.72760688]
 [ 1.          4.          1.21267813]]

Inputs without missing values give the same results as before. Only float/complex inputs are checked for NaN; other dtypes keep the previous path.

Tests: added test_Center_missing_values and test_Standardize_missing_values to patsy/test_state.py, with two columns that have NaN in different rows, memorized in one chunk and in two chunks. They fail on main and pass with this change. The full test suite passes locally and ruff format is clean.

@bashtage

bashtage commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

patsy is in effectively long-term maintenance mode with mostly only seeing absolutely required fixes to stop it from failing to operate as it has been on new versions of python.

I wouldn't say this is a hard no, but just to set expectations that we mostly don't take features or bug fixes

@raashish1601

Copy link
Copy Markdown
Author

Understood, thanks for setting expectations. I'll leave it open in case it's useful, and I'm happy to close it if you'd rather keep patsy as is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standardize(x) comes before drop na

2 participants