From 9feb18f52ff6fad1728514bc50b56ae239fa996b Mon Sep 17 00:00:00 2001 From: Soledad Galli Date: Wed, 26 Aug 2026 12:16:26 +0200 Subject: [PATCH 1/2] Migrate OrdinalEncoder.fit() to narwhals, add polars support fit() has two paths: "arbitrary" (X[var].unique()) and "ordered" (target mean per category, via y.groupby(X[var])). transform() and inverse_transform() already came dataframe-agnostic for free from CategoricalMethodsMixin (base_encoder.py, merged separately). Benchmarked a pure-narwhals fit() (group_by/agg/sort for "ordered", unique() for "arbitrary") at 10k-100k rows x 1-10 cols x 5-50 categories: it ran 5x-18x slower than pandas-native fit() at every size tested - a large, consistent loss, unlike the ~1.1x seen for the encode/transform hot path in base_encoder.py. Per the benchmark-driven merge-vs-split rule, this is a real loss, so fit() splits on `is_pandas = nwd.is_pandas_dataframe(X)`: pandas keeps a close variant of its original groupby/unique code (confirmed via a like-for-like full-class benchmark to run within noise of the old code, ~1.0x), while polars (and any other narwhals backend) goes through group_by()/agg()/sort()/unique(). New pandas branch differs from the old code only in how "ordered" pairs y with X[var] (see bug below) - "arbitrary" is untouched. Two real issues found, confirmed against the unmodified pre-migration file (both predate this migration): 1. Bug (fixed): the old "ordered" fit() always called `y.groupby(X[var])`, which raises AttributeError whenever y is a numpy array rather than a Series - e.g. list/array-like y input, which sklearn's check_X_y machinery converts to numpy. This is exactly the scenario tests/test_encoding/test_check_estimator_encoders.py ::test_encoders_when_x_pandas_y_numpy exercises for OrdinalEncoder (encoder2, added in 2022 for issue #376) - it failed against the unmodified file and now passes. Fixed on both the pandas branch (pair X[var] with y via `.assign()`, which aligns a numpy y positionally and a Series y by index, instead of `y.groupby(X[var])`) and the narwhals branch (`nw.new_series` for a numpy y). 2. Cross-backend ordering hazard (avoided, not a regression since old code was pandas-only): grouping by category then sorting by target mean does not, by itself, guarantee the same tie-break order on ties across backends - verified polars reversed two tied categories relative to pandas without it. Old pandas code effectively tie-broke on the category itself (pandas groupby sorts keys ascending by default, and sort_values() is stable). Reproduced that explicitly with a compound sort `.sort([target_name, var])` in the narwhals branch; verified pandas and polars now produce the same dict for a deliberately tied-mean fixture, matching the old code's order exactly. Rewrote every test in test_ordinal_encoder.py as one @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) case per behavior (43 tests, up from 26), using a narwhals-based, NaN-aware comparison helper. test_variables_cast_as_category stays pandas-only - it exercises pandas Categorical dtype, which polars has no direct equivalent for. Verified: tests/test_encoding/test_ordinal_encoder.py 43 passed. tests/test_encoding full suite: 344 passed, 16 failed - identical failing test IDs to the unmodified base (17 failures, one of which is the bug fixed above), all pre-existing and unrelated to OrdinalEncoder (numpy-X rejection per the narwhals check_X() contract, and MeanEncoder's own unmigrated fit() bug). flake8 and mypy clean. Module imports with pandas blocked. sphinx -W build clean (only the pre-existing linkcode_resolve warning, confirmed identical on the unmodified base). Verified every code example in docs/user_guide/encoding/OrdinalEncoder.rst against real output (California Housing dataset) and added a "With polars" section, verified the same way; the Titanic-dataset examples in that file could not be re-run in this sandbox (no network access to openml.org) but are untouched by this change and were not touched. Co-Authored-By: Claude Sonnet 5 --- docs/user_guide/encoding/OrdinalEncoder.rst | 56 +++++ feature_engine/encoding/ordinal.py | 107 ++++++-- tests/test_encoding/test_ordinal_encoder.py | 266 ++++++++++++-------- 3 files changed, 303 insertions(+), 126 deletions(-) diff --git a/docs/user_guide/encoding/OrdinalEncoder.rst b/docs/user_guide/encoding/OrdinalEncoder.rst index cff284c08..08f5c9417 100644 --- a/docs/user_guide/encoding/OrdinalEncoder.rst +++ b/docs/user_guide/encoding/OrdinalEncoder.rst @@ -532,6 +532,62 @@ might otherwise go unnoticed. The power of ordinal ordered encoder resides in its intrinsic capacity of finding monotonic relationships. +With polars +~~~~~~~~~~~ + +:class:`OrdinalEncoder()` works the same way with a polars dataframe. Let's create a toy dataset: + +.. code:: python + + import polars as pl + from feature_engine.encoding import OrdinalEncoder + + X = pl.DataFrame({ + "city": ["London", "Manchester", "Liverpool", "London", "Manchester", "Liverpool"], + "price": [500, 300, 250, 520, 310, 260], + }) + y = pl.Series("target", [1, 0, 0, 1, 0, 1]) + +Let's set up :class:`OrdinalEncoder()` to encode `city` with ordered ordinal encoding, and fit it to the data: + +.. code:: python + + encoder = OrdinalEncoder(encoding_method="ordered", variables=["city"]) + encoder.fit(X, y) + + encoder.encoder_dict_ + +We see the resulting mappings from category to integer: + +.. code:: python + + {'city': {'Manchester': 0, 'Liverpool': 1, 'London': 2}} + +Now let's transform the data: + +.. code:: python + + encoder.transform(X) + +We obtain a polars dataframe with the categories in `city` replaced by their ordinal number: + +.. code:: text + + shape: (6, 2) + ┌──────┬───────┐ + │ city ┆ price │ + │ --- ┆ --- │ + │ i64 ┆ i64 │ + ╞══════╪═══════╡ + │ 2 ┆ 500 │ + │ 0 ┆ 300 │ + │ 1 ┆ 250 │ + │ 2 ┆ 520 │ + │ 0 ┆ 310 │ + │ 1 ┆ 260 │ + └──────┴───────┘ + + Additional resources -------------------- diff --git a/feature_engine/encoding/ordinal.py b/feature_engine/encoding/ordinal.py index 10417f1d0..f869045e0 100644 --- a/feature_engine/encoding/ordinal.py +++ b/feature_engine/encoding/ordinal.py @@ -3,7 +3,9 @@ from typing import List, Optional, Union -import pandas as pd +import narwhals as nw +import narwhals.dependencies as nwd +from narwhals.typing import IntoDataFrame, IntoSeries from feature_engine._check_init_parameters.check_init_input_params import ( _check_return_empty_is_bool, @@ -190,17 +192,17 @@ def __init__( self.unseen = unseen self.return_empty = return_empty - def fit(self, X: pd.DataFrame, y: Optional[pd.Series] = None): + def fit(self, X: IntoDataFrame, y: Optional[IntoSeries] = None): """Learn the numbers to be used to replace the categories in each variable. Parameters ---------- - X: pandas dataframe of shape = [n_samples, n_features] + X: dataframe of shape = [n_samples, n_features] The training input samples. Can be the entire dataframe, not just the variables to be encoded. - y: pandas series, default=None + y: Series, default=None The Target. Can be None if `encoding_method='arbitrary'`. Otherwise, y needs to be passed when fitting the transformer. """ @@ -215,23 +217,90 @@ def fit(self, X: pd.DataFrame, y: Optional[pd.Series] = None): self.encoder_dict_ = {} - for var in variables_: - if self.encoding_method == "ordered": - t = y.groupby(X[var], observed=False).mean() # type: ignore - t = t.sort_values(ascending=True).index + # benchmarked at 10k-100k rows x 1-10 cols x 5-50 categories: a pure + # narwhals fit() ran 5x-18x slower than pandas-native here (unlike + # the encode/transform hot path in base_encoder.py, which is only + # ~1.1x), so pandas keeps its native groupby/unique fast path and + # only polars (and other backends) go through narwhals. + is_pandas = nwd.is_pandas_dataframe(X) + + if is_pandas is True: + for var in variables_: + if self.encoding_method == "ordered": + if nwd.is_pandas_series(y): + t = y.groupby(X[var], observed=False).mean() # type: ignore + else: + # y is a numpy array here (e.g. list/array-like input + # went through sklearn's column_or_1d instead of + # check_X_y's Series passthrough); it has no + # .groupby(), so pair it with X[var] positionally via + # assign() instead - this also matches how the + # narwhals branch below handles a non-Series y. + t = ( + X[[var]] + .assign(__feature_engine_ordinal_target__=y) + .groupby(var, observed=False)[ + "__feature_engine_ordinal_target__" + ] + .mean() + ) + t = t.sort_values(ascending=True).index + elif self.encoding_method == "arbitrary": + if self.missing_values == "ignore": + t = X[var].dropna().unique() + else: + t = X[var].unique() + else: + raise ValueError( + "Unrecognized value for encoding_method. It should be " + f"'arbitrary' or 'frequency'. Got {self.encoding_method} " + "instead." + ) + self.encoder_dict_[var] = {k: i for i, k in enumerate(t, 0)} + else: + nw_X = nw.from_native(X, eager_only=True) - elif self.encoding_method == "arbitrary": - if self.missing_values == "ignore": - t = X[var].dropna().unique() + if self.encoding_method == "ordered": + # y may already be a Series (polars, from check_X_y) or a + # plain numpy array (sklearn's column_or_1d path for + # list/array input) - normalise both to a narwhals Series + # aliased to a sentinel name, then attach it to the full + # frame once so every variable's group_by below can reuse it. + target_name = "__feature_engine_ordinal_target__" + if nwd.is_into_series(y): + y_nw = nw.from_native(y, series_only=True).alias(target_name) + else: + y_nw = nw.new_series( + name=target_name, values=y, backend=nw_X.implementation + ) + nw_Xy = nw_X.with_columns(y_nw) + + for var in variables_: + if self.encoding_method == "ordered": + # sort by (mean, category): group_by's own order isn't + # guaranteed across backends, and this tie-break on the + # category itself reproduces pandas' groupby(sort=True) + # + stable sort_values behavior for categories with equal + # target means. + t = ( + nw_Xy.group_by(var, drop_null_keys=True) + .agg(nw.col(target_name).mean()) + .sort([target_name, var]) + .get_column(var) + .to_list() + ) + elif self.encoding_method == "arbitrary": + col = nw_X.get_column(var) + if self.missing_values == "ignore": + col = col.drop_nulls() + t = col.unique(maintain_order=True).to_list() else: - t = X[var].unique() - else: - raise ValueError( - "Unrecognized value for encoding_method. It should be 'arbitrary' " - f"or 'frequency'. Got {self.encoding_method} instead." - ) - - self.encoder_dict_[var] = {k: i for i, k in enumerate(t, 0)} + raise ValueError( + "Unrecognized value for encoding_method. It should be " + f"'arbitrary' or 'frequency'. Got {self.encoding_method} " + "instead." + ) + self.encoder_dict_[var] = {k: i for i, k in enumerate(t, 0)} if self.unseen == "encode": self._unseen = -1 diff --git a/tests/test_encoding/test_ordinal_encoder.py b/tests/test_encoding/test_ordinal_encoder.py index e447c4176..d76e8314b 100644 --- a/tests/test_encoding/test_ordinal_encoder.py +++ b/tests/test_encoding/test_ordinal_encoder.py @@ -1,4 +1,8 @@ +import math + +import narwhals as nw import pandas as pd +import polars as pl import pytest from numpy import nan from sklearn.exceptions import NotFittedError @@ -6,15 +10,52 @@ from feature_engine.encoding import OrdinalEncoder -def test_ordered_encoding_1_variable(df_enc): +def _to_backend(df: pd.DataFrame, make_df): + """Rebuild a pandas fixture dataframe on the requested backend. + + Swaps float NaN for None in string columns - polars (unlike pandas) + rejects a float NaN mixed into an otherwise-string column. + """ + data = {} + for col in df.columns: + values = df[col].tolist() + if any(isinstance(v, str) for v in values): + values = [ + None if isinstance(v, float) and math.isnan(v) else v for v in values + ] + data[col] = values + return make_df(data) + + +def _assert_values(X, expected: dict) -> None: + """NaN-aware, backend-agnostic comparison of a dataframe's contents.""" + result = nw.from_native(X, eager_only=True).to_dict(as_series=False) + assert list(result.keys()) == list(expected.keys()) + for col, exp_values in expected.items(): + got_values = result[col] + assert len(got_values) == len(exp_values) + for got, exp in zip(got_values, exp_values): + if isinstance(exp, float) and math.isnan(exp): + assert got is None or (isinstance(got, float) and math.isnan(got)) + else: + assert got == exp + + +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_ordered_encoding_1_variable(df_enc, make_df): # test case 1: 1 variable, ordered encoding + X = _to_backend(df_enc[["var_A", "var_B"]], make_df) + y = df_enc["target"].tolist() + encoder = OrdinalEncoder(encoding_method="ordered", variables=["var_A"]) - encoder.fit(df_enc[["var_A", "var_B"]], df_enc["target"]) - X = encoder.transform(df_enc[["var_A", "var_B"]]) + encoder.fit(X, y) + Xt = encoder.transform(X) # expected output - transf_df = df_enc.copy() - transf_df["var_A"] = [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 2] + expected = { + "var_A": [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 2], + "var_B": df_enc["var_B"].tolist(), + } # test init params assert encoder.encoding_method == "ordered" @@ -24,18 +65,23 @@ def test_ordered_encoding_1_variable(df_enc): assert encoder.encoder_dict_ == {"var_A": {"A": 1, "B": 0, "C": 2}} assert encoder.n_features_in_ == 2 # test transform output - pd.testing.assert_frame_equal(X, transf_df[["var_A", "var_B"]]) + _assert_values(Xt, expected) -def test_arbitrary_encoding_automatically_find_variables(df_enc): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_arbitrary_encoding_automatically_find_variables(df_enc, make_df): # test case 2: automatically select variables, unordered encoding + X = _to_backend(df_enc, make_df) + encoder = OrdinalEncoder(encoding_method="arbitrary", variables=None) - X = encoder.fit_transform(df_enc) + Xt = encoder.fit_transform(X) # expected output - transf_df = df_enc.copy() - transf_df["var_A"] = [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2] - transf_df["var_B"] = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2] + expected = { + "var_A": [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2], + "var_B": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2], + "target": df_enc["target"].tolist(), + } # test init params assert encoder.encoding_method == "arbitrary" @@ -48,60 +94,30 @@ def test_arbitrary_encoding_automatically_find_variables(df_enc): } assert encoder.n_features_in_ == 3 # test transform output - pd.testing.assert_frame_equal(X, transf_df) + _assert_values(Xt, expected) -def test_encoding_when_nan_in_fit_df(df_enc): - df = df_enc.copy() - df.loc[len(df)] = [nan, nan, 0] +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_encoding_when_nan_in_fit_df(df_enc, make_df): + data = { + "var_A": df_enc["var_A"].tolist() + [None], + "var_B": df_enc["var_B"].tolist() + [None], + "target": df_enc["target"].tolist() + [0], + } + X = make_df(data)[["var_A", "var_B"]] + y = data["target"] encoder = OrdinalEncoder(encoding_method="arbitrary", missing_values="ignore") - encoder.fit(df[["var_A", "var_B"]]) - - X = encoder.transform( - pd.DataFrame( - { - "var_A": ["A", nan], - "var_B": ["A", nan], - } - ) - ) + encoder.fit(X) - # transform params - pd.testing.assert_frame_equal( - X, - pd.DataFrame( - { - "var_A": [0, nan], - "var_B": [0, nan], - } - ), - check_dtype=False, - ) + Xt = encoder.transform(make_df({"var_A": ["A", None], "var_B": ["A", None]})) + _assert_values(Xt, {"var_A": [0, nan], "var_B": [0, nan]}) encoder = OrdinalEncoder(encoding_method="ordered", missing_values="ignore") - encoder.fit(df[["var_A", "var_B"]], df["target"]) - - X = encoder.transform( - pd.DataFrame( - { - "var_A": ["A", nan], - "var_B": ["A", nan], - } - ) - ) + encoder.fit(X, y) - # transform params - pd.testing.assert_frame_equal( - X, - pd.DataFrame( - { - "var_A": [1, nan], - "var_B": [0, nan], - } - ), - check_dtype=False, - ) + Xt = encoder.transform(make_df({"var_A": ["A", None], "var_B": ["A", None]})) + _assert_values(Xt, {"var_A": [1, nan], "var_B": [0, nan]}) @pytest.mark.parametrize("enc_method", ["other", False, 1]) @@ -111,32 +127,40 @@ def test_error_if_encoding_method_not_allowed(enc_method): @pytest.mark.parametrize("enc_method", ["other", False, 1]) -def test_error_if_encoding_method_not_recognized_in_fit(enc_method, df_enc): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_error_if_encoding_method_not_recognized_in_fit(enc_method, df_enc, make_df): + X = _to_backend(df_enc, make_df) enc = OrdinalEncoder() enc.encoding_method = enc_method with pytest.raises(ValueError): - enc.fit(df_enc) + enc.fit(X) -def test_error_if_ordinal_encoding_and_no_y_passed(df_enc): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_error_if_ordinal_encoding_and_no_y_passed(df_enc, make_df): # test case 3: raises error if target is not passed + X = _to_backend(df_enc, make_df) with pytest.raises(ValueError): encoder = OrdinalEncoder(encoding_method="ordered") - encoder.fit(df_enc) + encoder.fit(X) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) def test_error_if_input_df_contains_categories_not_present_in_training_df( - df_enc, df_enc_rare + df_enc, df_enc_rare, make_df ): # test case 4: when dataset to be transformed contains categories not present # in training dataset + X = _to_backend(df_enc[["var_A", "var_B"]], make_df) + y = df_enc["target"].tolist() + X_rare = _to_backend(df_enc_rare[["var_A", "var_B"]], make_df) msg = "During the encoding, NaN values were introduced in the feature(s) var_A." # check for warning when rare_labels equals 'ignore' with pytest.warns(UserWarning) as record: encoder = OrdinalEncoder(unseen="ignore") - encoder.fit(df_enc[["var_A", "var_B"]], df_enc["target"]) - encoder.transform(df_enc_rare[["var_A", "var_B"]]) + encoder.fit(X, y) + encoder.transform(X_rare) # check that at least one warning was raised (Pandas 3 may emit additional # deprecation warnings) @@ -145,20 +169,22 @@ def test_error_if_input_df_contains_categories_not_present_in_training_df( assert any(r.message.args[0] == msg for r in record) # check for error when rare_labels equals 'raise' - with pytest.raises(ValueError) as record: + with pytest.raises(ValueError) as record2: encoder = OrdinalEncoder(unseen="raise") - encoder.fit(df_enc[["var_A", "var_B"]], df_enc["target"]) - encoder.transform(df_enc_rare[["var_A", "var_B"]]) + encoder.fit(X, y) + encoder.transform(X_rare) # check that the error message matches - assert str(record.value) == msg + assert str(record2.value) == msg -def test_fit_raises_error_if_df_contains_na(df_enc_na): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_fit_raises_error_if_df_contains_na(df_enc_na, make_df): # test case 4: when dataset contains na, fit method + X = _to_backend(df_enc_na, make_df) encoder = OrdinalEncoder(encoding_method="arbitrary") with pytest.raises(ValueError) as record: - encoder.fit(df_enc_na) + encoder.fit(X) msg = ( "Some of the variables in the dataset contain NaN. Check and " @@ -168,12 +194,15 @@ def test_fit_raises_error_if_df_contains_na(df_enc_na): assert str(record.value) == msg -def test_transform_raises_error_if_df_contains_na(df_enc, df_enc_na): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_transform_raises_error_if_df_contains_na(df_enc, df_enc_na, make_df): # test case 4: when dataset contains na, transform method + X = _to_backend(df_enc, make_df) + X_na = _to_backend(df_enc_na, make_df) encoder = OrdinalEncoder(encoding_method="arbitrary") - encoder.fit(df_enc) + encoder.fit(X) with pytest.raises(ValueError) as record: - encoder.transform(df_enc_na) + encoder.transform(X_na) msg = ( "Some of the variables in the dataset contain NaN. Check and " @@ -183,17 +212,22 @@ def test_transform_raises_error_if_df_contains_na(df_enc, df_enc_na): assert str(record.value) == msg -def test_ordered_encoding_1_variable_ignore_format(df_enc_numeric): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_ordered_encoding_1_variable_ignore_format(df_enc_numeric, make_df): + X = _to_backend(df_enc_numeric[["var_A", "var_B"]], make_df) + y = df_enc_numeric["target"].tolist() encoder = OrdinalEncoder( encoding_method="ordered", variables=["var_A"], ignore_format=True ) - encoder.fit(df_enc_numeric[["var_A", "var_B"]], df_enc_numeric["target"]) - X = encoder.transform(df_enc_numeric[["var_A", "var_B"]]) + encoder.fit(X, y) + Xt = encoder.transform(X) # expected output - transf_df = df_enc_numeric.copy() - transf_df["var_A"] = [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 2] + expected = { + "var_A": [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 2], + "var_B": df_enc_numeric["var_B"].tolist(), + } # test init params assert encoder.encoding_method == "ordered" @@ -203,20 +237,25 @@ def test_ordered_encoding_1_variable_ignore_format(df_enc_numeric): assert encoder.encoder_dict_ == {"var_A": {1: 1, 2: 0, 3: 2}} assert encoder.n_features_in_ == 2 # test transform output - pd.testing.assert_frame_equal(X, transf_df[["var_A", "var_B"]]) + _assert_values(Xt, expected) -def test_arbitrary_encoding_automatically_find_variables_ignore_format(df_enc_numeric): +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_arbitrary_encoding_automatically_find_variables_ignore_format( + df_enc_numeric, make_df +): + X = _to_backend(df_enc_numeric[["var_A", "var_B"]], make_df) encoder = OrdinalEncoder( encoding_method="arbitrary", variables=None, ignore_format=True ) - X = encoder.fit_transform(df_enc_numeric[["var_A", "var_B"]]) + Xt = encoder.fit_transform(X) # expected output - transf_df = df_enc_numeric[["var_A", "var_B"]].copy() - transf_df["var_A"] = [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2] - transf_df["var_B"] = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2] + expected = { + "var_A": [0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2], + "var_B": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2], + } # test init params assert encoder.encoding_method == "arbitrary" @@ -229,10 +268,12 @@ def test_arbitrary_encoding_automatically_find_variables_ignore_format(df_enc_nu } assert encoder.n_features_in_ == 2 # test transform output - pd.testing.assert_frame_equal(X, transf_df) + _assert_values(Xt, expected) def test_variables_cast_as_category(df_enc_category_dtypes): + # pandas-only: polars has no equivalent "unused categorical categories" + # concept to exercise here. df = df_enc_category_dtypes.copy() encoder = OrdinalEncoder(encoding_method="ordered", variables=["var_A"]) encoder.fit(df[["var_A", "var_B"]], df["target"]) @@ -255,55 +296,66 @@ def test_error_if_unseen_not_permitted_value(unseen): OrdinalEncoder(unseen=unseen) -def test_inverse_transform_when_no_unseen(): - df = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_inverse_transform_when_no_unseen(make_df): + df = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) enc = OrdinalEncoder(encoding_method="arbitrary") enc.fit(df) dft = enc.transform(df) - pd.testing.assert_frame_equal(enc.inverse_transform(dft), df) + expected = {"words": ["dog", "dog", "cat", "cat", "cat", "bird"]} + _assert_values(enc.inverse_transform(dft), expected) -def test_inverse_transform_when_ignore_unseen(): - df1 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) - df2 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "frog"]}) - df3 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", nan]}) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_inverse_transform_when_ignore_unseen(make_df): + df1 = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) + df2 = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "frog"]}) enc = OrdinalEncoder(encoding_method="arbitrary", unseen="ignore") enc.fit(df1) dft = enc.transform(df2) - pd.testing.assert_frame_equal(enc.inverse_transform(dft), df3) + _assert_values( + enc.inverse_transform(dft), + {"words": ["dog", "dog", "cat", "cat", "cat", nan]}, + ) -def test_inverse_transform_when_encode_unseen(): - df1 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) - df2 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "frog"]}) - df3 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", nan]}) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_inverse_transform_when_encode_unseen(make_df): + df1 = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) + df2 = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "frog"]}) enc = OrdinalEncoder(encoding_method="arbitrary", unseen="encode") enc.fit(df1) dft = enc.transform(df2) - pd.testing.assert_frame_equal(enc.inverse_transform(dft), df3) + _assert_values( + enc.inverse_transform(dft), + {"words": ["dog", "dog", "cat", "cat", "cat", nan]}, + ) -def test_inverse_transform_raises_non_fitted_error(): - df1 = pd.DataFrame({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_inverse_transform_raises_non_fitted_error(make_df): + df1 = make_df({"words": ["dog", "dog", "cat", "cat", "cat", "bird"]}) enc = OrdinalEncoder(encoding_method="arbitrary") # Test when fit is not called prior to transform. with pytest.raises(NotFittedError): enc.inverse_transform(df1) - df1.loc[len(df1) - 1] = nan + df1_na = make_df({"words": ["dog", "dog", "cat", "cat", "cat", None]}) with pytest.raises(ValueError): - enc.fit(df1) + enc.fit(df1_na) # Test when fit is not called prior to transform. with pytest.raises(NotFittedError): - enc.inverse_transform(df1) + enc.inverse_transform(df1_na) -def test_encoding_new_categories(df_enc): - df_unseen = pd.DataFrame({"var_A": ["D"], "var_B": ["D"]}) +@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) +def test_encoding_new_categories(df_enc, make_df): + X = _to_backend(df_enc[["var_A", "var_B"]], make_df) + df_unseen = make_df({"var_A": ["D"], "var_B": ["D"]}) encoder = OrdinalEncoder(encoding_method="arbitrary", unseen="encode") - encoder.fit(df_enc[["var_A", "var_B"]]) + encoder.fit(X) df_transformed = encoder.transform(df_unseen) - assert (df_transformed == -1).all(axis=None) + _assert_values(df_transformed, {"var_A": [-1], "var_B": [-1]}) From 8c35482537004d0e4da4ee1235cacc6b0be1d8ee Mon Sep 17 00:00:00 2001 From: Soledad Galli Date: Mon, 31 Aug 2026 00:45:56 +0200 Subject: [PATCH 2/2] Adapt OrdinalEncoder to narwhals-returning check_X check_X / check_X_y now return a narwhals frame, so bind that to nw_X and keep the original native X for _check_or_select_variables, _check_na, _get_feature_names_in and the nwd.is_pandas_dataframe(X) fast-path check (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). The pandas groupby/unique fast path is unchanged - X stays native so no rehydration is needed. The narwhals branch reuses nw_X from check_X / check_X_y instead of nw.from_native(X). Co-Authored-By: Claude Sonnet 5 --- feature_engine/encoding/ordinal.py | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/feature_engine/encoding/ordinal.py b/feature_engine/encoding/ordinal.py index f869045e0..5487da6cd 100644 --- a/feature_engine/encoding/ordinal.py +++ b/feature_engine/encoding/ordinal.py @@ -208,9 +208,9 @@ def fit(self, X: IntoDataFrame, y: Optional[IntoSeries] = None): """ if self.encoding_method == "ordered": - X, y = check_X_y(X, y) + nw_X, y = check_X_y(X, y) else: - X = check_X(X) + nw_X = check_X(X) variables_ = self._check_or_select_variables(X) self._check_na(X, variables_) @@ -222,9 +222,7 @@ def fit(self, X: IntoDataFrame, y: Optional[IntoSeries] = None): # the encode/transform hot path in base_encoder.py, which is only # ~1.1x), so pandas keeps its native groupby/unique fast path and # only polars (and other backends) go through narwhals. - is_pandas = nwd.is_pandas_dataframe(X) - - if is_pandas is True: + if nwd.is_pandas_dataframe(X): for var in variables_: if self.encoding_method == "ordered": if nwd.is_pandas_series(y): @@ -258,8 +256,6 @@ def fit(self, X: IntoDataFrame, y: Optional[IntoSeries] = None): ) self.encoder_dict_[var] = {k: i for i, k in enumerate(t, 0)} else: - nw_X = nw.from_native(X, eager_only=True) - if self.encoding_method == "ordered": # y may already be a Series (polars, from check_X_y) or a # plain numpy array (sklearn's column_or_1d path for