Repository navigation
Support polars and other dataframe libraries through narwhals - #1145
Open
jeroenjanssens wants to merge 27 commits into
Open
jeroenjanssens wants to merge 27 commits into
jeroenjanssens wants to merge 27 commits into
Conversation
This makes the baseline test suite green on pandas 3.1.
- plotnine/_utils/frame.py has backend-agnostic helpers on top of narwhals: frame and series construction, concat with dtype relaxation, sorted group splitting, missing values, levels and ordinal detection (Enum = ordinal). - `pytest --backend polars` (or `make test-polars`) converts the pandas test data to polars as it enters a plot, so the whole test suite, including the image comparisons, exercises polars. - Known polars failures can be listed in tests/known_failures_polars.txt; they are strict xfails.
The grouping, interaction, margin, missing-value and join helpers now work on narwhals dataframes of any backend. A transitional native_io adapter lets the remaining pandas code keep calling them with pandas dataframes. Polars cannot have a nominal categorical with custom ordered levels, so plotnine uses an Enum for those and records its categories as nominal for the duration of the plot build. Enums are otherwise ordinal; pandas keeps its native unordered categoricals.
Aesthetics can now be narwhals expressions, e.g.
aes(y=nw.col("x") * 2). String expressions remain supported. They
are first evaluated with the column names standing in for narwhals
expressions, which works for every dataframe library. If that does
not give an expression, the string is evaluated with the native
series, and lastly with numpy arrays.
factor() and reorder() no longer need pandas, and factor() now
honours ordered=True.
The layer data is normalised at the entry point: pandas data gets
positional row labels and string column names, and decimals become
floats for every backend. A stable sort helper replaces the unstable
narwhals pandas sort.
The layout of the panels is a narwhals dataframe and PANEL is a plain integer column (it was a pandas categorical). Facet variables, margins and missing combinations are computed with narwhals, so they work with any dataframe library, and layers with data from different libraries can be faceted together.
Scales train, map and transform narwhals dataframes. Scale methods get numpy arrays, or narwhals series for categorical columns so the order of their levels is known. Discrete ranges are trained without pandas categoricals. I() no longer relies on a pandas extension dtype to tag columns. The layer records which aesthetics are literal and keeps those columns away from the scales, so the tag works with every dataframe library.
An aesthetic mapped to a narwhals expression is labelled with the name of the column the expression creates.
Positions get and return narwhals dataframes. Stacking and dodging no longer use the pandas index: the computations are positional, and a width for each row (e.g. position_dodge(width=[...])) is matched to the rows with an explicit row number. Group keys that are NaN are left out for every backend, as pandas does.
Stats compute with narwhals dataframes and numpy arrays, and the dataframes they create are of the same library as the layer data. - pandas.cut is replaced by an equivalent numpy implementation. - The ECDF is computed with numpy; it gives the same values as the statsmodels ECDF. - Formula based models get a dict of numpy arrays instead of a pandas dataframe. - User functions for stat_summary get native series and those for stat_smooth get native dataframes, e.g. polars series for polars data. The built-in summary functions work with any array-like. - The swarm, stack and dodge computations no longer use the pandas index to put results back into place. - Fix stats.density.get_var_type, which took every discrete column to be ordered.
Geoms get narwhals dataframes, and they pass numpy arrays and lists to matplotlib. The legend keys are narwhals dataframes, and geom.draw_legend gets a row of the legend data as a dict. - Coordinate munching no longer uses the pandas index. - Polygons, rugs and paths are grouped without pandas groupby, respecting the order of first appearance where pandas used sort=False. - geom_map gets the bounds from the shapely geometries, so it does not need a geopandas GeoSeries. - Helpers for dataframes with object columns (e.g. tuples for linetypes): unique_rows and inner_join.
Running the test suite with polars data revealed these differences: - Missing values, None and NaN, match each other when values are matched to the limits of a scale. - A column of only Nones is kept as an object column (pandas) or a string column (other libraries), instead of a null column that numpy turns into NaN floats. - mizani gets python scalars for the alpha values. - Summary functions like np.mean that do not work with a polars series get a numpy array. The tests that inspect the layer data use narwhals, so that they work with both backends. The one test that uses the pandas index is a known polars failure.
- The datasets are loaded with pandas if it is installed and otherwise with polars. load_dataset(name, backend=...) loads a dataset with a specific library. - plotnine.typing describes dataframes and series of any library supported by narwhals. - qplot evaluates its expressions like the other aesthetics. - ggplot.layer_data returns a dataframe of the library of the layer data. - The copy-on-write option is only set when pandas<3 is in use.
The layer data is no longer converted to a native dataframe between the steps of building a plot. Dataframes that plotnine creates when there is no data to take the library from, e.g. for annotate(), use polars if it is installed and pandas otherwise. The helpers that accept narwhals dataframes keep accepting native ones too, which is useful for code (e.g. extensions) that calls them with pandas dataframes.
The new tests draw plots with polars data in a process in which pandas cannot be imported. They are skipped when the installed version of mizani requires pandas. Whether a package is installed is checked without failing when the import machinery reports that it is missing.
statsmodels requires pandas, and plotnine used it for the default computations of geom_density, geom_violin, geom_sina, geom_beeswarm, geom_density_2d, geom_pointdensity and geom_smooth(method="lm") and of the default smoothing method for groups of 1000 or more points. The kernel density estimators and the least squares fit are now ported from statsmodels. They give the same results: the univariate densities differ by at most a few units in the last place, and the multivariate densities and least squares fits are identical. Options that the ports do not support (e.g. other bandwidth methods for multivariate densities, method_args for the linear models) and the other smoothing methods still use statsmodels. When statsmodels is not installed, they fail with a helpful error.
Older versions of polars sort a categorical in the order in which the values appear. plotnine sorts them in the lexical order of the values, which is the order of their levels. The CI runs the test suite with the polars backend too.
The type aliases for series describe pandas series for static type checking only, so plotnine.typing does not import pandas.
Scalars are added to dataframes as typed literals, the dtype of python values is inferred from the set of their types, and matching values to the limits of scales does not call a function per value. In benchmarks with the diamonds data, plots with pandas data take 0-12% longer to draw than before the switch to narwhals, and plots with polars data take as long or less.
pivot_apply is no longer used. The parameter documentation describes the data as a dataframe of any library.
Empty dataframes are created without assuming the class of the dataframe library. Libraries that cannot have Enums (pyarrow) get categoricals whose categories are in lexical order.
- mizani transforms python timedeltas, so the values of timedelta columns are given to the scales as python timedeltas. - The columns of an empty polars dataframe have no type; they are now float columns, like those of an empty pandas dataframe. - The order of appearance of groups is found with the smallest row number of each group, which does not depend on the order of the rows within the groups.
…olars Polars dataframes do not have an index, so the test that reorders by x.index is marked pandas_only instead of being a known polars failure. A new test reorders by row order with an expression that works for all backends.
The conflicts were where upstream changed code that this branch converts to narwhals: - The removal of the pandas inplace= calls (has2k1#1137) is superseded by the narwhals code. - The modernized type annotations (has2k1#1140) are applied to the code of this branch too. - The drawing context (has2k1#1143) uses copy-on-write for pandas<3 only if pandas has been imported. - The fix of the kde nan check for numpy 2.5 (has2k1#1144) does not apply; the density is computed without statsmodels. The new stat_bindot test reads the layer data with narwhals, so that it works with the polars test backend.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1145 +/- ##
==========================================
- Coverage 88.92% 88.84% -0.09%
==========================================
Files 229 232 +3
Lines 17320 18242 +922
Branches 2204 2411 +207
==========================================
+ Hits 15402 16207 +805
- Misses 1289 1356 +67
- Partials 629 679 +50 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes plotnine work on dataframes through narwhals instead of pandas. Polars dataframes, and the other libraries that narwhals supports (e.g. PyArrow tables), are used natively and are not converted to pandas. pandas dataframes work as before, and the test suite produces the same images for pandas and for polars data.
The goal is that plotnine can be used without pandas installed. pandas stays a required dependency in this PR, so nothing changes for anyone's installation. Making it an optional extra needs a mizani release without pandas (has2k1/mizani#91), and can follow in a later PR.
Closes #602. Fixes #977.
You can try it without installing anything: this notebook was run in an environment without pandas.
Categoricals
In #602 you wrote that plotnine needs both nominal and ordinal categoricals to carry an explicit order, and that polars categoricals do not. This PR handles that as follows:
ordered=Trueordered=FalseEnumEnumCategoricalfactor(x, ordered=False)with anEnumEnumfactor(x, categories=[...])Enummeans ordinal. A polarsEnumis the only polars type with a fixed, explicit order, and narwhals already maps an ordered pandas categorical tonw.Enum(and an unordered one tonw.Categorical).Categoricalis nominal and sorted lexically. The levels are the values that occur in the column, so the global string cache does not leak other values into the legend (All categorical values are displayed in the legend when source dataframe is polars.DataFrame #977).ordered=False. With other libraries it is anEnumthat plotnine registers as nominal while the plot is built. The registry is aContextVarscoped to the build, keyed by the categories.Expressions
An aesthetic can be a narwhals expression, e.g.
aes(y=nw.col("hwy") * 0.425). String expressions remain supported. A string is evaluated in this order:nw.col(...). That makes"np.log(displ)","hwy / cty"and"factor(cyl)"backend-agnostic.The index
plotnine no longer relies on the pandas index. Where it mattered (dodging, merging the computed statistics back with the data, sorting), the code uses row positions or an explicit row id column instead. Sorts are stable for every backend.
Changes for extension authors
These are in the changelog:
data.to_native()andnw.from_native().PANELis an integer column; it was a pandas categorical.layout.layoutis a narwhals dataframe.geom.draw_legendandgeom.legend_key_sizeget the legend row as a dict.I()returns anAsIsobject instead of a pandas extension array.ggplot.layer_data()returns a dataframe of the same library as the layer data.stat_summaryget a series of the data's library (with a numpy fallback). Functions passed as themethodofstat_smoothget a dataframe of the data's library.statsmodels
statsmodels requires pandas, so the defaults that used it now have ports in
stats/_kde.pyandstats/_ols.py:stat_densityandstat_ydensity;stat_density_2dandstat_pointdensity;lmand defaultglmsmooths;tests/test_kde_ols.pycompares the ports to statsmodels across kernels, bandwidths and weights; they agree to a relative tolerance of 1e-12 or better. The other methods (rlm,gls,lowess, formulas,method_args,stat_quantile, discrete KDE variables) still use statsmodels. Without it, the smoothing methods andstat_quantileraise an error that says statsmodels is needed.Other changes
plotnine.data.load_dataset(name, backend=None)loads a dataset with pandas or polars. When pandas is not installed, the datasets inplotnine.dataare polars dataframes.factor(x, ordered=True)now creates an ordered categorical; the argument was ignored.Testing
make test, andmake test-polars(pytest --backend polars). The polars mode converts the test data to polars as it enters a plot, so every image test also tests polars. There is a polars job in the CI matrix.x.index), is markedpandas_onlyand skipped.tests/test_no_pandas.pydraws 21 kinds of plots in a subprocess in which pandas cannot be imported, and checks that pandas was not imported. It skips itself while the installed mizani needs pandas.tests/test_kde_ols.pycompares the statsmodels ports to statsmodels.Performance
I benchmarked seven plots at 10k, 100k and 1M rows against
main, with identical package versions. (I ran it before merging the latestmaininto this branch.) Each run was a fresh process, with 6–10 samples per measurement and 95% bootstrap confidence intervals. All setups rendered pixel-identical images.mainwith pandas data, and 0.36–0.83× with polars data. For a 1M-point plot with 12 facets, a continuous colour scale and linear smooths, the full plot goes from 10.6 s to 8.3 s (pandas) and 7.8 s (polars).main.import plotninegoes from 388 ms to 255 ms, because it no longer imports pandas.Reviewing
The diff is large (141 files, +6.6k/−3.0k), but the commits are organized by area:
plotnine/_utils/frame.pyand the polars test backend.frame.pyis where most of the backend-specific behaviour lives. It covers stable sorts, typed literals, categoricals, object columns in polars and missing values. It is the best place to start.