fix: make _data code and unit tests pandas-3 compatible#711
Merged
mplatzer merged 3 commits intoApr 24, 2026
Merged
Conversation
…t.cast under pandas 3 pandas 3.0 removed the 'ignore' value from pd.to_numeric's errors kwarg, so any call with errors='ignore' now raises ValueError. Our VirtualInteger.cast / VirtualFloat.cast passed errors='ignore' (so that non-numeric input could be returned unchanged and encompasses() could fall back). After the bump, every cast() raised, the outer try/except in encompasses() swallowed it, and encompasses() silently returned False for valid integer/float series — turning VirtualInteger/Float encompassment into VirtualVarchar. Replace with pd.to_numeric(errors='raise') wrapped in a try/except that falls back to the original series on ValueError/TypeError, matching the previous contract on both pandas 2.2 and 3.0. Also normalize time_coerce's null output to pd.NA (instead of pd.NaT) so pd.testing.assert_series_equal on object-dtype series behaves consistently across pandas versions. Co-authored-by: Michi Platzer <michael.platzer@gmail.com>
…backed strings pandas 3.0 removed StringMethods.__getitem__ for pyarrow-backed extension arrays, so the old df[c].str[:-7] trim (used to strip the trailing '.XXXXXX' microseconds that pyarrow-backed timestamp strftime still emits regardless of format string) raises AttributeError. Replace with .str.slice(stop=-7), which is supported on both numpy- and pyarrow-backed string arrays on pandas 2.x and 3.0. Co-authored-by: Michi Platzer <michael.platzer@gmail.com>
… compatible - np.array_split(df, n) returns ndarrays on numpy 2 / pandas 3 instead of DataFrames, which breaks df_part.to_parquet(...). Split via positional indices so each part stays a DataFrame. - test_export_data_to_excel compared a read_excel output (float NaN) against an expected column of [None]*9 (object dtype). Pandas 3's assert_frame_equal is strict about the NaN vs None distinction; assert the column is fully NA via .isna() instead. Co-authored-by: Michi Platzer <michael.platzer@gmail.com>
mplatzer
merged commit 5b7d18f
into
chore/update-engine-2.5.0-qa-1.10.4-github
9 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Changes
Fixes the remaining 10 failed / 2 errored unit tests in
tests/_data/unitonchore/update-engine-2.5.0-qa-1.10.4-github, all caused by pandas 3.0 / numpy 2 / pyarrow 24 behavior changes that CI now picks up on this branch. Three small commits:mostlyai/sdk/_data/dtype.py— restoreerrors="ignore"semantics forVirtualInteger.cast/VirtualFloat.cast(pandas 3 removed that value frompd.to_numeric); normalizetime_coerce's null output frompd.NaTtopd.NAfor consistent equality.mostlyai/sdk/_data/file/table/json.py—Series.str[:-7]→Series.str.slice(stop=-7)(pandas 3 removed__getitem__on the string accessor for pyarrow extension arrays).tests/_data/unit/test_finalize_generation.py,tests/_data/unit/test_push.py—np.array_split(df, n)now returns ndarrays under numpy 2 / pandas 3; split via positional indices so each part stays a DataFrame. And compare the all-null column via.isna()instead of equality against[None] * 9.Why this change?
On PR #708 (engine 2.5.0 / qa 1.10.4 from GitHub) the CPU CI job was red. The E2E hang was already fixed in #709; the remaining failures are all unit tests in
tests/_data/unitthat regressed because pandas 3.0 / numpy 2.4 / pyarrow 24 changed behavior:pd.to_numeric(..., errors="ignore")now raisesValueError: invalid error value specified. Our.cast()methods used that to return input unchanged on non-numeric data; after the bump every.cast()raised, the outertry/exceptinencompasses()swallowed it, andVirtualInteger/Float.encompasses()silently returnedFalsefor valid int/float series (cascading intoVirtualInteger().encompass(int) == VirtualVarchar()intest_encompass_integer).objectdtype, pandas 3'sassert_series_equalnow distinguishespd.NaTandpd.NA;time_coercewas returningNaTwhile tests asserted<NA>.np.array_split(df, n)now returns a list of ndarrays instead of DataFrames for a DataFrame input, breakingdf_part.to_parquet(...)in two fixtures.Series.str[key](StringMethods.__getitem__) no longer works on pyarrow-backed extension arrays;Series.str.slice(...)still does.assert_frame_equalwithcheck_dtype=Falsestill rejectsNaNvsNonefor all-null columns.The first two are real runtime bugs in
mostlyai/sdk/_data/(they would hit users writing JSON or using auto-typing on pandas 3), not just test fragility.Testing
With the CI-matched env (
pandas==3.0.2,numpy==2.4.3,pyarrow==24.0.0, engine 2.5.0 + qa 1.10.4 from git):pytest -vv tests/_data/unit tests/_local/unit tests/client/unit tests/test_domain.py→ 380 passed, 2 skipped (the 2 skips are pre-existing@pytest.mark.skips).pytest -vv tests/_local/end_to_end -k 'not (client and mode)'→ 9 passed, 2 deselected (unchanged from after fix(pull): preserve ctx_key column when per-group sampling under pandas 3.0 #709).Each commit is scoped to one logical change so they can be cherry-picked or reverted independently.
Additional Notes
Targets the PR #708 branch, not
main. After #708 merges this can be folded in on top.