fix(iot-platform): align mock model with SDP serverless Python 3.12 - #241
Open
ianepreston wants to merge 3 commits into
Open
fix(iot-platform): align mock model with SDP serverless Python 3.12#241ianepreston wants to merge 3 commits into
ianepreston wants to merge 3 commits into
Conversation
mlflow.pyfunc.spark_udf with env_manager='virtualenv' tries to provision a fresh Python environment inside the SDP serverless UDF sandbox. That sandbox is capped at 1GB combined memory+disk, has no reliable pyenv for arbitrary Python versions, and adds multi-minute overhead per task. MLflow docs explicitly recommend env_manager='local' or prebuilt_env_uri over 'virtualenv' for Databricks Serverless. env_manager='local' reuses the pipeline's own Python process; the required dependencies are already declared in the pipeline's environment.dependencies block (mlflow==3.1.0). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…k-model init The mock pyfunc model was logged inside a mock.patch wrapper that forced mlflow.utils.environment.PYTHON_VERSION to a hardcoded value (3.11.10 or 3.12.3 via DBDemos.get_python_version_mlflow). Combined with mlflow.pyfunc.log_model serializing the model class via cloudpickle, this baked the logging interpreter's bytecode into the model artifact. When the SDP serverless pipeline (Python 3.12.3) loaded a model pickled under Python 3.11, it segfaulted while executing the deserialized __code__ object — CPython bytecode is not portable across 3.11 -> 3.12. Removing the mock.patch lets the logged python_env.yaml reflect whatever runtime ran the init notebook (now serverless env v5 = Python 3.12.3, matching SDP). Also drop the pip_requirements kwarg on log_model and the matching mlflow/numpy/pandas/databricks-sdk version pins in the %pip install cell. These pinned mlflow==2.22.0 (env v4) — which is the wrong major on env v5 (mlflow 3.8.1) — and pinning numpy/pandas keeps the model artifact's requirements.txt frozen at versions that no longer match the runtime. With env_manager='local' on the loader side, MLflow auto- infers pip_requirements from the current env. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Field Engineering workspaces deploy this demo to serverless and the classic spark_version is dropped by dbdemos at install time, so this block is dead config on serverless. But for workspaces without serverless, the cluster is provisioned, and DBR ML 16.4 ships Python 3.11 — which then mismatches the SDP serverless Python 3.12.3 runtime and reintroduces the cloudpickle bytecode segfault. Bump both job_cluster and standalone cluster blocks to 17.3.x-cpu-ml-scala2.13 (Python 3.12) to match the same change made to the MLOps demo in a83fbbe. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The IoT wind turbine demo's SDP pipeline fails on a fresh install on serverless workspaces because the mock pyfunc model registered by
_resources/01-load-data.pyis incompatible with the SDP serverless runtime (Python 3.12.3 / MLflow 3.8.1 / cloudpickle 3.0.0).Three independent root causes addressed across three atomic commits:
env_manager='virtualenv'on the SDP UDF loader asks MLflow to provision a fresh Python env inside the serverless UDF sandbox. The sandbox is capped at 1GB combined memory+disk and has no reliable pyenv — MLflow docs explicitly recommendlocalorprebuilt_env_urifor serverless. Switched toenv_manager='local'in both the SQL and Python SDP transformations; the pipeline's existingenvironment.dependenciesblock already supplies the needed packages.mock.patch("mlflow.utils.environment.PYTHON_VERSION", ...)in init forced the logged model'spython_env.yamlto a hardcoded Python version. Combined with cloudpickle (which serializes__code__objects as CPython bytecode), this baked the logging interpreter's bytecode into the artifact. When SDP serverless (Python 3.12) loaded a model pickled under Python 3.11 it segfaulted — bytecode is not portable across 3.11→3.12. Dropped themock.patch, thefrom unittest import mockimport, and the model'spip_requirementspins (which hadmlflow==2.22.0— wrong major for serverless env v5's mlflow 3.8.1)._resources/bundle_config.pyto17.3.x-cpu-ml-scala2.13(Python 3.12), matching the same change in the MLOps demo (fc48898). FE workspaces strip this on serverless installs, but workspaces without serverless honor it and would otherwise re-introduce the 3.11→3.12 mismatch.Also drops stale pinned versions from the init notebook's
%pip install(mlflow==2.22.0 numpy==1.26.4 pandas==2.2.3) — these were tied to env v2/v4 and are wrong for env v5.Test plan
fevm-aws(serverless env v5):init_datasucceeds,start_sdp_pipelinesucceeds end-to-end with the mock model. Prior runs were segfaulting inpredictdue to cloudpickle bytecode mismatch.Known follow-up (not in this PR)
After the same job's
deploy_best_modeltask runs, AutoML's sklearn-flavored model overwrites the@prodalias. The next SDP execution then tries to load that model and fails withModuleNotFoundError: No module named 'databricks.automl_runtime'because the SDP env doesn't include it. Tracked separately — the fix is either to haveinit_dataalways force-replace the prod alias with the mock, or to adddatabricks-automl-runtime+ sklearn deps to the pipeline'senvironment.dependencies.The retail-c360 demo has the exact same vulnerable pattern (same
mock.patch+env_manager='virtualenv'+ DBR 16.4 fallback). Out of scope for this PR; will follow up with a separate one once this lands.This pull request and its description were written by Claude Code.