[azure-ai-ml] Enable hdfs input/output mode for Spark jobs and components by lavakumarrepala · Pull Request #48314 · Azure/azure-sdk-for-python

lavakumarrepala · 2026-07-28T23:58:25Z

Description

Enables the hdfs input/output delivery mode for Spark jobs and components in azure-ai-ml.

Fixes ICM the service defaults Spark inputs to hdfs, but the Python SDK previously rejected any Spark input/output mode other than direct, so users could not explicitly set mode="hdfs". This blocked region-expansion work for the MSAI Data Platform team.

Root cause

spark_helpers._validate_input_output_mode allowed only InputOutputModes.DIRECT and raised a ValidationException before serialization for any other mode. Adding an HDFS constant/mapping alone (as in #46925) is insufficient because this validator rejects hdfs first.

Changes

spark_helpers.py — Spark input/output validator now accepts both direct and hdfs (new _SPARK_SUPPORTED_MODES). Error message updated accordingly.
constants/_common.py — added InputOutputModes.HDFS = "hdfs".
_input_output_helpers.py — REST mappings HDFS -> "Hdfs" and "Hdfs"/"hdfs" -> HDFS; binding-input path uses .mode.lower() for consistency.
pipeline/_io/mixin.py — tolerate both enum and string mode values.
Tests — added HDFS REST round-trip test, Spark accepts hdfs/direct test, Spark rejects unsupported mode test; updated 4 existing Spark validation message expectations.
CHANGELOG.md — entry under 1.35.0.

Validation

New + updated unit tests pass (3 new, 4 updated, 10 spark-related).
black formatting clean on changed files.

Note for reviewers

InputDeliveryMode (v2023_04_01_preview) has no Hdfs enum member, so the raw string "Hdfs" is sent on the wire. Per the ICM, other Spark components already submit with hdfs, so the service accepts it — please confirm for this API version.

Checklist

Tests added/updated for the changed behavior.
No breaking changes.

ICR: https://msdata.visualstudio.com/Vienna/_workitems/edit/3842468

Adds InputOutputModes.HDFS and REST mappings, and updates the Spark input/output mode validator to accept 'hdfs' in addition to 'direct'. Previously the Python SDK rejected any Spark input/output mode other than 'direct', blocking users from explicitly setting 'hdfs' delivery mode (ICM 586585073). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b49997ca-8f69-4ee8-beca-78dadc4816e5

azure-pipelines · 2026-07-28T23:58:58Z

Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

azure-pipelines · 2026-07-29T04:35:19Z

Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

Adds HDFS delivery-mode support for Azure ML Spark jobs and components.

Changes:

Adds the public hdfs mode constant and REST input mappings.
Expands Spark validation to accept direct and hdfs.
Adds tests and release notes for the new behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`CHANGELOG.md`	Documents HDFS support.
`constants/_common.py`	Defines `InputOutputModes.HDFS`.
`_input_output_helpers.py`	Adds HDFS input REST mappings.
`pipeline/_io/mixin.py`	Handles string and enum input modes.
`spark_helpers.py`	Allows HDFS during Spark validation.
`test_pipeline_job_schema.py`	Tests HDFS input round-tripping and validation.
`test_pipeline_job_entity.py`	Updates validation-message expectations.


-# Only "direct" mode is supported for spark job inputs and outputs
+# Only "direct" and "hdfs" modes are supported for spark job inputs and outputs
+_SPARK_SUPPORTED_MODES = (InputOutputModes.DIRECT, InputOutputModes.HDFS)


    InputOutputModes.EVAL_MOUNT: InputDeliveryMode.EVAL_MOUNT,
    InputOutputModes.EVAL_DOWNLOAD: InputDeliveryMode.EVAL_DOWNLOAD,
    InputOutputModes.DIRECT: InputDeliveryMode.DIRECT,
+    InputOutputModes.HDFS: "Hdfs",


Regenerated API surface to include the new HDFS enum member so the api-md-consistency CI check passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b49997ca-8f69-4ee8-beca-78dadc4816e5

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

sdk/ml/azure-ai-ml/azure/ai/ml/entities/_job/_input_output_helpers.py:118

The HDFS mapping is only added for inputs. A standalone Spark Output(mode="hdfs") now passes _validate_input_output_mode, but SparkJob._to_rest_object() calls to_rest_data_outputs(), which indexes OUTPUT_MOUNT_MAPPING_TO_REST and raises KeyError; reading an HDFS output similarly fails in from_rest_data_outputs(). Add the raw "Hdfs" mapping in both output mapping directions.

    InputOutputModes.HDFS: "Hdfs",

sdk/ml/azure-ai-ml/azure/ai/ml/entities/_job/spark_helpers.py:125

YAML-loaded Spark jobs still reject hdfs before this validator runs. SparkJobSchema uses InputsField/OutputsField, while DataInputSchema, MLTableInputSchema, and OutputSchema hard-code allowed modes without HDFS (_schema/job/input_output_entry.py:99-161). Add HDFS to the applicable Spark job YAML schemas and cover loading a Spark YAML with HDFS input/output modes.

_SPARK_SUPPORTED_MODES = (InputOutputModes.DIRECT, InputOutputModes.HDFS)

sdk/ml/azure-ai-ml/azure/ai/ml/constants/_common.py:840

The public Input and Output constructor docstrings still list their accepted modes without hdfs (entities/_inputs_outputs/input.py:38-42 and output.py:79-83). This leaves the API reference inconsistent with the new public constant and changelog. Add hdfs to both mode descriptions and clarify that it is supported for Spark jobs/components.

    HDFS = "hdfs"
    """HDFS asset type."""

+                mode_val = val.mode.value if hasattr(val.mode, "value") else val.mode
+                rest_dataset_literal_inputs[name].update({"mode": mode_val})


Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

sdk/ml/azure-ai-ml/azure/ai/ml/entities/_job/_input_output_helpers.py:118

HDFS is added only to the input mappings. A Spark Output(mode="hdfs") now passes validation, but to_rest_data_outputs indexes OUTPUT_MOUNT_MAPPING_TO_REST and raises KeyError; output deserialization has the same gap. Add raw Hdfs mappings in both output directions.

    InputOutputModes.HDFS: "Hdfs",

sdk/ml/azure-ai-ml/azure/ai/ml/entities/_job/pipeline/_io/mixin.py:274

This enum-or-string fallback is applied only to inputs. _to_rest_outputs still evaluates binding["mode"].value; because HDFS must use the raw string "Hdfs", a Spark output bound to a pipeline output will raise AttributeError. Apply the same normalization to output bindings.

                mode_val = val.mode.value if hasattr(val.mode, "value") else val.mode
                rest_dataset_literal_inputs[name].update({"mode": mode_val})

sdk/ml/azure-ai-ml/azure/ai/ml/entities/_job/spark_helpers.py:125

Spark jobs loaded from YAML still reject mode: hdfs before reaching this validator. SparkJobSchema uses DataInputSchema/ModelInputSchema/MLTableInputSchema and OutputSchema, and their allowed_values omit HDFS (_schema/job/input_output_entry.py:79-160). Add HDFS to the applicable Spark job schemas and cover standalone and inline Spark YAML loading.

# Only "direct" and "hdfs" modes are supported for spark job inputs and outputs
_SPARK_SUPPORTED_MODES = (InputOutputModes.DIRECT, InputOutputModes.HDFS)

sdk/ml/azure-ai-ml/azure/ai/ml/constants/_common.py:840

The public Input.mode and Output.mode docstrings still list only the previous modes, so generated API documentation does not describe when this new public value is supported. Update both mode descriptions to include HDFS for Spark jobs/components.

    HDFS = "hdfs"
    """HDFS asset type."""

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (1)

sdk/ml/azure-ai-ml/azure/ai/ml/entities/_job/_input_output_helpers.py:118

HDFS is added only to the input mappings, but the validator now accepts it for outputs too. Serializing a Spark Output(mode="hdfs") reaches to_rest_data_outputs() and indexes OUTPUT_MOUNT_MAPPING_TO_REST["hdfs"], which raises KeyError; deserializing an output returned as Hdfs fails similarly. Add HDFS to both output mappings so the advertised output support actually round-trips.

    InputOutputModes.HDFS: "Hdfs",

github-actions Bot added the Machine Learning label Jul 28, 2026

lavakumarrepala marked this pull request as ready for review July 29, 2026 04:34

Copilot AI review requested due to automatic review settings July 29, 2026 04:34

lavakumarrepala requested review from JustinFirsching, NonStatic2014, achauhan-scc, arunsu, jayesh-tanna, kingernupur, nick863, novaturient95, rtanase, sharma-riti and vivram as code owners July 29, 2026 04:34

Copilot started reviewing on behalf of lavakumarrepala July 29, 2026 04:35 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

Update api.md and api.metadata.yml for InputOutputModes.HDFS

f5692d1

Regenerated API surface to include the new HDFS enum member so the api-md-consistency CI check passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: b49997ca-8f69-4ee8-beca-78dadc4816e5

Copilot AI review requested due to automatic review settings July 29, 2026 06:23

Copilot started reviewing on behalf of lavakumarrepala July 29, 2026 06:24 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

Comment thread sdk/ml/azure-ai-ml/azure/ai/ml/entities/_job/pipeline/_io/mixin.py

Comment on lines +273 to +274

mode_val = val.mode.value if hasattr(val.mode, "value") else val.mode

rest_dataset_literal_inputs[name].update({"mode": mode_val})

Merge branch 'main' into v-rlava/spark-hdfs-input-mode

487c40b

Copilot AI review requested due to automatic review settings July 29, 2026 06:36

Copilot started reviewing on behalf of lavakumarrepala July 29, 2026 06:37 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

Merge branch 'main' into v-rlava/spark-hdfs-input-mode

0a0367d

Copilot AI review requested due to automatic review settings July 31, 2026 04:21

Copilot started reviewing on behalf of lavakumarrepala July 31, 2026 04:22 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[azure-ai-ml] Enable hdfs input/output mode for Spark jobs and components - #48314

[azure-ai-ml] Enable hdfs input/output mode for Spark jobs and components#48314
lavakumarrepala wants to merge 4 commits into
mainfrom
v-rlava/spark-hdfs-input-mode

lavakumarrepala commented Jul 28, 2026 •

edited

Loading

Uh oh!

azure-pipelines Bot commented Jul 28, 2026

Uh oh!

azure-pipelines Bot commented Jul 29, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		mode_val = val.mode.value if hasattr(val.mode, "value") else val.mode
		rest_dataset_literal_inputs[name].update({"mode": mode_val})

Uh oh!

Conversation

lavakumarrepala commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Root cause

Changes

Validation

Note for reviewers

Checklist

Uh oh!

azure-pipelines Bot commented Jul 28, 2026

Uh oh!

azure-pipelines Bot commented Jul 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lavakumarrepala commented Jul 28, 2026 •

edited

Loading