AML Datastore (hosted on ASH) download error for connected training on ASH

[penorouzi]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ x ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

1. Follow https://github.com/Azure/AML-Kubernetes/blob/master/docs/ASH/Train-AzureArc.md to create datastore on AML
2. Run all Cells until before "Create or attach existing ArcKubernetesCompute" section of [https://github.com/Azure/AML-Kubernetes/blob/master/docs/ASH/notebooks/distributed-tf2-cifar10/distributed-tf2-cifar10.ipynb](this sample notebook)
3. Run the following command in the same notebook above:

dataset = Dataset.get_by_name(ws, name=dataset_name)
dataset.download(target_path='.', overwrite=False)

4. You will get the error below:

Any log messages given by the failure

UserErrorException: UserErrorException:
Message: Execution failed in operation 'download' for Dataset(id='18838f8e-8dd0-4de4-b922-cc3071d2a35f', name='CIFAR-10', version=1, error_code=ScriptExecution.StreamAccess.Validation,error_message=ScriptExecutionException was caused by StreamAccessException.
StreamAccessException was caused by ValidationException.
'GetHttpResourceStream' for '[REDACTED]' on storage failed with status code 'BadRequest' (The value for one of the HTTP headers is not in the correct format.), client request ID 'c2a3a196-72a9-42f1-9166-6e946f3e32c3', request ID '7c7d8a46-f423-0972-4ded-92df603efee9'. Error message: [REDACTED]Make sure server has no special header requirements or try using different datasource type.
| session_id=436df111-0f04-4071-bd7d-6feaf688878a) ErrorCode: ScriptExecution.StreamAccess.Validation
InnerException
Error Code: ScriptExecution.StreamAccess.Validation
Validation Error Code: BadRequest
Validation Target: HttpRequest
Failed Step: d05d7c55-ae86-4734-858e-e10368436345
Error Message: ScriptExecutionException was caused by StreamAccessException.
StreamAccessException was caused by ValidationException.
'GetHttpResourceStream' for 'https://stackstorage.blob.orlando.azurestack.corp.microsoft.com/datasets/UI/01-15-2021_014825_UTC/dataset.csv' on storage failed with status code 'BadRequest' (The value for one of the HTTP headers is not in the correct format.), client request ID 'c2a3a196-72a9-42f1-9166-6e946f3e32c3', request ID '7c7d8a46-f423-0972-4ded-92df603efee9'. Error message: InvalidHeaderValueThe value for one of the HTTP headers is not in the correct format.
RequestId:7c7d8a46-f423-0972-4ded-92df603efee9
Time:2021-01-22T00:22:52.5839090Zx-ms-version2019-12-12Make sure server has no special header requirements or try using different datasource type.
| session_id=436df111-0f04-4071-bd7d-6feaf688878a
ErrorResponse
{
"error": {
"code": "UserError",
"message": "Execution failed in operation 'download' for Dataset(id='18838f8e-8dd0-4de4-b922-cc3071d2a35f', name='CIFAR-10', version=1, error_code=ScriptExecution.StreamAccess.Validation,error_message=ScriptExecutionException was caused by StreamAccessException.\r\n StreamAccessException was caused by ValidationException.\r\n 'GetHttpResourceStream' for '[REDACTED]' on storage failed with status code 'BadRequest' (The value for one of the HTTP headers is not in the correct format.), client request ID 'c2a3a196-72a9-42f1-9166-6e946f3e32c3', request ID '7c7d8a46-f423-0972-4ded-92df603efee9'. Error message: [REDACTED]Make sure server has no special header requirements or try using different datasource type.\r\n| session_id=436df111-0f04-4071-bd7d-6feaf688878a) ErrorCode: ScriptExecution.StreamAccess.Validation"
}
}

Expected/desired behavior

Dataset files should get downloaded without any errors. Instead of ASH storage, I used Azure storage for creating the datastore in step 1 above and there was no issue.

OS and Version?

Windows 10

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.