Encoding issues with Non-US characters in orchestrator input object

[ThomasContini]

Hi all,

I'm using Azure Durable Functions with PowerShell.
I have a Client function with a HTTP trigger, in charge of passing the HTTP Body JSON-serialized object to my orchestrator which need it to determine which activity functions it will trigger.

The input JSON contains non-US characters (french ones: "é"/"à"/...), but my Client function seems to be able to read them. Write-Host instruction prints well the entire JSON object.
Then I'm passing the same JSON as an input for a new orchestrator. When I'm trying to print the content of that JSON inside the orchestrator, it's now printing "�" instead of french characters...

Default encoding seems to be UTF-8. I tried to re-encode the same JSON with iso-8859-1, but still the same issues...

I don't really understand why I could face this kind of issues between different functions, any help is welcome! :)

Here is my code:

Client function (HTTP Trigger)

using namespace System.Net

param($Request, $TriggerMetadata)

$json = ConvertTo-Json $Request.Body -Depth 10
Write-Host $json

try {
    $orchestrator = $Request.Params.FunctionName
    $InstanceId = Start-NewOrchestration -FunctionName $orchestrator -InputObject $json    
    $response = New-OrchestrationCheckStatusResponse -Request $Request -InstanceId $InstanceId
        
    Push-OutputBinding -Name Response -Value $response
}
catch {
    Write-Error $_.Exception.Message
    Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
        StatusCode = 400
        Body       = '{"error": "Invalid JSON."}'
    })
}

Orchestrator

param($Context)

Write-Host $Context.Input.gettype()
Write-Host $Context.Input

$JSON = ConvertFrom-Json $Context.Input -AsHashtable

And the corresponding log with local emulator:

[2021-07-26T14:51:11.701Z] Host lock lease acquired by instance ID '0000000000000000000000003C60A9FE'.
[2021-07-26T14:51:17.235Z] Executing 'Functions.ProvisionningHttpStart' (Reason='This function was programmatically called via the host APIs.', Id=2ba1a283-5358-47f6-8779-4c9ec830b9c5)
[2021-07-26T14:51:17.847Z] INFORMATION: {
[2021-07-26T14:51:17.848Z]   "actions": [
[2021-07-26T14:51:17.849Z]     {
[2021-07-26T14:51:17.849Z]       "action": "NewSite",
[2021-07-26T14:51:17.849Z]       "input": {
[2021-07-26T14:51:17.850Z]         "url": "https://m365x478483.sharepoint.com/sites/test-enc14",
[2021-07-26T14:51:17.850Z]         "owner": "[email protected]",
[2021-07-26T14:51:17.851Z]         "type": "CommunicationSite",
[2021-07-26T14:51:17.851Z]         "title": "Comité d'éléction"
[2021-07-26T14:51:17.852Z]       },
[2021-07-26T14:51:17.853Z]       "id": 0
[2021-07-26T14:51:17.854Z]     }
[2021-07-26T14:51:17.854Z]   ],
[2021-07-26T14:51:17.855Z]   "client": {
[2021-07-26T14:51:17.855Z]     "contact": [],
[2021-07-26T14:51:17.856Z]     "sharePointUrl": "https://m365x478483.sharepoint.com",
[2021-07-26T14:51:17.856Z]     "name": "EKILOG",
[2021-07-26T14:51:17.857Z]     "tenant": "m365x478483.onmicrosoft.com",
[2021-07-26T14:51:17.857Z]     "appId": "b27e1fd2-dfc7-4ece-b17a-30bd8b449910"
[2021-07-26T14:51:17.857Z]   }
[2021-07-26T14:51:17.858Z] }
[2021-07-26T14:51:20.533Z] Executing 'Functions.ProvisionningOrchestrator' (Reason='(null)', Id=d8f282e3-e075-4257-9cdd-2f6720544b4f)
[2021-07-26T14:51:20.543Z] Your function 'ProvisionningOrchestrator' is queuing requests as there are no available runspaces. You may be able to increase your throughput by following the best practices on https://aka.ms/functions-powershell-concurrency.
[2021-07-26T14:51:20.561Z] Executed 'Functions.ProvisionningHttpStart' (Succeeded, Id=2ba1a283-5358-47f6-8779-4c9ec830b9c5, Duration=3326ms)
[2021-07-26T14:51:20.611Z] INFORMATION: System.String
[2021-07-26T14:51:20.612Z] INFORMATION: {
[2021-07-26T14:51:20.613Z]   "actions": [
[2021-07-26T14:51:20.614Z]     {
[2021-07-26T14:51:20.614Z]       "action": "NewSite",
[2021-07-26T14:51:20.615Z]       "input": {
[2021-07-26T14:51:20.615Z]         "url": "https://m365x478483.sharepoint.com/sites/test-enc14",
[2021-07-26T14:51:20.616Z]         "owner": "[email protected]",
[2021-07-26T14:51:20.617Z]         "type": "CommunicationSite",
[2021-07-26T14:51:20.617Z]         "title": "Comit� d'�l�ction"
[2021-07-26T14:51:20.618Z]       },
[2021-07-26T14:51:20.618Z]       "id": 0
[2021-07-26T14:51:20.619Z]     }
[2021-07-26T14:51:20.620Z]   ],
[2021-07-26T14:51:20.620Z]   "client": {
[2021-07-26T14:51:20.621Z]     "contact": [],
[2021-07-26T14:51:20.622Z]     "sharePointUrl": "https://m365x478483.sharepoint.com",
[2021-07-26T14:51:20.623Z]     "name": "EKILOG",
[2021-07-26T14:51:20.623Z]     "tenant": "m365x478483.onmicrosoft.com",
[2021-07-26T14:51:20.624Z]     "appId": "b27e1fd2-dfc7-4ece-b17a-30bd8b449910"
[2021-07-26T14:51:20.625Z]   }
[2021-07-26T14:51:20.626Z] }
[2021-07-26T14:51:20.673Z] OUTPUT: []
[2021-07-26T14:51:20.736Z] Executed 'Functions.ProvisionningOrchestrator' (Succeeded, Id=d8f282e3-e075-4257-9cdd-2f6720544b4f, Duration=206ms)

[cgillum]

Adding @AnatoliB, who might have some insights on this given his experience with the PowerShell worker.

Just to confirm, you see the right characters printed when you use Write-Host in your HTTP trigger, right?

[ThomasContini]

First, thank you for your reply,

The right characters are correctly printed with Write-Host but only in the HTTP Trigger function.
When I'm passing the same JSON as input for the Orchestrator, then printing it, I got the "�".

[TKMane]

Hello,

Looking to revive this topic as I have come across the same situation, except I am having a problem with a currency symbol £ (British Pound).

I can see the HTTP Trigger receiving the correct data, the £ symbol is visible.

As soon as the input is passed to the Orchestration, I also receive "�"

I have tried several suggestion such as UTF-8 encoding, but nothing seems to encode the symbol correctly. This is either before or after the input is passed to the orchestration.

$ (Dollar) appears to be encoded fine and is passed on.

Kind regards,

TK

[TKMane]

To reply to my own post, I have managed to resolve the issue by moving the powershell version from 7.2 to 7.4

I also noticed that the encoding used when the HTTP trigger received the body, it is US-ASCII , but as soon as the orchestrator receives it and I inspect $Context , the encoding changes to unicode.

I used the following function to get this data:

function Get-Encoding {
    param(
        [string]$InputString
    )
    
$encodings = @(
        [System.Text.Encoding]::ASCII,
        [System.Text.Encoding]::UTF8,
        [System.Text.Encoding]::Unicode,
        [System.Text.Encoding]::UTF32,
        [System.Text.Encoding]::UTF7,
        [System.Text.Encoding]::Default
    )

    foreach ($encoding in $encodings) {
        $bytes = [System.Text.Encoding]::Unicode.GetBytes($InputString)
        $decodedString = $encoding.GetString($bytes)

        if ($decodedString -eq $InputString) {
            return $encoding.EncodingName
        }
    }

    return "Unknown"
}