Windows agents have different character set than in the past

[MarkEWaite]

Service(s)

ci.jenkins.io

Summary

The git parameter plugin includes a test that asserts the presence of a non-ASCII character in a Jenkins list box. The test consistently passed on the Azure Windows agents and consistently fails on the AWS Windows agents.

Reproduction steps

1. Review the test failure and the change and its PR that skips the test on CI with Windows

[smerle33]

this PR jenkins-infra/packer-images#1774 change the windows 2019 AMD line used by this build (see jenkins-infra/packer-images#1773), so we may need to test again when the new packer-image will be deployed (probably after the weekly release as this one will get a lot of changes, because it was lock for a while see: #4565)

from Microsoft Windows Server 2019 Core Locale English AMI provided by Amazon
to Microsoft Windows Server 2019 Core with EC2LaunchV2\, English locale AMI provided by Amazon

so shouldn't impact on the character set

[dduportal]

A new packer Windows image with a new base image has been deployed to ci.jio for agents (ref. jenkins-infra/jenkins-infra#3921).

I've opened jenkinsci/git-parameter-plugin#146 to have a playground in which to try the disabled test and validate our settings.

[smerle33]

new character set:

[System.Text.Encoding]::Default                                                                                                                                                                                                                                                
                                                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                                          
IsSingleByte      : True                                                                                                                                                                                                                                                                                  
BodyName          : iso-8859-1                                                                                                                                                                                                                                                                            
EncodingName      : Western European (Windows)                                                                                                                                                                                                                                                            
HeaderName        : Windows-1252                                                                                                                                                                                                                                                                          
WebName           : Windows-1252                                                                                                                                                                                                                                                                          
WindowsCodePage   : 1252                                                                                                                                                                                                                                                                                  
IsBrowserDisplay  : True                                                                                                                                                                                                                                                                                  
IsBrowserSave     : True                                                                                                                                                                                                                                                                                  
IsMailNewsDisplay : True                                                                                                                                                                                                                                                                                  
IsMailNewsSave    : True                                                                                                                                                                                                                                                                                  
EncoderFallback   : System.Text.InternalEncoderBestFitFallback                                                                                                                                                                                                                                            
DecoderFallback   : System.Text.InternalDecoderBestFitFallback                                                                                                                                                                                                                                            
IsReadOnly        : True                                                                                                                                                                                                                                                                                  
CodePage          : 1252

old character set:

[System.Text.Encoding]::Default

Preamble          :
BodyName          : utf-8
EncodingName      : Unicode (UTF-8)
HeaderName        : utf-8
WebName           : utf-8
WindowsCodePage   : 1200
IsBrowserDisplay  : True
IsBrowserSave     : True
IsMailNewsDisplay : True
IsMailNewsSave    : True
IsSingleByte      : False
EncoderFallback   : System.Text.EncoderReplacementFallback
DecoderFallback   : System.Text.DecoderReplacementFallback
IsReadOnly        : True
CodePage          : 65001

lets try to set it back to utf-8

[smerle33]

those commands don't work:

PS C:\Users\administrator> [System.Text.Encoding]::Default                                                                                                                      
                                                                                                                                                                                
                                                                                                                                                                                
IsSingleByte      : True                                                                                                                                                        
BodyName          : iso-8859-1                                                                                                                                                  
EncodingName      : Western European (Windows)                                                                                                                                  
HeaderName        : Windows-1252                                                                                                                                                
WebName           : Windows-1252                                                                                                                                                
WindowsCodePage   : 1252                                                                                                                                                        
IsBrowserDisplay  : True                                                                                                                                                        
IsBrowserSave     : True                                                                                                                                                        
IsMailNewsDisplay : True                                                                                                                                                        
IsMailNewsSave    : True                                                                                                                                                        
EncoderFallback   : System.Text.InternalEncoderBestFitFallback                                                                                                                  
DecoderFallback   : System.Text.InternalDecoderBestFitFallback                                                                                                                  
IsReadOnly        : True                                                                                                                                                        
CodePage          : 1252                                                                                                                                                        
                                                                                                                                                                                
                                                                                                                                                                                
                                                                                                                                                                                
PS C:\Users\administrator> [System.Console]::OutputEncoding = [System.Console]::InputEncoding = [System.Text.Encoding]::UTF8                                                    
PS C:\Users\administrator> $PSDefaultParameterValues['*:Encoding'] = 'utf8'                                                                                                     
PS C:\Users\administrator>                                                                                                                                                      
PS C:\Users\administrator> [System.Text.Encoding]::Default                                                                                                                      
                                                                                                                                                                                
                                                                                                                                                                                
IsSingleByte      : True                                                                                                                                                        
BodyName          : iso-8859-1                                                                                                                                                  
EncodingName      : Western European (Windows)                                                                                                                                  
HeaderName        : Windows-1252                                                                                                                                                
WebName           : Windows-1252                                                                                                                                                
WindowsCodePage   : 1252                                                                                                                                                        
IsBrowserDisplay  : True                                                                                                                                                        
IsBrowserSave     : True                                                                                                                                                        
IsMailNewsDisplay : True                                                                                                                                                        
IsMailNewsSave    : True                                                                                                                                                        
EncoderFallback   : System.Text.InternalEncoderBestFitFallback                                                                                                                  
DecoderFallback   : System.Text.InternalDecoderBestFitFallback                                                                                                                  
IsReadOnly        : True                                                                                                                                                        
CodePage          : 1252                                                                                                                                                        
                                                                                                                                                                                
                                                                                                                                                                                
                                                                                                                                                                                
PS C:\Users\administrator>

[dduportal]

those commands don't work:

I guess they require a restart once applied: worth trying within packer-image (a PR, once merged it will publish a "staging" AMI which we can test manually)

[smerle33]

PR was open https://infra.ci.jenkins.io/job/infra-tools/job/packer-images/job/PR-1783/1/pipeline-console/?selected-node=150

[smerle33]

source of knowledge: https://stackoverflow.com/questions/51933189/character-encoding-utf-8-in-powershell-session