Missing charset on response

[ahankinson]

@jkunze suggested I open this issue here.

This affects this page: https://arks.org/ark:33864 (but also others with an accented character in the name field)

The Content-Type header is missing ;charset=utf-8 on JSON responses. This has the effect of using the default browser encoding which, for Safari, is Western Latin 1.

This means that any JSON response with accented characters will not appear correctly when viewed in Safari.

To fix this all JSON responses should set the charset property on the Content-Type header.

[ahankinson]

Are there any updates on this? @jkunze @datadavev ?

[datadavev]

Thanks for raising the issue.

The default encoding for JSON is UTF-8 and can only be UTF-8, UTF-16, or UTF-32 ¹. Hence, it is a misconfiguration or bug for Safari to apply Western Latin 1 encoding to JSON.

Nonetheless, this will be patched in an upcoming release.

edit: Actually the spec (rfc8259#section-8.1) now indicates that JSON must be UTF-8. Adding the char set identifier is redundant, albeit harmless.

edit2: Yep, a bug in Safari: https://bugs.webkit.org/show_bug.cgi?id=197369

Footnotes

1. https://www.rfc-editor.org/rfc/rfc7159#section-8.1 ↩

[datadavev]

HTTP uses charset names to indicate or negotiate the character encoding scheme of a textual representation¹, that is media of text subtypes, e.g. text/plain, text/html.

The JSON media type is application/json ² which is an application subtype, and the JSON spec has no provision for an encoding in the media type.

Hence, it would appear to be incorrect to include a charset parameter in the Content-Type for a JSON response since: 1. the media type is an application sub type, and 2. UTF-8 is the only valid charset for JSON.

Footnotes

1. https://www.rfc-editor.org/rfc/rfc9110.html#section-8.3.2 ↩

2. https://www.rfc-editor.org/rfc/rfc8259#section-11 ↩

[ahankinson]

If your server is always returning JSON in UTF-8, though, it does no harm to include it in the Cintent Type header then, right? It might be redundant, but it’s not wrong.

[ahankinson]

I would be interested in knowing the solution you come up with, if including the charset in the JSON content type header is not acceptable to you. As it is, though, the ARKS site is not displaying correctly for Safari users.

[datadavev]

Adding the charset to the response header is redundant but seems to be harmless (compliant parsers will ignore it). It'll be added (likely in the next patch release), but not a high priority.

Note that this issue only impacts Safari users who have not adjusted their settings to use UTF-8, which would generally be the preferred default (Settings, General, Default encoding).