Adding RTL markers in RTL localization for GameTextCompiler

[DevGeniusCode]

Background:
Users wishing to add RTL (Right-to-Left) languages to CNC Generals face an issue where the GUI engine renders text left-to-right, which forces users to manually reverse strings. Currently, there is a tool called GameTextCompiler that converts text to Unicode supported by the game. The idea of adding RTL support to this tool has been proposed, so the tool would automatically handle the reversal of RTL strings.

So, the text needs to be displayed from right to left. However, when working with .str files, all lines are reversed, making the text unreadable. A potential solution could be to reverse each line's characters to make it more readable, and then, when generating the .csf files, reverse them again to ensure proper rendering in the game.

Example:

string = "قﺎﻄﻨﻟا ﻦﻤﺿ تايانبلاو تﺎﺑﺮﻌﻟا ﺢﯿﻠﺼﺗ\n\nيناوث ٠١ : ١ ﻒﻨﺻ\nةيناث ٠٢ : ٢ ﻒﻨﺻ\nةيناث ٠٣ : ٣ ﻒﻨﺻ\n\nةدﺎﯿﻘﻟا ﺰﻛﺮﻣ : ىﺪﻟ ﻊﺑﺎﺗ"
lines = string.split("\n")
for i, line in enumerate(lines):
    lines[i] = "".join(reversed(line))

print("\n".join(lines))

Challenges & Edge Cases:

• Mixed RTL and LTR word order.
• Handling of numeric sequences and format specifiers.
• Special cases like when - appears at the beginning of a word or sentence (e.g., "- GLA has SCUD Storm").
• Ensuring that untranslated strings or numbers do not get reversed unintentionally.

Solution Proposal:

In response to the problem of mixed RTL and LTR content, and to handle edge cases more efficiently, one potential approach is to use specific Unicode characters that mark RTL segments. These characters, like LRM (Left-to-Right Mark) and RLM (Right-to-Left Mark), can help the engine automatically detect RTL text segments and render them accordingly.

To address these, I suggest adding a feature where the reversal is only applied if certain RTL characters are present in the string, and avoiding reversal for specific sections like % format fields or ASCII characters.

Example Usage:

// context: Button to go to the game options screen
GUI:GameOptions
US: "GAME OPTIONS"
AR: "[U+200F]خيارات اللعبة" // No need for [U+200E] if RTL text is at the end.

// context: The title that will have a player's name and location added to it on the stats screen
GUI:PlayerStatistics
US: "%hs, from %ls"
AR: "%hs[U+200F], من [U+200E]%ls"

CONTROLBAR:UpgradeChinaOverlordBattleBunker
US: "Battle Bunker: &B"
AR: "[U+200F]مخبأ المعركة: [U+200E]&B"

// context: Refresh the information about NAT (Network Address Translation)
GUI:FirewallRefresh
US: "Refresh NAT"
AR: "[U+200F]تحديث [U+200E]NAT"

// context: Beacons can be placed and removed from the map
GUI:DeleteBeacon
US: "Delete\nBeacon"
AR: "[U+200F]حذف\nالمنارة[U+200E]" 

How it Works:

• The [U+200F] (RLM) character forces the text after it to be displayed from right to left, while the [U+200E] (LRM) character enforces left-to-right rendering.
• This solution allows the engine to properly render RTL text without requiring manual reversal for each case.
• Note that It's not enough to type the string "[U+200E]" as it is. need to copy and paste the actual character (U+200E) directly.

Using LRM and RLM characters will also allow for future updates to the engine's Unicode library, enabling native support for RTL languages without manual intervention.

Links:

• GameTextCompiler engine
• Unicode Bidirectional Algorithm
• https://en.wikipedia.org/wiki/Arabic_script_in_Unicode

[xezon]

I think that is a good idea, as it gives full control over the direction to the text.

[DevGeniusCode]

The algorithm can be very simple

while (start <= end_line && (buffer[start] != 0x200F)) {
start++;
}
while (end <= end_line && ( buffer[start] != 0x200E)) {
end++;
}
Reverse(buffer, start, end);

But it won't know how to handle punctuation marks and numbers that appear inside a substring, so that (hello) becomes )hello(, and 123 becomes 321.
Should we expect maintainers to be careful to mark the numbers and punctuation marks, or should the algorithm know how to handle that?