Skip to content

add CI building .jar; parser quick'n'dirty compat with 3.8.4#9

Open
azrdev wants to merge 22 commits intoturanar:masterfrom
azrdev:master
Open

add CI building .jar; parser quick'n'dirty compat with 3.8.4#9
azrdev wants to merge 22 commits intoturanar:masterfrom
azrdev:master

Conversation

@azrdev
Copy link
Copy Markdown

@azrdev azrdev commented Aug 3, 2023

  • the GitHub Actions CI builds a .jar including all dependencies, for everyone to use. it's retained for 90 days after the push of the triggering commit
  • push a git tag, and the CI creates a github release attaching the matching .jar, so it's retained indefinitely
  • maven now generates the parser from the Stellaris.g4 grammar as part of the build process
  • the java code has been beaten at until it doesn't break with (german) Stellaris 3.8.4 from steam -- I did not properly implement all the changes in the game files, so the resulting tech tree might be wrong (and is surely missing something), but...

the result is at least a usable tech tree: https://azrdev.github.io/stellaris-tech-tree/3.8.4-gemini/

azrdev and others added 22 commits August 1, 2023 21:40
found in 3.8.3 common/scripted_triggers/00_scripted_triggers.txt
found in many files on 3.8.3, e.g. files/common/pop_jobs/03_worker_jobs.txt
as found in e.g. common/component_templates/00_utilities.txt
would catch common/edicts/99_README_EDICTS.txt
as found in common/buildings/07_amenity_buildings.txt
@benaryorg
Copy link
Copy Markdown

benaryorg commented Sep 25, 2024

A quick note on the compatibility with current versions of Stellaris (3.13.0):

  • some of the localization files in 3.13.0 seem to be broken in terms of YAML (I mean.… technically using single spaces for indent is already awkward): e.g. localisation/english/machine_age_situations_3_l_english.yml lacks spaces in line 77 (this affects more lines and files)
  • similarly localisation/english/nemesis_crisis_l_english.yml line 11 spells crisis_path_destruction_name:0 "Path of Destruction" which is not YAML (also affects a lot of files)
  • files/common/technology/00_repeatable.txt being empty threw a hard error
  • localisation/english/astral_rifts_1_l_english.yml line 346 features a beautiful \T that should've been \nT
  • the lowercasing of internationalisation strings causes issues as localisation/english/astral_planes_l_english.yml contains ASTRAL_THREADS: "$astral_threads$" in line 128 which then causes infinite recursion (and thus a stack overflow)
  • common/edicts/00_edicts.txt has line 1692 with mult = value:percent_monthly_resource_income_edict_cost|RESOURCE|minerals|PERCENT|1| which seemed to make the parser unhappy (though I'm not 100% sure about that one, may have been me tampering with the file)
  • common/technology/00_cosmic_storm_tech.txt had several cost settings that threw the parser off, presumably because of the inline_script in line 9-11, I just cut out all the costs and didn't debug further

To be honest, at this point I think using an actual YAML parser causes more issues than it solves considering the seeming leniency that Stellaris uses with its own parser, but that's just me dunking on the blatant YAML violations of Stellaris upstream *sigh*
Personally I'd go with the following (been years since I wrote Java so please don't mind me jotting down some Rust please):

file.lines()
  // skip the first line containing the `l_english:`
  .skip(1)
  // trim whitespace on each line, this means removing the indentation
  .map(str::trim)
  // no empty lines
  .filter(|line| !line.is_empty())
  // filter comments
  .filter(|line| line.chars().next().unwrap() != '#')
  .map(|line| {
    // everything before the first colon, may need to be changed so spaces depending on how `menace_name:1 "value"` is *supposed* to be parsed
    let key = line.split_once(':')?.0;
    // everything after the first space
    let raw_value = line.split_once(' ')?.1
      // replace all the unknown escape sequences with question marks (i.e. \T becomes \u003f), see below for more info
      .replace({ let mut is_escape = false; move |ch| { if is_escape { is_escape = false; !['t', 'n', '"', '\\'].contains(&ch) } else { is_escape = ch == '\\'; false } }, "u003f");
    // actually parse YAML from the rvalue, this one seems mostly fine, it's just the format of the file that's off, not the escaped strings, *mostly*
    let value = yaml_parse(raw_value);
    (key, value)
  })

This would fix the first four issues in the list, leaving only the latter three open, but those need actual code/parser fixes.

The following is a grep'd list of escape characters used (non-overlapping matches, which means x\\txx\tx but x\\\txx\<tab>x):

grep -hP '^\s*[^ #]' /path/to/steamapps/common/Stellaris/localisation/**/*.yml | grep -Po '\\.' | sort | uniq -c | sort -n
Count Escape
239380 \n
16077 \"
663 \\
76 \t
1 \T

So that shouldn't be much of a problem, it's just that one file that's broken it seems, however who knows what the future holds when this one slipped through.
As said it's mostly the format of the file, e.g. yq-go utterly fails when faced with localisation/english/additional_content_l_english.yml switches from double space to single space indent in line 38, however overall the errors are just three types:

Count Error
5 could not find expected ':'
72 mapping values are not allowed in this context
97 did not find expected key

I mean.… localisation/english/nemesis_crisis_l_english.yml actually chokes yq-go on that line: menace_name:1 "$menace$:" before it even reaches the \T so.… eh.

Anyway, hope this here helps anyone down the line.

Edit: Small bonus, if you could also bump the spring boot dependency to 2.1.18 (i.e. the last 2.1.x release) that'd be cool, since my local build system requires a fix for an upstream bug and bumping to 2.1.18 seems to work without a hitch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants