kenlm: add package.

[Redbeanw44602]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @Redbeanw44602, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates the 'kenlm' library into the build system as a new package. The changes define how 'kenlm' is sourced, built, and configured across different platforms, including its dependencies and specific build flags. This addition aims to make the 'kenlm' library readily available for other projects, streamlining its consumption and ensuring a consistent build process.

Highlights

• New Package Definition: Introduces the 'kenlm' library as a new package within the build system, complete with its metadata, source, and version.
• Dependency Management: Specifies all necessary dependencies for 'kenlm', including 'cmake', 'boost' (with specific configurations), 'zlib', 'bzip2', and 'xz', ensuring a complete build environment.
• Configurable Build Options: Adds configurable options for building 'kenlm', such as enabling shared library compilation (Windows-specific) and setting the maximum supported ngram order.
• Platform-Specific Adjustments: Includes conditional logic and build flags to handle platform differences (e.g., Windows, Linux/BSD) and modifies CMake files to ensure proper integration with the build system.
• Functional Test Snippet: Provides a C++ snippet test to verify that the 'kenlm' library can be successfully linked and its core functionality (like lm::ngram::Model) can be used.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds a new package for kenlm. The implementation is mostly correct, but I've found a few issues. There seems to be a typo in the version string. The configuration for shared libraries is incorrect and prevents building shared libraries on platforms other than Windows. I've also suggested an improvement to the include path handling to follow best practices and avoid polluting the global include namespace. Please see my detailed comments for suggestions.

[star-hengxing]

I can't reproduce mingw32 ci error🤔

[star-hengxing]

I use the artifact's binary, and it can still link successfully locally.

[Redbeanw44602]

I think I found the problem... for some reason, mingw32 in actions uses the wrong calling convention and generates the wrong symbols.

_BZ2_bzDecompressInit@12 (STDCALL) are the correct symbols, I verified this on my local mingw32, but I also can't reproduce the behavior of mingw32 in actions.

> nm .../xmake-home-i686-static/cache/packages/2509/k/kenlm/2025.03.31/source/kenlm/build_1a69e582/util/CMakeFiles/kenlm_util.dir/read_compressed.cc.obj | grep BZ2_bzDecompressInit
         U _BZ2_bzDecompressInit <-- CDECL
> nm .../xmake-home-i686-static/packages/b/bzip2/1.0.8/a870fcdce81d46329faaad0d721dd583/lib/libbz2.a | grep BZ2_bzDecompressInit
00000fb0 T _BZ2_bzDecompressInit@12

[Redbeanw44602]

I solved this problem.

mingw-w64-i686-bzip2 package caused interfere, because it's one of the toolchain's dependencies, so it's present in the CI environment.

Compile command:

[30/64] D:\a\_temp\msys64\mingw32\bin\i686-w64-mingw32-g++.exe -DBOOST_ATOMIC_NO_LIB -DBOOST_ATOMIC_STATIC_LINK -DBOOST_CHRONO_NO_LIB -DBOOST_CHRONO_STATIC_LINK -DBOOST_CONTAINER_NO_LIB -DBOOST_CONTAINER_STATIC_LINK -DBOOST_DATE_TIME_NO_LIB -DBOOST_DATE_TIME_STATIC_LINK -DBOOST_PROGRAM_OPTIONS_NO_LIB -DBOOST_PROGRAM_OPTIONS_STATIC_LINK -DBOOST_THREAD_NO_LIB -DBOOST_THREAD_STATIC_LINK -DBOOST_THREAD_USE_LIB -isystem C:/Users/runneradmin/AppData/Local/.xmake/packages/b/boost/1.88.0/46ec3c41e4804f928303b7c28fcdbb84/include -m32 -DLZMA_API_STATIC -O3 -DNDEBUG -std=c++11  -DHAVE_ZLIB -DHAVE_BZLIB -DHAVE_XZLIB -MD -MT util/CMakeFiles/kenlm_util.dir/read_compressed.cc.obj -MF util\CMakeFiles\kenlm_util.dir\read_compressed.cc.obj.d -o util/CMakeFiles/kenlm_util.dir/read_compressed.cc.obj -c C:/Users/runneradmin/AppData/Local/.xmake/cache/packages/2509/k/kenlm/2025.03.31/source/kenlm/util/read_compressed.cc

Apparently, bzip2 was not included in the header search directory list, which should have been considered a problem, but we ignored it :(

Then comes the most incredible thing I think. The packager of MSYS2 actually modified the header file of bzip2. I totally cannot understand why they did this!

https://github.com/msys2/MINGW-packages/blob/fc7b78887864e62079a63b0267af2eeafc0407fb/mingw-w64-bzip2/bzip2-cygming-1.0.6.src.all.patch#L104

So, the fix is obvious. First we should add packagedeps and then patch bzip2 to make it consistent with MSYS2 system package manager (although I really don't like doing that).

Attachment: https://fars.ee/5W1t (Compiler output after adding the -E)

[star-hengxing]

Seems xmake dosen't fetch the system bzip2, and package cmake find the system bzip2, not good😡