Skip to content

Optional Lindera tokenizer support (was: Custom tokenizer support) #200

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
cjrh wants to merge 10 commits into
base: master
Choose a base branch
from
Draft

Optional Lindera tokenizer support (was: Custom tokenizer support) #200

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d9fc544
WIP
cjrh Jan 21, 2024
6e82b23
Add a lindera test case
cjrh Jan 29, 2024
64dc8df
Merge remote-tracking branch 'upstream/master' into custom-tokenizer-…
cjrh Feb 17, 2024
27e2f41
Made all lindera options configurable.
cjrh Feb 18, 2024
a94c3cc
Fix docstring import
cjrh Feb 18, 2024
7ea1d8f
Fix cargo fmt
cjrh Feb 18, 2024
051e790
hide test imports for non-lindera test run
cjrh Feb 18, 2024
3765098
include lindera in the CI matrix
cjrh Feb 18, 2024
0a3f53c
Fix download host for lindera data
cjrh Feb 18, 2024
a985769
fix quotes
cjrh Feb 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,16 +40,13 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: [3.9]
lindera: ["", "_lindera"]
python-version: ["3.10"]
include:
- os: ubuntu-latest
python-version: "3.12"
- os: ubuntu-latest
python-version: "3.11"
- os: ubuntu-latest
python-version: "3.10"
- os: ubuntu-latest
python-version: 3.8
runs-on: "${{ matrix.os }}"
steps:
- name: Harden Runner
Expand All @@ -67,6 +64,7 @@ jobs:
index.crates.io:443
static.rust-lang.org:443
objects.githubusercontent.com:443
download.johtani.info:443

- uses: actions/checkout@61b9e3751b92087fd0b06925ba6dd6314e06f089
- uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # 5.0.0
Expand All @@ -79,4 +77,4 @@ jobs:

- uses: Swatinem/rust-cache@23bce251a8cd2ffc3c1075eaa2367cf899916d84 # 2.7.3
- run: python3 -m pip install nox
- run: nox -s test-${{ matrix.python-version }}
- run: nox -s test${{matrix.lindera}}-${{ matrix.python-version }}
Loading