Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: <title> Failed to get encoding for cl100k_base when getting num_tokens_from_string #660

Closed
Anthonyfhd opened this issue Jul 23, 2024 · 1 comment
Closed

[Bug]: <title> Failed to get encoding for cl100k_base when getting num_tokens_from_string #660

Anthonyfhd opened this issue Jul 23, 2024 · 1 comment
Labels
community_support Issue handled by community members

Comments

@Anthonyfhd
Copy link

Describe the bug

INFO: Generating entity extraction prompt...
Failed to get encoding for cl100k_base when getting num_tokens_from_string. Fall back to default encoding cl100k_base

Steps to reproduce

python -m graphrag.prompt_tune --root . --domain "Chinese law and policy about Management Measures and Regulations" --language Chinese --chunk-size 300 --output prompt_zh

Expected Behavior

No response

GraphRAG Config Used

No response

Logs and screenshots

No response

Additional Information

  • GraphRAG Version:
  • Operating System:
  • Python Version:
  • Related Issues:
@Anthonyfhd Anthonyfhd added bug Something isn't working triage Default label assignment, indicates new issue needs reviewed by a maintainer labels Jul 23, 2024
@natoverse
Copy link
Collaborator

Routing to language support issue #696.

We also recently release 0.2.0 which enforces utf-8 encoding on input reads.

@natoverse natoverse closed this as not planned Won't fix, can't repro, duplicate, stale Jul 25, 2024
@natoverse natoverse added community_support Issue handled by community members and removed bug Something isn't working triage Default label assignment, indicates new issue needs reviewed by a maintainer labels Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
community_support Issue handled by community members
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@natoverse @Anthonyfhd