The entities extracted from Chinese manual documents are very messy

[Nuclear6]

I used the Chinese manual document to build it, and found that the extracted entities were very messy. Is there any good way to optimize it?

[Trip1ewhy]

Have you tried Qwen or Doubao to extract the entities?

[Nuclear6]

To be honest, I used these models to extract entities. I also tried Baidu's Wenxin 4, and the extraction results were relatively poor. I am now going to change the prompt.

[mengshouchen]

How well does using openai gpt4 work for Chinese? Which models have you already tried?
I've tried a few Microsoft models before and none of them seem to work well in Chinese.

[KylinMountain]

Before I am using llama3 and gemma2, they perform not well on the Chinese documents like network novel. Yesterday, I was trying DeepSeeker, this looks good for me. And now it is able to neo4j to visualize them. You can access my 微信公众号 喂饭教程！全网首发Neo4J可视化GraphRAG索引to test it.

[KylinMountain]

@Nuclear6 if you trying prompt tune, you should try a large language model and optimize for chinese like qwen or moonshot. Before I was using gemma2 9b, it is very bad, the prompt is too bad, the generated example is wrong.

Besides, it's hard to complete index procedure when using prompt-tune, you will meet lots of errors. I have tried this for an entire afternoon and I gave up.

But you can try.

python -m graphrag.prompt_tune --root . --domain "Chinese web novels" --language Chinese --chunk-size 300 --output prompt_zh

Remember update your entity in settings.yaml when you done prompt-tune.

[pg572]

To be honest, I used these models to extract entities. I also tried Baidu's Wenxin 4, and the extraction results were relatively poor. I am now going to change the prompt.

Hi, do you mean prompts that are written in Chinese? Could you share them if possible? Thanks!

[Trip1ewhy]

I changed the entities extraction prompt to Chinese and got a graph with less entities but they looks kind of better than English prompt's result. Is there any test data like a pair of raw file with high quality generated graph to compare Chinese prompt with English prompt for graph generation?

[Nuclear6]

我这边有8个说明书文档，总共130KB，做了如下优化效果才稍微好点：

构建索引阶段，模型使用豆包128k，跑一次10块钱：

1. embedding服务自己部署开源的bge-large-zh模型，借助oneapi进行部署；
2. 分块逻辑参考LangchainChatChat做了改造，避免使用cl100k_base切分token出现乱码的问题；
3. 重新定义实体类型，我把文档摘取一部分交给4o，让他帮我总结下需要定义哪些实体类型；
4. prompt改为中文，去掉跟说明书文档不相关的示例，可以借助4o模型生成对应的示例；

查询阶段发现查出来的实体和query相差太大，原因是采用自定义的embedding服务，需要去掉和cl100k_base的相关操作，修改之后，效果有所提升。

这是我的中文电子说明书优化经验，供大家参考！！！

I made the following optimizations:

Index building phase:

1. The embedding service deploys the open source bge-large-zh model by itself, and deploys it with the help of oneapi;

2. The block logic is modified with reference to LangchainChatChat to avoid the problem of garbled characters when using cl100k_base to split tokens;

3. Redefine the entity type. I gave a part of my document to 4o and asked him to help me summarize which entities need to be defined;

4. The prompt is changed to Chinese, and examples that are not related to the document are removed. The corresponding examples are generated with the help of the 4o model;

In the query phase, it is found that the searched entities are too different from the query. The reason is that the custom embedding service is used, and the operations related to cl100k_base need to be removed. After the modification, the effect is improved.

This is my experience in optimizing Chinese electronic manuals for your reference! ! !

[KylinMountain]

@Nuclear6

感谢大佬分享，我用deepseeker构建索引 注册后免费500万，应该足够跑了。
2. 分块逻辑 是指 修改GraphRAG中的分块吗？是否需要按照所用大模型的分块？
3和4中你没有使用Prompt-Tune自动调嘛？直接用ChatGPT翻译加修改吗？

另外有一个问题请教
我有多篇文档，都是论文摘要，是放在一个文档里使用自动切分比较好，还是每个摘要一个文档会比较好？感谢

Thank you for sharing. I used Deepseeker to build the index. After registering, I get 5 million for free, which should be enough to run.

2.	Is the chunking logic referring to modifying the chunking in GraphRAG? Does it need to be done according to the chunking of the large model used?
3.	In points 3 and 4, you didn’t use Prompt-Tune for automatic tuning? Did you directly use ChatGPT for translation and modification?

Additionally, I have a question to ask:
I have multiple documents, all of which are paper abstracts. Is it better to put them in one document and use automatic splitting, or is it better to have each abstract in a separate document? Thank you.

[Nuclear6]

@KylinMountain

1 官方用的分块是先把文档token化，按照token数进行切分，对于中文来说容易出现乱码，我看Langchain-ChatChat开源项目中用中文字符数进行切分，有效避免chunk存在乱码。

官方chunk：https://github.com/microsoft/graphrag/blob/main/graphrag/index/verbs/text/chunk/strategies/tokens.py
参考chunk：https://github.com/chatchat-space/Langchain-Chatchat/blob/master/libs/chatchat-server/chatchat/server/file_rag/text_splitter/chinese_recursive_text_splitter.py

2 我感觉分块跟模型没有太大关系，选择中文那种分块逻辑能够保证句子完整性，模型理解可能更好点。

3 没有使用官方的prompt调优，听你说容易报错，我直接拿4o对照翻译生成对应的模板。

4 按照我的理解，一个文档还是多个文档区别不大。它是针对分块抽取实体，然后针对实体和描述构建embedding，文档名没看到有太大的联系。

[KylinMountain]

非法感谢～我确实有看到Chunk中存在乱码，感谢解惑～ 👍👍👍

[dinhngoc267]

@Nuclear6 if you trying prompt tune, you should try a large language model and optimize for chinese like qwen or moonshot. Before I was using gemma2 9b, it is very bad, the prompt is too bad, the generated example is wrong.

Besides, it's hard to complete index procedure when using prompt-tune, you will meet lots of errors. I have tried this for an entire afternoon and I gave up.

But you can try.

python -m graphrag.prompt_tune --root . --domain "Chinese web novels" --language Chinese --chunk-size 300 --output prompt_zh

Remember update your entity in settings.yaml when you done prompt-tune.

hi, will the auto prompting help me define all the entities in domain? or I have to manually define in the setting

[Nuclear6]

@dinhngoc267 It is recommended that the input document example defines the entity type with the help of the gpt-4o model

[KylinMountain]

from my experience, using auto prompt tune, it didn’t generate all of the domain entities. It references your input document and generate some examples. I feel it doesn’t perform well. As @Nuclear6 said, maybe it’s better using gpt4 help to generate prompt giving your input document as example for gpt-4 reference.