Custom Classification model behavior

[galvangoh]

• Package Name: azure-ai-documentintelligence
• Package Version: 1.0.0b2
• Operating System: Windows
• Python Version: 3.10.9

Describe the bug
I trained the custom model via studio on 2 different titled document which share very similar pattern in its template. My classified label from the custom model is used to decide the logic flow of my application later on. Sometimes, document A gets classified as document B and vice versa.

The documentation mentioned "Custom classification models are deep-learning-model types that combine layout and language features to accurately detect and identify documents...". I don't think the "layout" here points to the layout model because blocks of text is extracted by the layout model which the custom classification does not do that. Unless there is a way to composed prebuilt and custom models, how can I classify my documents more properly? I'm happy to remain at the current version of the API and will only upgrade if there are improvement to the classification capability of the base neural model itself.

In the screenshot below, I show 2 document which I want to classify into its own label (see top right).

Questions:

1. As the document are titled differently, does the custom model picks up the title as a feature to distinguish them properly?
2. Handwritings and stamps can appear in random locations in the document. Does the custom model picks these as features during training?
3. What sort of training data is sufficient so that both document can be correctly classified?

Expected behavior
Accurate classification of different titled document despite sharing the same template.

Screenshots

[xiangyan99]

Thanks for the feedback, we’ll investigate asap.

[galvangoh]

hello there, is there any updates?

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @bojunehsu @vkurpad.

[YalinLi0312]

@bojunehsu can you help to explain the behavior?

[bojunehsu]

This is indeed a more challenging case for custom classifiers. Ideally, by providing 10-100+ diverse examples of each class (DELIVERY DOCKET, PICKING SLIP), the algorithm will learn to prioritize these words when making a classification decision. In practice, the model architecture is currently not optimized for this type of classification where classes only differ by a few key words. Though it may still work though given sufficient training examples, for this specific scenario, I would recommend running prebuilt-read and using the existence of these 2 key phrases for classification.

[galvangoh]

@bojunehsu thanks for the explanation