Skip to content

✏️ Added 23 new model properties and harden JSON extraction [#15] - #16

Open
MikeACedric wants to merge 3 commits into
mainfrom
feature/bug-fixes
Open

✏️ Added 23 new model properties and harden JSON extraction [#15]#16
MikeACedric wants to merge 3 commits into
mainfrom
feature/bug-fixes

Conversation

@MikeACedric

@MikeACedric MikeACedric commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds 23 new LLM/VLM properties from recent releases and hardens JSON response.

New properties

  • The ORKG template R609825 predates recent architectures (MoE, reasoning modes, agentic RL,
    multimodal). 23 new properties are now integrated end-to-end — schema, prompt
    (<Conditional-Fields>), few-shot examples (GLM-5 / Kimi K2 / Phi-4). Examples:
    activated_parameters, attention_mechanism, moe_configuration, reasoning_mode,
    rl_algorithm, benchmark_result, vision_encoder, base_model.

JSON response robustness

  • Unescaped inner quotes — key/value-aware escaping pass.
  • Stray ints for string fields (e.g. number_of_attention_heads: 64) — coerced to string.
  • Inconsistent , vs ; separators — comma fields split on both; ; fields stay ;-only.
  • Bracket-aware splitting — commas inside ()/[]/{} don't fragment values.

Example Runs (Sandbox)

Example Runs (Live)

Related Issue:

[#15]

@MikeACedric
MikeACedric requested a review from jd-coderepos July 24, 2026 16:47
@MikeACedric MikeACedric self-assigned this Jul 24, 2026

@jd-coderepos jd-coderepos left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dear @MikeACedric , thank you for this PR and the examples. Very helpful to get an idea about how the model is doing. I picked one use case https://orkg.org/papers/R1911271 and circled some extraction anomalies below. Might it be possible to define stricter criteria around the extraction for properties?

Examples

Image

When on visits the property link for pretraining corpus https://orkg.org/properties/P41655 you actually see that it has a description. It is best even for new properties added to give them a description. This you can do by visiting the property link directly and then click the plus against description and enter the text there. This description should be written in a way that makes clear which kind of values the property supports. So it is a way to disambiguate similar named properties.

To handle unmeaningful extractions as in 1 as well as 2 and 3 below, as discussed in the meeting yesterday, aside from heuristics I am wondering if a semantic final screening pass can be applied for the values extracted before they are confirmed as valid values. This semantic pass could be based on the description of the property.

Image
Image

@jd-coderepos

Copy link
Copy Markdown
Member

I've found another issue with this automated check of resources from the ORKG. Same paper https://orkg.org/papers/R1911271

Note the Reasoning resource. I believe this was automatically fetched via the API. Note what happens when one clicks the plus beside it.

Screenshot 2026-08-07 095224

It expands as follows.

Screenshot 2026-08-07 095242

Basically this is then not the right resource to be added as a value here.

Maybe when you query against the API with the label Reasoning to see if the resource exists or not, at the moment, I assume the code picks the first one in the list. We would have to think about how to not pick the wrong resource. The only other alternative as a low hanging fruit is to check of the user id of the contributor. So you can have a list of "approved" user ids -- these would be contributors who are guaranteed to make only AI/Gen AI related contributions and to accept the existing resource only if it were created by these valid user ids. If none can be found then create a new resource. Something like.

@MikeACedric

Copy link
Copy Markdown
Collaborator Author

Runs on Sandbox with the two fixes mentioned above:

  1. GLM-5: https://sandbox.orkg.org/papers/R2178035
  2. KIMI-K2: https://sandbox.orkg.org/papers/R2178056

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants