Support for Structured Output

[daniel-j-h]

At the moment llm is great when the user is a human. What would make llm extremely powerful is if it could deterministically always output a user-defined format so that we can use it in combination with other programs, think in a pipeline where you can be sure that the output is e.g. in JSON with certain properties and you can safely pipe it through jq.

There seems to be various ways to restrict model outputs

• Some folks use json schema to describe a tool and its arguments and force a model to use a tool; this has the downside that e.g. ollama doesn't support forcing a model to use a tool and tool usage is optional - related: Tool usage research #607
• Some other folks can restrict a model's output with grammars or something along the lines of grammars

I've seen you looked into grammars before in

https://til.simonwillison.net/llms/llama-cpp-python-grammars

and for example ollama now supports a higher level version of that by providing a json schema when calling the model

https://github.com/ollama/ollama/releases/tag/v0.5.0

With structured output support, the llm would be super powerful in pipelines and bash scripts and pretty much everything that wants to build up workflows on top of llm's output.

The feature could look like a new option for the llm cli where it takes a json schema for a start? Or maybe you have better ideas how to design this feature.

Any thoughts around this?

[daniel-j-h]

As an example of the kind of use cases this would allow:

Say you want to use llm to summarize markdown files and rename them based on a summary or a few keywords. If we could constrain the llm's output to e.g. [a-z] and - we could immediate pipe llm's output into a rename workflow. Or better: constrain the llm's output to json with properties keyword and summary and pipe it through jq to move a file into its keyword directory and rename it based on a summary.

Workflows on the cli similar to the one above would get supercharged and we don't have to prompt and pray that the llm will do the right thing.

[kmad]

I implemented this as a plugin, here: https://github.com/kmad/llm-structure

You can install it using llm install -U llm-structure.

To your point above, it supports STDIN so you can use it in a standard pipeline a la echo "Chase, 53, and 49 year old Mia went to the market." | llm structure - --schema people.yaml -m gpt-4o