interface for interacting with LLMs is thru the traditional chat UI present in ChatGPT, Gemini, or DeepSeek. The interface is kind of easy, the place the consumer inputs a physique of textual content and the mannequin responds with one other physique, which can or might not comply with a selected construction. Since people can perceive unstructured pure language, this interface is appropriate and fairly efficient for the audience it was designed for.
Nonetheless, the consumer base of LLMs is way bigger than the 8 billion people dwelling on Earth. It expands to tens of millions of software program applications that may doubtlessly harness the ability of such massive generative fashions. In contrast to people, software program applications can not perceive unstructured knowledge, stopping them from exploiting the information generated by these neural networks.
To deal with this problem, varied strategies have been developed to generate outputs from LLMs following a predefined schema. This text will overview three of the most well-liked approaches for producing structured outputs from LLMs. It’s written for engineers curious about integrating LLMs into their software program purposes.
Structured Output Era
Structured output technology from LLMs entails utilizing these fashions to provide knowledge that adheres to a predefined schema, moderately than producing unstructured textual content. The schema will be outlined in varied codecs, with JSON and regex being the commonest. For instance, when using JSON format, the schema specifies the anticipated keys and the information varieties (reminiscent of int, string, float, and many others.) for every worth. The LLM then outputs a JSON object that features solely the outlined keys and appropriately formatted values.
There are numerous conditions the place structured output is required from LLMs. Formatting unstructured our bodies of textual content is one massive software space of this know-how. You should utilize a mannequin to extract particular data from massive our bodies of textual content and even pictures (utilizing VLMs). For instance, you need to use a normal VLM to extract the acquisition date, whole value, and retailer identify from receipts.
There are numerous strategies to generate structured outputs from LLMs. This text will focus on three.
- Counting on API Suppliers
- Prompting and Reprompting Methods
- Constrained Decoding
Counting on API Suppliers ‘Magic’
A number of LLM service API suppliers, together with OpenAI and Google’s Gemini, enable customers to outline a schema for the mannequin’s output. This schema is often outlined utilizing a Pydantic class and supplied to the API endpoint. In case you are utilizing LangChain, you possibly can comply with this tutorial to combine structured outputs into your software.
Simplicity is the best facet of this explicit method. You outline the required schema in a way acquainted to you, cross it to the API supplier, and sit again and calm down because the service supplier performs all of the magic for you.
Utilizing this system, nevertheless, will restrict you to utilizing solely API suppliers that present the described service. This limits the expansion and adaptability of your tasks, because it shuts the door to utilizing a number of fashions, significantly open supply ones. If the API suppliers instantly determine to spike the value of the service, you may be pressured both to simply accept the additional prices or search for one other supplier.
Furthermore, it isn’t precisely Hogwarts Magic that the service supplier does. The supplier follows a sure method to generate the structured output for you. Information of the underlying know-how will facilitate the app growth and speed up the debugging course of and error understanding. For the talked about causes, greedy the underlying science might be definitely worth the effort.
Prompting and Reprompting-Primarily based Strategies
If in case you have chatted with an LLM earlier than, then this system might be in your thoughts. In order for you a mannequin to comply with a sure construction, simply inform it to take action! Within the system immediate, instruct the mannequin to comply with a sure construction, present a couple of examples, and ask it to not add any extra textual content or description.
After the mannequin responds to the consumer request and the system receives the output, you must use a parser to rework the sequence of bytes to an acceptable illustration within the system. If parsing succeeds, then congratulate your self and thank the ability of immediate engineering. If parsing fails, then your system should recuperate from the error.
Prompting is Not Sufficient
The issue with prompting is unreliability. By itself, prompting isn’t sufficient to belief a mannequin to comply with a required construction. It would add further rationalization, disregard sure fields, and use an incorrect knowledge sort. Prompting will be and needs to be coupled with error restoration strategies that deal with the case the place the mannequin defies the schema, which is detected by parsing failure.
Some individuals may suppose {that a} parser acts like a boolean operate. It takes a string as enter, checks its adherence to predefined grammar guidelines, and returns a easy ‘sure’ or ‘no’ reply. In actuality, parsers are extra complicated than that and supply a lot richer data than ‘follows’ or ‘doesn’t comply with’ construction.
Parsers can detect errors and incorrect tokens in enter textual content in keeping with grammar guidelines (Aho et al. 2007, 192–96). This data offers us with useful data on the specifics of misalignments within the enter string. For instance, the parser is what detects a lacking semicolon error if you’re working Java code.
Figure 1 depicts the stream used within the prompting-based strategies.
Prompting Instruments
Probably the most standard libraries for immediate primarily based structured output technology from LLMs is instructor. Teacher is a Python library with over 11k stars on GitHub. It helps knowledge definition with Pydantic, integrates with over 15 suppliers, and offers computerized retries on parsing failure. Along with Python, the package deal can be avillable in TypeScript, Go, Ruby, and Rust (2).
The fantastic thing about Teacher lies in its simplicity. All you want is to outline a Pydantic class, initialize a shopper utilizing solely its identify and API key (if required), and cross your request. The pattern code under, from the docs, shows the simplicity of Teacher.
import teacher
from pydantic import BaseModel
from openai import OpenAI
class Particular person(BaseModel):
identify: str
age: int
occupation: str
shopper = teacher.from_openai(OpenAI())
particular person = shopper.chat.completions.create(
mannequin="gpt-4o-mini",
response_model=Particular person,
messages=[
{
"role": "user",
"content": "Extract: John is a 30-year-old software engineer"
}
],
)
print(particular person) # Particular person(identify='John', age=30, occupation='software program engineer')The Price of Reprompting
As handy because the reprompting approach is perhaps, it comes at a hefty value. LLM utilization value, both service supplier API prices or GPU utilization, scales linearly with the variety of enter tokens and the variety of generated tokens.
As talked about earlier prompting primarily based strategies may require reprompting. The reprompt can have roughly the identical value as the unique one. Therefore, the associated fee scales linearly with the variety of reprompts.
For those who’re going to make use of this system, it’s a must to maintain the associated fee downside in thoughts. Nobody desires to be stunned by a big invoice from an API supplier. One thought to assist reduce shocking prices is to place emergency brakes into the system by making use of a hard-coded restrict on the variety of allowed reprompts. This may make it easier to put an higher restrict on the prices of a single immediate and reprompt cycle.
Constrained Decoding
In contrast to the prompting, constrained decoding doesn’t want retries to generate a legitimate, structure-following output. Constrained decoding makes use of computational linguistics strategies and information of the token technology course of in LLMs to generate outputs which might be assured to comply with the required schema.
How It Works?
LLMs are autoregressive models. They generate one token at a time and the generated tokens are used as inputs to the same model.
The last layer of an LLM is basically a logistic regression model that calculates for each token in the model’s vocabulary the probability of it following the input sequence. The model calculates the logits value for each token, then using the softmax function, these value are scaled and transformed to probability values.
Constrained decoding produces structured outputs by limiting the available tokens at each generation step. The tokens are picked so that the final output obeys the required structure. To figure out how the set of possible next tokens can be determined, we need to visit RegEx.
Regular expressions, RegEx, are used to define specific patterns of text. They are used to check if a sequence of text matches an expected structure or schema. So basically, RegEx is a language that can be used to define expected structures from LLMs. Because of its popularity, there is a wide array of tools and libraries that transforms other forms of data structure definition like Pydantic classes and JSON to RegEx. Because of its flexibility and the wide availability of conversion tools, we can transform our goal now and focus on using LLMs to generate outputs following a RegEx pattern.
Deterministic Finite Automata (DFA)
One of the ways a RegEx pattern can be compiled and tested against a body of text is by transforming the pattern into a deterministic finite automata (DFA). A DFA is simply a state machine that is used to check if a string follows a certain structure or pattern.
A DFA consists of 5 components.
- A set of tokens (called the alphabet of the DFA)
- A set of states
- A set of transitions. Each transition connects two states (maybe connecting a state with itself) and is annotated with a token from the alphabet
- A start state (marked with an input arrow)
- One or more final states (marked as double circles)
A string is a sequence of tokens. To test a string against the pattern defined by a DFA, you begin at the start state and loop over the string’s tokens, taking the transition corresponding to the token at each move. If at any point you have a token for which no corresponding transition exists from the current state, parsing fails and the string defies the schema. If parsing ends at one of the final states, then the string matches the pattern; otherwise it also fails.
{a, b}, states {q0, q1, q2}, and a single final state, q2. Generated using Grpahviz by the Creator.For instance, the string abab matches the sample in Figure 2 as a result of beginning at q0 and following the transitions marked with a, b, a, and b on this order will land us at q2, which is a last state.
However, the string abba doesn’t match the sample as a result of its path ends at q0 which isn’t a last state.
A beauty of RegEx is that it may be compiled right into a DFA; in any case, they’re simply two other ways to specify patterns. Dialogue of such a change is out of scope for this text. The reader can test Aho et al. (2007, 152–66) for a dialogue of two strategies to carry out the transformation.
DFA for Legitimate Subsequent Tokens Set
a(b|c)*d. Generated utilizing Grpahviz by the Creator.Let’s recap what now we have reached to this point. We wished a method to determine the set of legitimate subsequent tokens to comply with a sure schema. We outlined the schema utilizing RegEx and remodeled it right into a DFA. Now we’re going to present {that a} DFA informs us of the set of attainable tokens at any level throughout parsing, becoming our necessities and wishes.
After constructing the DFA, we are able to simply decide in O(1) the set of legitimate subsequent tokens whereas standing at any state. It’s the set of tokens annotating any transition exiting from the present state.
Think about the DFA in Figure 3, for instance. The next desk exhibits the set of legitimate subsequent tokens for every state.
| State | Legitimate Subsequent Tokens |
|---|---|
q0 |
{a} |
q1 |
{b, c, d} |
q2 |
{} |
Making use of the DFA to LLMs
Getting again to our structured output from LLMs downside, we are able to rework our schema to a RegEx then to a DFA. The alphabet of this DFA might be set to the LLM’s vocabulary (the set of all tokens the mannequin can generate). Whereas the mannequin generates tokens, we’ll transfer by the DFA, beginning at the beginning state. At every step, we will decide the set of legitimate subsequent tokens.
The trick now occurs on the softmax scaling stage. By zeroing out the logits of all tokens that aren’t within the legitimate tokens set, we’ll calculate possibilities just for legitimate tokens, forcing the mannequin to generate a sequence of tokens that follows the schema. That approach, we are able to generate structured outputs with zero extra prices!
Constrained Decoding Instruments
One of the most popular Python libraries for constrained decoding is Outlines (Willard and Louf 2023). It is vitally easy to make use of and integrates with many LLM suppliers like OpenAI, Anthropic, Ollama, and vLLM.
You’ll be able to outline the schema utilizing a Pydantic class, for which the library handles the RegEx transformation, or instantly utilizing a RegEx sample.
from pydantic import BaseModel
from typing import Literal
import outlines
import openai
class Buyer(BaseModel):
identify: str
urgency: Literal["high", "medium", "low"]
problem: str
shopper = openai.OpenAI()
mannequin = outlines.from_openai(shopper, "gpt-4o")
buyer = mannequin(
"Alice wants assist with login points ASAP",
Buyer
)
# ✓ At all times returns legitimate Buyer object
# ✓ No parsing, no errors, no retriesThe code snippet above from the docs shows the simplicity of utilizing Outlines. For extra data on the library, you possibly can test the docs and the dottxt blogs.
Conclusion
Structured output technology from LLMs is a robust software that expands the attainable use instances of LLMs past the easy human chat. This text mentioned three approaches: counting on API suppliers, prompting and reprompting methods, and constrained decoding. For many situations, constrained decoding is the favoured technique due to its flexibility and low value. Furthermore, the existence of standard libraries like Outlines simplifies the introduction of constrained decoding to software program tasks.
If you wish to study extra about constrained decoding, then I’d extremely advocate this course from deeplearning.ai and dottxt, the creators of Outlines library. Utilizing movies and code examples, this course will make it easier to get hands-on expertise getting structured outputs from LLMs utilizing the strategies mentioned on this publish.
References
[1] Aho, Alfred V., Monica S. Lam, Ravi Sethi, and Jeffrey D. Ullman, Compilers: Principles, Techniques, & Tools (2007), Pearson/Addison Wesley
[2] Willard, Brandon T., and Rémi Louf, Efficient Guided Generation for Large Language Models (2023), https://arxiv.org/abs/2307.09702.
