Automated doc processing is without doubt one of the greatest winners of the ChatGPT revolution, as LLMs are capable of sort out a variety of topics and duties in a zero-shot setting, that means with out in-domain labeled coaching information. This has made constructing AI-powered purposes to course of, parse, and routinely perceive arbitrary paperwork a lot simpler. Although naive approaches utilizing LLMs are nonetheless hindered by non-text context, similar to figures, pictures, and tables, that is what we’ll attempt to tackle on this weblog submit, with a particular deal with PDFs.
At a primary stage, PDFs are only a assortment of characters, pictures, and contours together with their precise coordinates. They don’t have any inherent “textual content” construction and weren’t constructed to be processed as textual content however solely to be considered as is. That is what makes working with them tough, as text-only approaches fail to seize all of the structure and visible parts in a majority of these paperwork, leading to a major lack of context and data.
One technique to bypass this “text-only” limitation is to do heavy pre-processing of the doc by detecting tables, pictures, and structure earlier than feeding them to the LLM. Tables might be parsed to Markdown or JSON, pictures and figures might be represented by their captions, and the textual content might be fed as is. Nonetheless, this strategy requires customized fashions and can nonetheless lead to some lack of info, so can we do higher?
Most up-to-date massive fashions at the moment are multi-modal, that means they will course of a number of modalities like textual content, code, and pictures. This opens the best way to an easier resolution to our downside the place one mannequin does every part without delay. So, as a substitute of captioning pictures and parsing tables, we are able to simply feed the web page as a picture and course of it as is. Our pipeline will be capable of load the PDF, extract every web page as a picture, break up it into chunks (utilizing the LLM), and index every chunk. If a piece is retrieved, then the total web page is included within the LLM context to carry out the duty. In what follows, we’ll element how this may be carried out in observe.
The pipeline we’re implementing is a two-step course of. First, we section every web page into vital chunks and summarize every of them. Second, we index chunks as soon as then search the chunks every time we get a request and embody the total context with every retrieved chunk within the LLM context.
Step 1: Web page Segmentation and Summarization
We extract the pages as pictures and go every of them to the multi-modal LLM to section them. Fashions like Gemini can perceive and course of web page structure simply:
- Tables are recognized as one chunk.
- Figures type one other chunk.
- Textual content blocks are segmented into particular person chunks.
- …
For every ingredient, the LLM generates a abstract than might be embedded and listed right into a vector database.
Step 2: Embedding and Contextual Retrieval
On this tutorial we’ll use textual content embedding just for simplicity however one enchancment could be to make use of imaginative and prescient embeddings instantly.
Every entry within the database contains:
- The abstract of the chunk.
- The web page quantity the place it was discovered.
- A hyperlink to the picture illustration of the total web page for added context.
This schema permits for native stage searches (on the chunk stage) whereas maintaining observe of the context (by linking again to the total web page). For instance, if a search question retrieves an merchandise, the Agent can embody your complete web page picture to supply full structure and additional context to the LLM with a purpose to maximize response high quality.
By offering the total picture, all of the visible cues and essential structure info (like pictures, titles, bullet factors… ) and neighboring gadgets (tables, paragraph, …) can be found to the LLM on the time of producing a response.
We are going to implement every step as a separate, re-usable agent:
The primary agent is for parsing, chunking, and summarization. This entails the segmentation of the doc into vital chunks, adopted by the era of summaries for every of them. This agent solely must be run as soon as per PDF to preprocess the doc.
The second agent manages indexing, search, and retrieval. This contains inserting the embedding of chunks into the vector database for environment friendly search. Indexing is carried out as soon as per doc, whereas searches might be repeated as many occasions as wanted for various queries.
For each brokers, we use Gemini, a multimodal LLM with sturdy imaginative and prescient understanding skills.
Parsing and Chunking Agent
The primary agent is accountable for segmenting every web page into significant chunks and summarizing every of them, following these steps:
Step 1: Extracting PDF Pages as Photos
We use the pdf2image library. The pictures are then encoded in Base64 format to simplify including them to the LLM request.
Right here’s the implementation:
from document_ai_agents.document_utils import extract_images_from_pdf
from document_ai_agents.image_utils import pil_image_to_base64_jpeg
from pathlib import Pathclass DocumentParsingAgent:
@classmethod
def get_images(cls, state):
"""
Extract pages of a PDF as Base64-encoded JPEG pictures.
"""
assert Path(state.document_path).is_file(), "File doesn't exist"
# Extract pictures from PDF
pictures = extract_images_from_pdf(state.document_path)
assert pictures, "No pictures extracted"
# Convert pictures to Base64-encoded JPEG
pages_as_base64_jpeg_images = [pil_image_to_base64_jpeg(x) for x in images]
return {"pages_as_base64_jpeg_images": pages_as_base64_jpeg_images}
extract_images_from_pdf: Extracts every web page of the PDF as a PIL picture.
pil_image_to_base64_jpeg: Converts the picture right into a Base64-encoded JPEG format.
Step 2: Chunking and Summarization
Every picture is then despatched to the LLM for segmentation and summarization. We use structured outputs to make sure we get the predictions within the format we anticipate:
from pydantic import BaseModel, Discipline
from typing import Literal
import json
import google.generativeai as genai
from langchain_core.paperwork import Docclass DetectedLayoutItem(BaseModel):
"""
Schema for every detected structure ingredient on a web page.
"""
element_type: Literal["Table", "Figure", "Image", "Text-block"] = Discipline(
...,
description="Kind of detected merchandise. Examples: Desk, Determine, Picture, Textual content-block."
)
abstract: str = Discipline(..., description="An in depth description of the structure merchandise.")
class LayoutElements(BaseModel):
"""
Schema for the record of structure parts on a web page.
"""
layout_items: record[DetectedLayoutItem] = []
class FindLayoutItemsInput(BaseModel):
"""
Enter schema for processing a single web page.
"""
document_path: str
base64_jpeg: str
page_number: int
class DocumentParsingAgent:
def __init__(self, model_name="gemini-1.5-flash-002"):
"""
Initialize the LLM with the suitable schema.
"""
layout_elements_schema = prepare_schema_for_gemini(LayoutElements)
self.model_name = model_name
self.mannequin = genai.GenerativeModel(
self.model_name,
generation_config={
"response_mime_type": "software/json",
"response_schema": layout_elements_schema,
},
)
def find_layout_items(self, state: FindLayoutItemsInput):
"""
Ship a web page picture to the LLM for segmentation and summarization.
"""
messages = [
f"Find and summarize all the relevant layout elements in this PDF page in the following format: "
f"{LayoutElements.schema_json()}. "
f"Tables should have at least two columns and at least two rows. "
f"The coordinates should overlap with each layout item.",
{"mime_type": "image/jpeg", "data": state.base64_jpeg},
]
# Ship the immediate to the LLM
outcome = self.mannequin.generate_content(messages)
information = json.masses(outcome.textual content)
# Convert the JSON output into paperwork
paperwork = [
Document(
page_content=item["summary"],
metadata={
"page_number": state.page_number,
"element_type": merchandise["element_type"],
"document_path": state.document_path,
},
)
for merchandise in information["layout_items"]
]
return {"paperwork": paperwork}
The LayoutElements schema defines the construction of the output, with every structure merchandise sort (Desk, Determine, … ) and its abstract.
Step 3: Parallel Processing of Pages
Pages are processed in parallel for pace. The next methodology creates an inventory of duties to deal with all of the web page picture without delay for the reason that processing is io-bound:
from langgraph.varieties import Shipclass DocumentParsingAgent:
@classmethod
def continue_to_find_layout_items(cls, state):
"""
Generate duties to course of every web page in parallel.
"""
return [
Send(
"find_layout_items",
FindLayoutItemsInput(
base64_jpeg=base64_jpeg,
page_number=i,
document_path=state.document_path,
),
)
for i, base64_jpeg in enumerate(state.pages_as_base64_jpeg_images)
]
Every web page is shipped to the find_layout_items operate as an unbiased activity.
Full workflow
The agent’s workflow is constructed utilizing a StateGraph, linking the picture extraction and structure detection steps right into a unified pipeline ->
from langgraph.graph import StateGraph, START, ENDclass DocumentParsingAgent:
def build_agent(self):
"""
Construct the agent workflow utilizing a state graph.
"""
builder = StateGraph(DocumentLayoutParsingState)
# Add nodes for picture extraction and structure merchandise detection
builder.add_node("get_images", self.get_images)
builder.add_node("find_layout_items", self.find_layout_items)
# Outline the stream of the graph
builder.add_edge(START, "get_images")
builder.add_conditional_edges("get_images", self.continue_to_find_layout_items)
builder.add_edge("find_layout_items", END)
self.graph = builder.compile()
To run the agent on a pattern PDF we do:
if __name__ == "__main__":
_state = DocumentLayoutParsingState(
document_path="path/to/doc.pdf"
)
agent = DocumentParsingAgent()# Step 1: Extract pictures from PDF
result_images = agent.get_images(_state)
_state.pages_as_base64_jpeg_images = result_images["pages_as_base64_jpeg_images"]
# Step 2: Course of the primary web page (for instance)
result_layout = agent.find_layout_items(
FindLayoutItemsInput(
base64_jpeg=_state.pages_as_base64_jpeg_images[0],
page_number=0,
document_path=_state.document_path,
)
)
# Show the outcomes
for merchandise in result_layout["documents"]:
print(merchandise.page_content)
print(merchandise.metadata["element_type"])
This ends in a parsed, segmented, and summarized illustration of the PDF, which is the enter of the second agent we’ll construct subsequent.
RAG Agent
This second agent handles the indexing and retrieval half. It saves the paperwork of the earlier agent right into a vector database and makes use of the outcome for retrieval. This may be break up into two seprate steps, indexing and retrieval.
Step 1: Indexing the Cut up Doc
Utilizing the summaries generated, we vectorize them and save them in a ChromaDB database:
class DocumentRAGAgent:
def index_documents(self, state: DocumentRAGState):
"""
Index the parsed paperwork into the vector retailer.
"""
assert state.paperwork, "Paperwork ought to have at the least one ingredient"
# Test if the doc is already listed
if self.vector_store.get(the place={"document_path": state.document_path})["ids"]:
logger.information(
"Paperwork for this file are already listed, exiting this node"
)
return # Skip indexing if already achieved
# Add parsed paperwork to the vector retailer
self.vector_store.add_documents(state.paperwork)
logger.information(f"Listed {len(state.paperwork)} paperwork for {state.document_path}")
The index_documents methodology embeds the chunk summaries into the vector retailer. We hold metadata such because the doc path and web page quantity for later use.
Step 2: Dealing with Questions
When a person asks a query, the agent searches for probably the most related chunks within the vector retailer. It retrieves the summaries and corresponding web page pictures for contextual understanding.
class DocumentRAGAgent:
def answer_question(self, state: DocumentRAGState):
"""
Retrieve related chunks and generate a response to the person's query.
"""
# Retrieve the top-k related paperwork primarily based on the question
relevant_documents: record[Document] = self.retriever.invoke(state.query)# Retrieve corresponding web page pictures (keep away from duplicates)
pictures = record(
set(
[
state.pages_as_base64_jpeg_images[doc.metadata["page_number"]]
for doc in relevant_documents
]
)
)
logger.information(f"Responding to query: {state.query}")
# Assemble the immediate: Mix pictures, related summaries, and the query
messages = (
[{"mime_type": "image/jpeg", "data": base64_jpeg} for base64_jpeg in images]
+ [doc.page_content for doc in relevant_documents]
+ [
f"Answer this question using the context images and text elements only: {state.question}",
]
)
# Generate the response utilizing the LLM
response = self.mannequin.generate_content(messages)
return {"response": response.textual content, "relevant_documents": relevant_documents}
The retriever queries the vector retailer to search out the chunks most related to the person’s query. We then construct the context for the LLM (Gemini), which mixes textual content chunks and pictures with a purpose to generate a response.
The complete agent Workflow
The agent workflow has two phases, an indexing stage and a query answering stage:
class DocumentRAGAgent:
def build_agent(self):
"""
Construct the RAG agent workflow.
"""
builder = StateGraph(DocumentRAGState)
# Add nodes for indexing and answering questions
builder.add_node("index_documents", self.index_documents)
builder.add_node("answer_question", self.answer_question)
# Outline the workflow
builder.add_edge(START, "index_documents")
builder.add_edge("index_documents", "answer_question")
builder.add_edge("answer_question", END)
self.graph = builder.compile()
Instance run
if __name__ == "__main__":
from pathlib import Path# Import the primary agent to parse the doc
from document_ai_agents.document_parsing_agent import (
DocumentLayoutParsingState,
DocumentParsingAgent,
)
# Step 1: Parse the doc utilizing the primary agent
state1 = DocumentLayoutParsingState(
document_path=str(Path(__file__).mother and father[1] / "information" / "docs.pdf")
)
agent1 = DocumentParsingAgent()
result1 = agent1.graph.invoke(state1)
# Step 2: Arrange the second agent for retrieval and answering
state2 = DocumentRAGState(
query="Who was acknowledged on this paper?",
document_path=str(Path(__file__).mother and father[1] / "information" / "docs.pdf"),
pages_as_base64_jpeg_images=result1["pages_as_base64_jpeg_images"],
paperwork=result1["documents"],
)
agent2 = DocumentRAGAgent()
# Index the paperwork
agent2.graph.invoke(state2)
# Reply the primary query
result2 = agent2.graph.invoke(state2)
print(result2["response"])
# Reply a second query
state3 = DocumentRAGState(
query="What's the macro common when fine-tuning on PubLayNet utilizing M-RCNN?",
document_path=str(Path(__file__).mother and father[1] / "information" / "docs.pdf"),
pages_as_base64_jpeg_images=result1["pages_as_base64_jpeg_images"],
paperwork=result1["documents"],
)
result3 = agent2.graph.invoke(state3)
print(result3["response"])
With this implementation, the pipeline is full for doc processing, retrieval, and query answering.
Let’s stroll by means of a sensible instance utilizing the doc LLM & Adaptation.pdf , a set of 39 slides containing textual content, equations, and figures (CC BY 4.0).
Step 1: Parsing and summarizing the Doc (Agent 1)
- Execution Time: Parsing the 39-page doc took 29 seconds.
- Outcome: Agent 1 produces an listed doc consisting of chunk summaries and base64-encoded JPEG pictures of every web page.
Step 2: Questioning the Doc (Agent 2)
We ask the next query:
“Clarify LoRA, give the related equations”
Outcome:
Retrieved pages:
