Era of Multimodal Monetary Studies by way of Llamaindex

PHOENIX_API_KEY = ""
os.environ("OTEL_EXPORTER_OTLP_HEADERS") = f"api_key={PHOENIX_API_KEY}"
llama_index.core.set_global_handler(
    "arize_phoenix", endpoint="https://llamatrace.com/v1/traces"
)

Step 3 – Add Knowledge – Get Your Slide Deck

As an indication, we used the slideshow from ConocoPhillips’ 2023 investor assembly. We obtain the PDF:

import os
import requests

# Create the directories (ignore errors in the event that they exist already)
os.makedirs("information", exist_ok=True)
os.makedirs("data_images", exist_ok=True)

# URL of the PDF
url = "https://static.conocophillips.com/recordsdata/2023-conocophillips-aim-presentation.pdf"

# Obtain and save to information/conocophillips.pdf
response = requests.get(url)
with open("information/conocophillips.pdf", "wb") as f:
    f.write(response.content material)

print("PDF downloaded to information/conocophillips.pdf")

Test if the PDF slide deck is within the information folder; in any other case, place it within the information folder and identify it no matter you need.

Step 4: configure fashions

You want an integration mannequin and an LLM. On this instance:

from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
embed_model = OpenAIEmbedding(mannequin="text-embedding-3-large")
llm = OpenAI(mannequin="gpt-4o")

Subsequent, register them as default for LlamaIndex:

from llama_index.core import Settings
Settings.embed_model = embed_model
Settings.llm = llm

Step 5: Parse the doc with LlamaParse

LlamaParse can extract textual content and pictures (through a big multi-modal mannequin). For every PDF web page, it returns:

• Gross sales textual content (with tables, titles, bullets, and so forth.)
• A rendered picture (saved domestically)

print(f"Parsing slide deck...")
md_json_objs = parser.get_json_result("information/conocophillips.pdf")
md_json_list = md_json_objs(0)("pages")

print(md_json_list(10)("md"))

print(md_json_list(1).keys())

image_dicts = parser.get_images(md_json_objs, download_path="data_images")

Step 6: Affiliate textual content and pictures

We create a listing of textual content node objects (LlamaIndex information construction) for every web page. Every node has metadata concerning the web page quantity and the trail of the corresponding picture file:

from llama_index.core.schema import TextNode
from typing import Non-compulsory

# get pages loaded by way of llamaparse
import re


def get_page_number(file_name):
    match = re.search(r"-page-(d+).jpg$", str(file_name))
    if match:
        return int(match.group(1))
    return 0


def _get_sorted_image_files(image_dir):
    """Get picture recordsdata sorted by web page."""
    raw_files = (f for f in checklist(Path(image_dir).iterdir()) if f.is_file())
    sorted_files = sorted(raw_files, key=get_page_number)
    return sorted_files
    
from copy import deepcopy
from pathlib import Path


# connect picture metadata to the textual content nodes
def get_text_nodes(json_dicts, image_dir=None):
    """Break up docs into nodes, by separator."""
    nodes = ()

    image_files = _get_sorted_image_files(image_dir) if image_dir just isn't None else None
    md_texts = (d("md") for d in json_dicts)

    for idx, md_text in enumerate(md_texts):
        chunk_metadata = {"page_num": idx + 1}
        if image_files just isn't None:
            image_file = image_files(idx)
            chunk_metadata("image_path") = str(image_file)
        chunk_metadata("parsed_text_markdown") = md_text
        node = TextNode(
            textual content="",
            metadata=chunk_metadata,
        )
        nodes.append(node)

    return nodes
    
# this can cut up into pages
text_nodes = get_text_nodes(md_json_list, image_dir="data_images")

print(text_nodes(10).get_content(metadata_mode="all"))

Step 7: Create a abstract index

With these textual content nodes in hand, you may create a SummaryIndex:

import os
from llama_index.core import (
    StorageContext,
    SummaryIndex,
    load_index_from_storage,
)

if not os.path.exists("storage_nodes_summary"):
    index = SummaryIndex(text_nodes)
    # save index to disk
    index.set_index_id("summary_index")
    index.storage_context.persist("./storage_nodes_summary")
else:
    # rebuild storage context
    storage_context = StorageContext.from_defaults(persist_dir="storage_nodes_summary")
    # load index
    index = load_index_from_storage(storage_context, index_id="summary_index")

SummaryIndex ensures that you could simply retrieve or generate high-level summaries of your complete doc.

Step 8: Outline a structured output scheme

Our course of goals to supply a last consequence with interlocking blocks of textual content and blocks of pictures. For that, we create a customized Pydantic mannequin (utilizing Pydantic v2 or guaranteeing compatibility) with two kinds of blocks:textual content block and Picture block—and a parental mannequin ReportOutput:

from llama_index.llms.openai import OpenAI
from pydantic import BaseModel, Subject
from typing import Listing
from IPython.show import show, Markdown, Picture
from typing import Union


class TextBlock(BaseModel):
    """Textual content block."""

    textual content: str = Subject(..., description="The textual content for this block.")


class ImageBlock(BaseModel):
    """Picture block."""

    file_path: str = Subject(..., description="File path to the picture.")


class ReportOutput(BaseModel):
    """Knowledge mannequin for a report.

    Can include a mixture of textual content and picture blocks. MUST include at the very least one picture block.

    """

    blocks: Listing(Union(TextBlock, ImageBlock)) = Subject(
        ..., description="A listing of textual content and picture blocks."
    )

    def render(self) -> None:
        """Render as HTML on the web page."""
        for b in self.blocks:
            if isinstance(b, TextBlock):
                show(Markdown(b.textual content))
            else:
                show(Picture(filename=b.file_path))


system_prompt = """
You're a report era assistant tasked with producing a well-formatted context given parsed context.

You can be given context from a number of experiences that take the type of parsed textual content.

You're chargeable for producing a report with interleaving textual content and pictures - within the format of interleaving textual content and "picture" blocks.
Since you can not instantly produce a picture, the picture block takes in a file path - you must write within the file path of the picture as a substitute.

How are you aware which picture to generate? Every context chunk will include metadata together with a picture render of the supply chunk, given as a file path. 
Embody ONLY the photographs from the chunks which have heavy visible parts (you may get a touch of this if the parsed textual content incorporates numerous tables).
You MUST embody at the very least one picture block within the output.

You MUST output your response as a software name as a way to adhere to the required output format. Do NOT give again regular textual content.

"""


llm = OpenAI(mannequin="gpt-4o", api_key="OpenAI_API_KEY", system_prompt=system_prompt)
sllm = llm.as_structured_llm(output_cls=ReportOutput)

The important thing level: ReportOutput requires at the very least one block of pictures, which ensures that the ultimate response is multimodal.

Step 9 – Create a structured question engine

LlamaIndex means that you can use a “structured LLM” (that’s, an LLM whose output is routinely parsed into a particular schema). Here is how:

query_engine = index.as_query_engine(
    similarity_top_k=10,
    llm=sllm,
    # response_mode="tree_summarize"
    response_mode="compact",
)

response = query_engine.question(
    "Give me a abstract of the monetary efficiency of the Alaska/Worldwide phase vs. the decrease 48 phase"
)

response.response.render()

# Output
The monetary efficiency of ConocoPhillips' Alaska/Worldwide phase and the Decrease 48 phase may be in contrast based mostly on a number of key metrics comparable to capital expenditure, manufacturing, and free money move over the following decade.

Alaska/Worldwide Phase
Capital Expenditure: The Alaska/Worldwide phase is projected to have capital expenditures of $3.7 billion in 2023, averaging $4.4 billion from 2024 to 2028, and $3.0 billion from 2029 to 2032.
Manufacturing: Manufacturing is anticipated to be round 750 MBOED in 2023, growing to a mean of 870 MBOED from 2024 to 2028, and reaching 1080 MBOED from 2029 to 2032.
Free Money Stream (FCF): The phase is anticipated to generate $5.5 billion in FCF in 2023, with a mean of $6.5 billion from 2024 to 2028, and $15.0 billion from 2029 to 2032.
Decrease 48 Phase
Capital Expenditure: The Decrease 48 phase is anticipated to have capital expenditures of $6.3 billion in 2023, averaging $6.5 billion from 2024 to 2028, and $8.0 billion from 2029 to 2032.
Manufacturing: Manufacturing is projected to be roughly 1050 MBOED in 2023, growing to a mean of 1200 MBOED from 2024 to 2028, and reaching 1500 MBOED from 2029 to 2032.
Free Money Stream (FCF): The phase is anticipated to generate $7 billion in FCF in 2023, with a mean of $8.5 billion from 2024 to 2028, and $13 billion from 2029 to 2032.
Total, the Decrease 48 phase reveals increased capital expenditure and manufacturing ranges in comparison with the Alaska/Worldwide phase, however each segments are projected to generate important free money move over the following decade.

# Making an attempt one other question
response = query_engine.question(
    "Give me a abstract of whether or not you suppose the monetary projections are secure, and if not, what are the potential threat components. "
    "Assist your analysis with sources."
)

response.response.render()

Conclusion

By combining LlamaIndex, LlamaParse, and OpenAI, you may create a multimodal report generator that processes a whole PDF (with textual content, tables, and pictures) into structured output. This strategy delivers richer, extra visually informative outcomes—precisely what stakeholders must glean important insights from advanced company or technical paperwork.

Be happy to adapt this pipeline to your individual paperwork, add a restoration step for giant recordsdata, or combine domain-specific fashions to research the underlying pictures. With the muse laid out right here, you may create dynamic, interactive, and visually wealthy experiences that go far past easy text-based queries.

Many due to Jerry Liu from LlamaIndex for growing this wonderful channel.

Key takeaways

• Rework PDF recordsdata with textual content and pictures into structured codecs whereas preserving the integrity of the unique content material utilizing LlamaParse and LlamaIndex.
• Generate visually wealthy experiences that interweave textual summaries and pictures for higher contextual understanding.
• Monetary reporting may be improved by integrating textual content and visuals for extra insightful and dynamic outcomes.
• Leveraging LlamaIndex and LlamaParse streamlines the monetary reporting course of, guaranteeing correct and structured outcomes.
• Retrieve related paperwork earlier than processing to optimize reporting for giant recordsdata.
• Enhance visible analytics, incorporate graph-specific analytics, and mix fashions for textual content and picture processing to realize deeper insights.

Often requested questions

The media proven on this article just isn’t the property of Analytics Vidhya and is used on the writer’s discretion.

Howdy! I’m Adarsh, a Enterprise Analytics graduate from ISB, presently immersed in analysis and exploring new frontiers. I’m enthusiastic about information science, synthetic intelligence, and all of the modern methods they’ll remodel industries. Whether or not it is constructing fashions, engaged on information pipelines, or diving into machine studying, I like experimenting with the most recent expertise. AI is not only my curiosity, it is the place I see the long run going, and I am all the time excited to be part of that journey!