ESG Report Analyzer by Multimodal AI Agent
An automated ESG analytics AI Agent powered by OpenAI's o1 multimodal model, delivering superior, unbiased sustainability evaluations beyond human capability.
Created on 12th April 2025
•
ESG Report Analyzer by Multimodal AI Agent
An automated ESG analytics AI Agent powered by OpenAI's o1 multimodal model, delivering superior, unbiased sustainability evaluations beyond human capability.
The problem ESG Report Analyzer by Multimodal AI Agent solves
Environmental, Social, and Governance (ESG) reports are crucial for evaluating corporate sustainability and social responsibility. However, these documents often contain dense, multi-format information making extraction, analysis, and interpretation challenging and resource-intensive.
This project addresses the complexity by implementing a Retrieval-Augmented Generation (RAG) pipeline using OpenAI’s multimodal model, o1. It automates the extraction and interpretation of critical ESG data from PDF reports by converting pages into images, summarizing content, and generating meaningful insights through advanced semantic understanding.
The key innovation lies in effectively combining keyword-based and semantic search methods, refining results through reranking with Cohere’s model, and validating relevance visually with GPT-4o. This approach significantly reduces manual effort, increases accuracy, and provides rapid, scalable insights into corporate sustainability activities, making ESG data more actionable and accessible.
Challenges I ran into
Multimodal Data Complexity:
Converting infographic PDF reports into embedding vector and accurately interpreting dense graphical and textual data required significant fine-tuning of the image-to-text summarization pipeline. I overcame this by iteratively adjusting the prompts and evaluating summaries through multiple rounds of feedback and testing.
Optimizing Retrieval Accuracy:
Balancing keyword-based search with semantic search initially produced inconsistent results. Implementing Cohere's reranker significantly improved accuracy but introduced additional latency. I resolved this by optimizing retrieval parameters and fine-tuning the reranking model thresholds, achieving a good balance between speed and relevance.
Relevance Assessment:
Evaluating relevance visually through GPT-4o occasionally led to misinterpretations due to ambiguous visual contexts. Addressing this involved enhancing the visual context provided to GPT-4o and incorporating additional context-checking mechanisms to improve consistency and reliability in relevance assessments.
Tracks Applied (1)