Traditional Data Ingestion Systems in Data centric companies and organizations that have multiple data sources rely on custom ingestion pipelines for each source. If a new data source is to be added, they create a separate pipeline for that source. This process is not scalable. As the number of data source increases, analysing their schema and creating an automation script to parse it to their DB acceptable schema becomes repetitive.

This tedious task can be automated using Machine Learning and Generative AI. If trained over our schema and the data points required, it can learn to parse the data points from any data source which is the ultimate aim of this project.