The problem AI Data Munging solves

For many years, courts have provided their opinions to the public, but almost always in portable document format ("PDF"). However, PDF is geared to making the opinion readable by humans because of its advanced (visual) formatting capabilities. Unfortunately, what makes PDF documents human-readable also makes those same documents difficult for machine learning and natural language processing. See https://unstract.com/blog/pdf-hell-and-practical-rag-applications/and https://stackoverflow.com/questions/51035220/why-is-it-so-hard-to-convert-pdf-to-plain-textfor but a few examples. To compound the problem, courts (as well as legal journals and legal treatises) often make extensive use of footnotes, some of which are so large that they span more than one page. Footnotes are particularly difficult to extract into the chunk of the document where they are relevant, which degrades the machine learning process.

There is a need, therefore, for a software extraction method for variously-formatted PDF documents that account for footnotes (in their entirety) as well as removing headers/footers/and page numbers so that the document is properly reformatted to maximize the performance of natural language processing libraries and subsequent AI applications. This project is meant to solve this pressing problem in the state of the art.

Challenges I ran into

Coding and Development