PDF parsing

[iamrobzy]

The key component in the resource ingestion pipeline is PDF parsing. There are plenty of libraries one could use to implement this including:

• Unstructured
• LlamaParse by LLamaIndex
• Marker

This is an exploratory issue to kick-off investigations into the different libraries. Feel free to create sub-issues exploring specific libraries. Be mindful of the specific JSON fields and metadata we may want include (corresponding to Twiga database schemas). Specific metadata include page numbers, chapter numbers, etc. Intelligent chunking strategies is also a very relevant topic to be explored.

Helpful resources & inspiration:

• Victor's Wikipedia chunker
• Nvidia's multimodal PDF data extractor

Sample PDFs:
I recommend testing your PDF parser with TIE resources, e.g. the Geography Form Two textbook. For faster development, you may want to split the PDF to around ~10-20 pages.