Docling-based PDF extraction pipeline

Problem

Zoning PDFs are a worst case for naïve text extraction:

• Multi-column layouts that confuse line-order extraction
• Footnotes that appear in the middle of the main text flow
• Tables with merged cells that break CSV-style parsers
• Section numbering embedded in headers that plain text strips away

PyPDF2 and pdfminer gave garbled output on ~30% of the test corpus. Docling gave usable output on all of them.

Pipeline

MinIO (raw PDFs) → Airflow DAG → Docling → structured chunks → MinIO (JSON) → embedding DAG

The Airflow DAG:

1. Polls MinIO for new PDFs (by modified-time watermark)
2. Passes each PDF to a Docling worker container
3. Docling returns a JSON document with page-level structure, bounding boxes, and section hierarchy
4. A post-processing step normalizes section numbering across jurisdictions (each city has its own numbering scheme)
5. Chunks are sized for the embedding model's context window, with section headers prepended for context

Provenance

Every chunk carries:

• source_url — the original PDF URL
• page — page number
• bbox — bounding box on the page (for the citation verifier to locate the text)
• section_path — e.g. ["4", "4.3", "4.3.2"]

This provenance is what makes the citation verifier in the RAG pipeline possible.

Performance

~40 PDFs/hour on the VPS (single Docling worker). Not fast, but the pipeline runs nightly and the backlog is manageable. A second worker would double throughput but the VPS RAM is the constraint.