One VPS, ten services, one operator.
A self-hosted lakehouse plus glue. Every service here supports real workloads — the zoning-compliance scrapers, the QA dashboard, the RAG chatbot. None of this is a homelab demo. If a service isn't earning its keep, it gets uninstalled.
Architecture
Inventory
01 · Ingestion
Per-jurisdiction Python scrapers (some headless Selenium, some plain HTTP) that pull zoning ordinances and emit raw documents to MinIO.
Why: every jurisdiction is a snowflake. A framework would have been false economy — a bag of small Python modules sharing a common BaseScraper beats one clever abstraction.
Three brokers, single-rack. Topics for ingestion events, extraction events, QA signals.
Why: at this scale Kafka is overkill — but it's the right overkill. Replays during pipeline rewrites are a single command, and the operational vocabulary it imposes keeps the system honest.
02 · Orchestration
Scheduler + 12 DAGs. Owns the heavy ETL: scrape → extract → embed → load to Iceberg.
Why: the DAG model fits batch document processing well, and the operator ecosystem covers everything I need. Considered Dagster; chose Airflow on familiarity.
Low-code workflows for the other half of automation: webhook routing, integrations (Slack, Notion), and short-lived glue jobs that don't deserve a DAG.
Why: some workflows are 90% integrations. n8n keeps those out of Airflow.
Layout-aware PDF extraction. Outputs structured chunks (page, bbox, section) for downstream embedding.
Why: zoning PDFs are a worst-case for naïve text extraction. Docling is the first tool that gave usable output without hand-tuning.
03 · Storage
S3-compatible object storage on the same VPS. Holds raw PDFs and Parquet files written by Airflow.
Why: S3 semantics without an S3 bill, and Iceberg/Trino speak it natively.
Table format on top of Parquet-in-MinIO. Time-travel queries, schema evolution, hidden partitioning.
Why: the killer feature is honest schema evolution. Adding a column to a 2-year-old table is a metadata operation, not a rewrite.
Git-style catalog for Iceberg. Branches and tags on top of table state.
Why: lets me prototype destructive transforms in a branch and merge only after validation. Worth the extra service.
Operational database for the chatbot, plus the qa schema of versioned views that ChatAEC reads.
Why: Postgres is the boring, correct answer. Iceberg doesn't replace it — they live next to each other.
04 · Query
Federated SQL across Iceberg, Postgres, and MinIO. Used by Superset and a handful of ad-hoc analyses.
Why: one engine, three sources, one query language. Reduced the 'where do I run this query?' question to zero.
A REST API auto-generated from the Postgres qa schema. The SQL view is the endpoint.
Why: see the ChatAEC case study. Replaces a warehouse, an ORM, and a hand-written API.
GraphRAG over zoning relationships: overlay districts, parent ordinances, amendment chains. Backed by Postgres + a small graph index.
Why: RAG over text alone misses structural relationships. The graph layer makes those answerable.
05 · Serving
Analyst-facing dashboards over Trino. Used internally for pipeline health and data exploration.
Why: open source, SQL-native, and good enough. Considered Metabase; Superset's deeper SQL surface won.
Reverse proxy and TLS terminator for everything. One Caddyfile, automatic certs.
Why: Nginx config is its own programming language. Caddy isn't.