Wikiparse

Wikiparse converts raw Wikipedia XML dumps into structured, per-article Markdown with YAML frontmatter, including a citation parser built on optimized regular expressions. It is the data pipeline that turns a multi-gigabyte dump into a queryable corpus ready for downstream search and analysis.

The problem

Wikipedia's official dumps are enormous XML blobs with wikitext markup, template expansions, and citation templates that resist straightforward extraction. Converting them into clean, structured documents suitable for search, analysis, or static publishing meant parsing wikitext — a grammar with no formal specification — at dump scale. Existing tools either produced garbled output or couldn't run on the full corpus without running out of time or memory.

The approach

The pipeline reads XML dumps incrementally and extracts per-article wikitext, then applies a multi-stage translation pass: template expansion, structural section extraction, and a citation parser built on hand-tuned regular expressions optimized for the patterns that dominate Wikipedia's citation templates. Output is Markdown with YAML frontmatter — version, title, categories, and metadata — so every downstream consumer gets a consistent, typed document. The 167-test suite covers the parser's long tail of edge cases, and CI gates every change.

What it demonstrates

A 167-test, fully typed parser for an underspecified grammar at dump scale demonstrates the kind of investment that only pays off when the pipeline has to be right every time — a single missed citation or broken template silently corrupts the corpus. Maintaining that rigor through version 0.7.0 with CI and pre-commit hooks shows the discipline for infrastructure that other tools depend on.