Wikiparse
Turns Wikipedia XML dumps into clean Markdown with structured citations.
Wikiparse converts raw wikitext into Markdown with YAML frontmatter and structured citation records. It's a from-scratch parser built on composed regular expressions (no MediaWiki API), and anything it can't handle gets recorded in an explicit failure census instead of silently disappearing.
$ cat coffee.wiki '''Coffee''' is a [[drink]] prepared from roasted [[coffee bean]]s.<ref>{{cite web |url=https://example.org/coffee-history |title=All About Coffee}}</ref> == History == Legend attributes its discovery to an Ethiopian goatherd. $ ./run.sh --- tally: 0 0 1 1 0 4 0 0 0 0 1 0 0 0 0 0 aliases: [] internal_links: coffee-bean: 1 drink: 1 external_links: {} mentions: [] peers: {} images: [] citations: web: - site-exampleorg__0000__all-about-coffee failed_citations: [] --- **Coffee** is a drink prepared from roasted coffee beans. ## `0` History Legend attributes its discovery to an Ethiopian goatherd.
Wikitext is an underappreciated, beautiful, massive thing. Citations are intentionally decoupled from their points of reference and aggregated at an article level, which would ruin the actual encyclopedia but works much better for agents: they can reliably infer which citations to investigate in milliseconds, falling back to the live HTML when in doubt. Its (AFAICT?) novel approach to deterministically parsing messy plaintext citations is something I'm proud of.
I've run it over an entire Wikipedia dump: the resulting dataset, 5.1 million articles across ten Parquet tables, is public on Hugging Face as robbd/wikiparse-2606. The parser itself is on PyPI.