China Judgments Online Preservation Program/Ingestion pipeline

This page documents the technical ingestion pipeline used by the China Judgments Online Preservation Program.

The implementation is open-sourced at prc-court-docs-importer-html. It converts an offline dump of CJO HTML pages into zhwikisource-compatible wikitext, then uploads the converted pages using bot credentials and conservative throttling.

Pipeline overview

The pipeline is intentionally split into two phases.

Phase 1 (Conversion) runs offline. It parses HTML from the archival dataset and produces a converted JSONL file where each record includes a complete, ready-to-upload wikitext payload.

Phase 2 (Upload) reads the converted JSONL and performs page creations/edits on Chinese Wikisource via the MediaWiki API (through Pywikibot), with rate limiting, maxlag, and structured logging.

Keeping conversion and upload separate makes it possible to review output, sample for issues, and rerun conversion without touching the wiki.

Input data format (archival dataset)

The converter reads JSONL (JSON Lines) or pretty-printed JSON files containing court-document records. The following fields are used:

s1: document title
s2: court name
s7: document number (文书号/案号)
s22: hierarchy string (court + document type + document number)
qwContent: HTML content
wsKey: unique identifier (a.k.a. docId from CJO)

The program does not fetch from CJO during conversion or upload. It operates on the archival dataset only.

Phase 1: Conversion (offline)

The conversion process reads the input stream and writes three outputs:

converted JSONL (successful conversions)
failed JSONL (records that could not be converted, with error details)
checkpoint JSON (progress tracking for interruption/resume)

The conversion step is streaming and resumable; it is designed to handle large JSONL files without loading them into memory, ideal for a bot running on Toolforge.

HTML parsing and text cleanup

The converter parses HTML into a structure that can be emitted as wikitext. After basic HTML extraction, it applies several normalizations that matter for zhwikisource quality:

CJK text cleanup (removing spurious spaces and common OCR artifacts)
signature formatting (judge/clerk titles and names spaced according to zhwikisource conventions)
location inference (deriving a geographic location field from the court name)

Filtering and limiting

Conversion can be scoped to specific document types using title filters, for example converting only 判决书 (judgments). Multiple filters can be combined with semicolons. The converter also supports limiting conversions to the first N successful matches for sampling and review.

Human-readable review output

For review workflows, the converter can also emit a plain TXT file containing formatted wikitext for a sample of documents, so that editors can inspect output without parsing JSONL.

Conversion output format

Each converted record is written as a JSON object that includes at least:

title
court
doc_type
doc_number (文书号/案号)
doc_id (docId from CJO)
wikitext (complete wikitext content, with properly filled {{header/裁判文书}})

The intention is that wikitext is directly uploadable with no additional transformations at upload time.

Phase 2: Upload (on-wiki)

The uploader reads the converted JSONL and uploads each record to Chinese Wikisource using bot credentials provided via BotPasswords.

The uploader produces separate logs for successful uploads, failed uploads, skipped pages, and pages that are considered overwritable (i.e., cases where a manual overwrite might be appropriate).

Authentication

Credentials are stored in a local .env file, using the BotPasswords mechanism:

MW_BOT_USERNAME
MW_BOT_PASSWORD

Throttling and server load safeguards

Uploads use Pywikibot throttling and MediaWiki maxlag:

interval between edits (configurable; default 10 seconds)
maxlag parameter (configurable; default 5)

Pywikibot also uses its throttle control file (throttle.ctrl) to manage throttling state across runs.

Conflict resolution for existing titles

When a page title already exists, the uploader supports limited automated conflict resolution patterns:

if a {{versions}} page already exists at the title, it can add a new entry and upload the case-specific title
if an existing page with {{header/裁判文书}} appears to be from the same court, it can move the existing page to a case-specific title, create a {{versions}} page at the original title, and upload the new document to its case-specific title

Conflicts that cannot be resolved automatically are skipped and logged for manual handling.

This behavior is meant to avoid destructive overwrites while still allowing the program to proceed in the presence of a small number of collisions.

How this pipeline supports the staged program

The staged program requires that each step can be audited and paused. The pipeline supports that operationally:

conversion is offline and produces reviewable artifacts (converted JSONL, failed JSONL, checkpoint, optional TXT)
upload is rate-limited, respects maxlag, and produces structured logs suitable for reporting
the checkpoint mechanism supports interruption and resumption without restarting long-running jobs

For Stage 1 and early Stage 2, the intended workflow is to generate small converted samples (and TXT review output), perform manual review, then upload under a conservative interval.

Links

Repository: prc-court-docs-importer-html on GitHub