Research:Understanding How Editors Use Machine Translation in Wikipedia: A Case Study in African Languages

Contact

Duration:  2023-July – 2025-June

 Supported by Wikimedia Research Fund

Grant ID: G-RS-2302-12035

Grant application

---

Research:Projects

This page documents a research project in progress.
Information may be incomplete and change as the project progresses.
Please contact the project lead before formally citing or reusing results from this page.

Introduction

The rapid expansion of Wikipedia content in low-resource, underserved African languages is heavily dependent on the accuracy of ContentTranslation, Wikipedia’s out-of-English Machine Translation (MT) tool using MT engines. However, translation quality remains inconsistent, especially for low-resource languages where MT does not adequately support both linguistic diversity and cultural suitability.

This project investigates how MT and human-edited (HT) translations differ on an aggregate segment level as well as on a fine-grained pairwise difference level, creating a new open source fully annotated dataset and analyses to inform both the Wikipedia translation workflow and future MT research on African languages, low-resource languages in general.

The contributions in this project are as follows:

• A dataset of 5,000 English source, MT output, and HE output parallel segments, fully annotated on the segment level and the pairwise difference level;
• Open source code to replicate the annotation interface;
• A list of actionable insights to Wikipedia language editions based on our analysis of the annotated data.

Annotating Post-Editing Behavior

Data Source and Global Statistics

We extract parallel English source, machine translation (MT) output, and human post-edit (HE) output segments from Wikipedia 06/13/2025 data dumps for Hausa, Igbo, Swahili, Yoruba, and Zulu.

Table 1 provides a statistical overview of this initial dataset. While Hausa and Igbo represent the largest corpora, a more telling metric is the Levenshtein character-level edit distance between their MT and HE pairs, which suggests that editors in Hausa and Zulu make fewer or less significant edits on average. Most importantly, all five languages have a null or near-zero average difference between the AfriCOMET Quality Estimation scores of their MT and HE pairs. This suprising observation---that a quality estimation metric detects little to no quality differences after human editing---is the primary motivation for a deeper analysis on those differences. Additionally, the word count average difference between the MT and HE pairs points to a trend of human post-edit additions in the Hausa, Igbo, and Yoruba contexts contrasted by a trend of deletions in the Swahili and Zulu contexts.

Global statistics comparing machine translation (MT) and human-edited (HE) Wikipedia segments across five African languages. n denotes the number of parallel segments found in the 06/13/2025 Wikipedia dump. Global AfriCOMET Difference refers to the difference between the HE and MT AfriCOMET scores. Global Edit Distance represents the Levenshtein edit distance between the MT and HE pairs. Global Token Difference refers to the word-count difference between the MT and HE pairs.

Target Language	Edit Dist. (MT vs. HE)	AfriCOMET (MT)	AfriCOMET (HE)	AfriCOMET Diff	Token Diff
Hausa (n = 177,387)	40.39	0.65	0.66	0.01	19.34
Igbo (n = 200,161)	89.35	0.59	0.58	0.00	38.71
Swahili (n = 10,100)	93.50	0.72	0.72	0.00	-7.41
Yoruba (n = 9,988)	91.97	0.56	0.58	0.02	24.45
Zulu (n = 7,319)	40.77	0.67	0.66	-0.01	-3.01

Curation of the Annotation Dataset

From the large corpora described in the preceding section, we curated a subset of 1,000 representative source-MT-HE triplets per language for annotation and analysis. To ensure the selected segments were informative and suitable for a detailed comparison, we applied the following filtering criteria:

• Bounded Quality Difference: The quality score differences between HE and MT must be between -0.5 and 0.5, as measured by AfriCOMET QE. We exclude segments with zero differences to avoid pairs with minimal variation.
• Meaningful Edit Distance: The character-level Levenshtein distance must be between 5 and 40 to filter out both cases with insignificant changes (e.g., punctuation edits) and complete re-translations which are hard to compare directly.
• Sufficient Source Length: The English source segments must contain at least 20 words to ensure segments are substantial enough to annotate.
• High Absolute Quality: The AfriCOMET QE score for both HE and MT must be greater than 0.5 to focus the analysis on higher quality translations.
• Data Cleaning: We remove duplicate segments and segments with excessive special characters (e.g., !]>*), which often signal lower quality segments.

Figure 1 presents the distribution of the AfriCOMET QE score difference between HE and MT for the final set of 1,000 selected segments across the five languages studied.

Distribution of AfriCOMET QE score difference between MT and HE in the selected 5,000 segments for all five African languages.

Human Annotation Protocol

We developed our annotation protocol through a pilot study before deploying the main annotation task.

Pilot Study: To refine our protocol, we first conducted a pilot study where three annotators per language evaluated the same 50 segments. Insights from this study were used to finalize the main annotation task which we detail below.

Main Annotation Task: For the primary annotation effort we built our annotation tool on the Appraise tool. For each of the five languages, each annotator is assigned the same 1,000 segments to annotate. The interface presents the English source followed by the two candidates (MT and HE) in a randomized order to prevent bias. To encourage annotators to focus on the differences between the two translations we highlight them in yellow for each example. The task requires annotators to:

• Select the better overall translation.
• Annotate the advantages of the selected translation and the disadvantages of the unselected translation with a simplified MQM-style set of options spanning adequacy, fluency, and explicitation.
• Indicate which translation, if any, qualifies as adequate for a Wikipedia entry.
• For each pairwise difference between the candidates, select their preferred span and explain their reasoning by selecting one of a pre-defined list of possible explanations. Annotators are encouraged to expand on their reasoning by selecting "Other" and typing in their observations as needed.

Recruiting Annotators: We recruited three annotators per language through Masakhane.

Post-annotation Processing and Final Statistics

Before analysis, the raw annotation data was post-processed to create the final dataset. During this phase, we reset the Hausa annotation effort halfway through after receiving feedback from the annotators that the initial segments contained repeated, incoherent text. As a result, only the most recent, smaller Hausa subset is included in the analysis.

Final Dataset: For our primary analysis, we only consider cases where a clear preference was established by a majority of annotators. We filter out segments where at least two out of three annotators spotted no difference between the candidate translations. As shown in Table 2, this filtering had a significant impact on the number of remaining segments, where a large number of post-edits are assessed as having no effect on quality.

Inter-annotator Agreement: Table 3 presents the Fleiss' kappa inter-annotator agreement scores for the aggregate preference task. Agreement is substantial for Igbo (0.71) and moderate for Yoruba and Zulu (0.51 and 0.52, respectively). However, IAA is lower for Swahili (0.37) and the new Hausa subset (0.27), indicating that judging overall translation quality was a more subjective task in these language contexts.

Table 2: Annotation statistics by language: excluded and remaining segment counts, and majority aggregate MT vs. HE preferences. Only segments with majority (≥2/3 annotators) agreement are kept.

Language	Excluded Segs.	Rem. Segs.	MT Pref.	HE Pref.
Hausa (old, 360)	49	311	291	20
Hausa (new, 176)	151	25	0	25
Igbo (1000)	3	997	615	382
Swahili (1000)	561	439	251	188
Yoruba (835)	149	686	177	509
Zulu (1000)	137	863	744	119

Table 3: Fleiss' kappa inter-annotator agreement scores for aggregate HE vs. MT preference across languages, considering only segments with exactly three annotators.

Language (Dataset)	Fleiss' κ	N Segments
Hausa (old)	0.51	360
Hausa (new)	0.27	176
Igbo	0.71	1000
Swahili	0.37	1000
Yoruba	0.51	835
Zulu	0.52	1000

Analysis of Post-Editing Behavior

Our analysis of the annotated data reveals a complex and often counterintuitive relationship between MT and HE, challenging the assumption that post-editing Wikipedia content targets improvements against the original MT. Figure 2 and Figure 3 summarize the distribution of factors cited for translation superiority and inferiority, respectively, across all five languages in our study. We summarize the findings and implications of those below.

Why is a translation better?

Our analysis reveals that fluency is overwhelmingly the dominant driver of annotator preference. In all languages, the most frequently cited factors for preferring one translation are Fluency_Natural and Fluency_Grammar_Spelling. Interestingly, machine translation (MT) outputs often outscore human post-edits on fluency grounds in Zulu, Swahili, and Igbo. In contrast, human-edited output tends to surpass MT in Yoruba and Hausa for fluency.

Adequacy plays a complex role. Contrary to common assumptions, MT is also frequently cited as more adequate than the human post-edits—in all languages except Hausa. This suggests that in some cases, human interventions introduce errors or omit important information that MT maintains. Explicitation (the addition of beneficial context or clarifications) emerges as a notable factor, especially in Hausa, Swahili, and Yoruba. Finally, segments with NoDiff (no real difference in fluency or meaning between candidates) form a substantial subset in most languages, except for Igbo, where differences are more often perceived.

These patterns, visualized in Figure 2, underscore that language-specific strategies may be necessary to improve translation workflows. They also suggest that some post-edits may be superficial or even detrimental to translation quality, particularly when editors make changes that reduce fluency or adequacy.

Figure 2: Distribution of justifications selected by annotators for why one translation is preferred over the other.

Why is a translation worse?

When annotators identify why a translation is inferior, a distinct language-dependent error profile emerges (Figure 3). Human editors are most frequently identified as omitting information present in the source, a pattern that is striking in Igbo, Swahili, and Zulu. In addition to omissions, human editors also sometimes add or elaborate information not present in the source text.

By contrast, the primary weaknesses of MT translations are less often tied to adequacy errors; more frequently, annotators cite general quality issues (Other or NoDiff). This indicates that, while MT typically remains closer to the original content, its output sometimes lacks the refinements or contextual adaptions made by human editors, though such adaptions may not always improve fidelity.

Collectively, these results suggest that MT is generally more faithful to the source, while humans are more likely to commit major errors per established MQM categories, particularly omissions or problematic additions. This raises important questions: Do human editors sometimes aim for goals beyond faithful translation, such as localization, summarization, or re-writing? If so, should such edits be distinguished in Wikipedia workflows (e.g., with a tag different from the standard HE marker), and how should downstream systems recognize and support these distinctions?

Figure 3: Distribution of factors cited by annotators for why a translation is judged worse than its counterpart.

Implications for Wikipedia

Our analysis strongly suggests that post-editing MT content on Wikipedia is not monolithic task of error correction. Editors often engage in more complex activities that go beyond faithful translation, such as adding or removing content.

While these adaptations can be valueble, our analysis show they are also risky as they can degrade fluency. Those findings have direct implications for the use of ContentTranslation tool in the languages studied:

• Distinguising Editorial Roles: Wikipedia's worklfow could benefit from distinguishing between different types of edits. The workflow could be enhanced by allowing editors to tag their intent, for example, distinguishing a Faithful Correction from a Cultural Adaptation. This distinction is critical for quality control; otherwise, valuable adaptations may be incorrectly flagged as translation errors, and the true nature of the editor's work is lost.
• Develop Language Specific Strategies: The clear differences in editing patterns and fluency outcomes between languages (e.g., Yoruba vs. Zulu) suggest that a one-size-fits-all approach to quality control is suboptimal. Workflows need to be adapted to the specific needs of each language community.
• Implement Fluency Checks: This study reveals that human edits can, paradoxically, decrease the fluency of machine-translated text, an issue mainly seen in Igbo, Swahili, and Zulu. To address this, the workflow could integrate lightweight automated checks that flag potentially awkward or ungrammatical sentences in post-edited content.