Wikimedia CH/Grant apply/Evaluating AI-assisted translations on Wikipedia

Infodata

Name of the project: Evaluating AI-assisted translations on Wikipedia
Amount requested: 10,000 CHF
Type of grantee: Organization (Open Knowledge Association, OKA)
Name of the contact: Jonathan Zimmermann
Contact: jz@oka.wiki

The problem and the context

What is the problem you're trying to solve?

While machine translation tools — including AI-based systems — have become increasingly accurate, there is limited empirical understanding of their reliability, contextual accuracy, and real-world usefulness for translating Wikipedia content. Many translations produced by automated systems contain subtle errors, cultural biases, or factual distortions that reduce the overall quality of multilingual content.

At the same time, professional human translation resources are limited, and there is a need for systematic evaluation of how AI-based tools can complement the work of professional translators in a Wikimedia context.

What is your solution to this problem (please explain the context and the solution)?

This project will test and compare several AI translation systems (e.g., ChatGPT, Claude, Gemini, Grok, DeepL, and open-source LLM translators) by having professional translators review and annotate their outputs for a small, representative set of Wikipedia articles (e.g., around five).

Based on this comparison, one system will be selected for in-depth evaluation, where a larger set of articles (e.g., several dozen) will be translated and analyzed in more detail.

Human translators will provide both qualitative and quantitative feedback, assessing:

Accuracy of meaning and terminology
Faithfulness to the original structure and tone
Handling of cultural or technical nuances
Ease of post-editing and time saved

The results will form the basis of a comparative study on translation quality and post-editing efficiency, leading to guidelines and recommendations for using AI-assisted translation responsibly in Wikimedia projects.

This work does not involve copyright concerns since all materials will be Wikipedia content under free licenses, and will focus on improving translation quality, not on paid content creation.

Project goals

Evaluate the quality and limitations of AI-assisted translations in the Wikimedia context
Compare different AI translation engines through systematic review
Identify best practices and practical workflows for human–AI collaboration in translation
Provide guidelines for community members and editors on responsible AI-assisted translation
Contribute to the Wikimedia CH Innovation Programme by producing a case study on human–AI collaboration and content quality

Project impact

How will you know if you have met your goals?

Success will be measured through:

A completed dataset of AI-translated and human-reviewed Wikipedia texts
Quantitative metrics of translation accuracy and error rates
Qualitative feedback from human translators
A published report or white paper summarizing findings and recommendations
Positive feedback or reuse of the results by Wikimedia translators and editors

To quantify quality and improvement, we will calculate an “error rate per 1,000 words”, with several complementary variants:

[Potential follow-up analysis] Baseline correction rate: Errors in the original article that the AI detected and corrected (how much the AI helped improve quality).
[Main analysis for initial publication] Post-editing correction rate: Edits made by the human translator to correct the AI’s output (how much the translator had to fix).
[Optional future study] Community revision rate: Edits made by the broader Wikipedia community after publication, compared to the AI+translator version (how durable the translation quality is).

Do you have any goals or metrics around participation or content?

At least 5–10 translators involved in reviewing AI-generated translations
Evaluation of a minimum of 100 translated Wikipedia segments across at least 4 language pairs (e.g., EN→PT, EN→ES, FR→EN)
Production of one synthesis report and a set of translation guidelines made publicly available under an open license

Project plan

Activities

Selection of Wikipedia articles and language pairs
Generation of AI-based translations using multiple tools
Translation review and annotation of results
Data collection and statistical analysis
Drafting of a final report and publication of recommendations
(Optional) Presentation of results at a Wikimedia CH Innovation event or roundtable

Budget

To achieve the desired coverage in terms of languages, topics, and participants, we request 10,000 CHF.

These funds will be disbursed as OKA grants to translators to cover their cost of living while participating in the project. This will support approximately 10–15 part-time translators over 2–3 months.

Each translator will receive the usual OKA stipend of about 450 USD per month (adjusted by experience), plus a small 5% participation bonus as incentive.

OKA will cover all non-grant costs (administrative expenses, licenses, coordination). Members of the OKA organization will volunteer their time for project coordination and reporting.

Community engagement

The project will engage:

Professional translators interested in participating in the project (grant recipients)
Wikimedia volunteers involved in translation and content quality
The Wikimedia CH Innovation Programme community, as part of its ongoing research on responsible AI use and human–AI collaboration.

The results will be shared openly on Meta-Wiki and through Wikimedia CH channels, inviting discussion and feedback from the broader community.

Decision

The grant is approved under Innovation programme. --Ilario (talk) 17:05, 22 October 2025 (UTC)[reply]