Research:MinT (Machine in Translation) Research
MinT (“Machine in Translation”) is a new translation service by the Wikimedia Foundation Language Team that is based on open source neural machine translation models. MinT currently supports over 200 languages, including many underserved languages that are getting machine translation for the first time. For example, the recent integration of NLLB-200 model into the Content Translation tool supported machine translation for Fula, spoken by 25+ million people, for the first time. MinT also helps expand the machine translation options for all languages more generally.
The goal of this project was to understand how to better leverage MinT to support more readers and contributors, especially those whose languages are receiving machine translation support for the first time or receiving additional machine translation options, in their aim of accessing, interacting with, and contributing to Wikipedia content, as well as knowledge more generally. We collaborated with Anagram Research to carry out this project, which included multiple rounds of concept testing and contextual inquiries around individuals' experiences with machine translation more generally.
Research Goals[edit]
Part I[edit]
Part I of the project is focused on evaluating new concepts for how we might help increase awareness of additional encyclopedic content that exists across language versions of an article or section. For example, by helping readers access more content regardless of the language it originates in, and expose readers to ways in which they can easily transition from readership to more active contributorship. We aim to gather feedback supporting iteration on an initial set of design concepts, and uncover additional concepts or ideas to explore in more detail.
General questions: (as they relate both to on and off Wikipedia experiences)
- How do readers’ awareness and understanding of other language versions of Wikipedia vary? What are common assumptions, misconceptions, and questions?
- How does multi-lingual reader trust vary as a function of the language in which they access encyclopedic content?
- What challenges do monolingual readers face when trying to access content in other languages via unedited machine translation outputs?
- To what degree do readers user MT, including services such as Google Translate, and how do challenges and reading patterns changes as a function of their availability?
- How do patterns vary across device types?
- How do monolingual readers generally approach learning more about a topic, especially when contents in their language may be limited?
- When presented the option to access content in other languages they don’t read fluently, what are tradeoffs they perceive? What relationships are there between MT quality and trustworthiness in their perceptions of these contents?
- What aspects of these individual solutions are valued most and why?
- What are participant perceptions of MT quality across current services? (a relevant question to understand across all parts and contexts of this project)
Part II[edit]
Part II of the project is more generative, and aims to understand how those who read and write lower resourced languages (especially languages for which machine translation options are expanding and improving - but may still vary in quality) are currently using machine translation in their daily pursuit of individual learning and education. In particular, we want to know in which ways could MinT help reduce language barriers to knowledge? For this part of the project, we want to consider MinT-supported access to knowledge most broadly; consider, for example, how MinT might be leveraged by contributors who use machine translation as a tool when researching an article or article section they write. Because lower resourced languages is a very broad category, we aim to take a comparative approach by selectively targeting two languages for which machine translation quality is both relatively high and those for which it is lower. This is because we recognize that experiences with machine translation vary significantly depending on the overall quality of the outputs generated.
Part II Research Questions:
- In what ways, if any, are readers using machine translation in their pursuit of knowledge?
- What workflows and tools have they developed?
- What are common pain points?
- How do language barriers affect readers’ abilities to access content, both on and off-Wikipedia and other Wikimedia projects?
- What are commonly held perceptions, including perception of value and concerns about, machine translation supports for their language?
- What sociocultural factors are most relevant for considering language supports for readers and potential contributors to Wikimedia projects?
- What barriers are most important to consider for how potential contributors might be leveraging machine translation supports for editing?
- What are common user journeys for reader workflows when they are interfacing with content in other languages (both in and outside of what they consider their preferred language(s) of reading)?
Research Approach[edit]
For Part I of the project, we used the Rapid Iterative Testing and Evaluation (RITE) Method, with two cycles of moderated testing and feedback with 12 participants per cycle, split across three language groups (Awadhi, Chhattisgarhi, and Hindi). Sessions were carried out remotely and had a duration of 60-75 minutes each. Between cycles we scheduled a window of time to review preliminary findings and iterate on the design concepts. Research sessions included small tasks for participants to complete with interactive (tappable/clickable prototypes of medium fidelity), as well as a pre- and post-task interview portion.
For Part II of this project, remote contextual inquiries with a duration of 60-75 minutes each were conducted with readers to understand how they currently use new machine translation services just coming online for their languages in their pursuit of knowledge. For Hindi, this meant in the context of additional machine translation services coming online, whereas for Awadhi, MinT represents the first machine translation support option in the context of Wikimedia’s Content Translation tool. Like Awadhi, Chhattisgarhi is another language just beginning to receive machine translation support. The contextual inquiries in this part of the project investigated the socio-cultural context of the individuals and target communities, also identifying specific language-related considerations and concerns around the use of MT. These language factors included structural, written, and sociolinguistic factors. Using phenomenological methods, the contextual inquiries emphasized task-based inquiry to uncover workflows and provide some evaluative assessment of resources like MinT in its current form.
For pragmatic reasons we narrowed the scope of inquiry based on particular target languages/wikis. The three selected included Awadhi, Hindi, and Chhattisgarhi. Our process for selection included the following as key criteria for inclusion/focus:
- Communities getting MT for the first time, with good levels of activity in the wikis. These represented areas where MinT may have the potential of bringing unique benefits to an area where there’s already momentum in knowledge curation and readership on Wikimedia projects. In assessing ‘levels of activity’ general readership metrics were just as important as contributorship metrics.
- A range of relative MT output quality, in order to take a comparative approach of the experiences among those for which MT quality is relatively poor and relatively high. Ideas and concepts may differ in appropriateness and usefulness based on MT quality.
- Medium-size wiki with potential to use MT to cover content gap. These are wikis in which there’s wide topical coverage, but more limited depth (for example, shorter articles with fewer sections). This is an area where MT could be a potentially potent tool for increasing article content coverage.
- Indic languages were considered due to a general interest at Wikimedia Foundation to define archetypes, but also motivated by anticipated efforts of other groups and companies in this area.
- General logistics were also considered, and include everything from recruitment considerations, to interest among communities, to project timeline impacts.
Results[edit]
Read the Full Part I Report This report contains a summary of general insights as well as detailed feedback on the participant-tested concepts.
Read the Full Part II Report This report contains general insights along with sample translation workflows, an exploration of content contribution through manual translation, and feedback about the current test instance of MinT.