Research:System Design for Increasing Adoption of AI-Assisted Image Tagging in Wikimedia Commons
Supported by Wikimedia Research Fund
This page documents a planned research project.
Information may be incomplete and change before the project starts.
In this research, we aim to investigate designs to increase the adoption and satisfaction of AI- assisted tools within commons-based peer production (CBPP) projects, with a specific focus on Wikimedia Commons. While AI- powered automation tools have long been integrated into CBPP projects for indirect tasks like content moderation, the utilization of AI for direct content generation has surged with recent advancements in generative AI algorithms. However, the impact of AI-assisted tools on human contributors and the design considerations to enhance their interaction, adoption, and satisfaction remain uncertain. This study proposes to co-design an AI-assisted image tagging tool with Wikimedia Commons contributors and users to increase adoption and satisfaction. We will perform a study of the prior WFM attempt to provide a computer-aided tagging (CAT) tool to understand the factors that led to its deactivation. We will then investigate technology designs to improve AI-assisted image tagging for structured Commons. The successful completion of this project is expected to advance the development of an AI-assisted image tagging tool on Wikimedia Commons, promoting greater adoption, usage, and satisfaction among contributors. Additionally, the insights gained from this study can be generalized to enhance interaction and collaboration between human contributors and AI-powered automation tools in the broader Wikimedia tools ecosystem and other CBPP projects.
Introduction[edit]
AI-powered automation tools have long been integrated into commons-based peer production (CBPP) projects for indirect tasks such as content moderation and contribution quality. In recent years, with the rapid advancement of generative AI algorithms, there has been a surge in attempts to utilize AI-powered tools for generating direct content in CBPP, including creating Wikipedia articles and generating image annotations. AI holds the promise of enhancing content creation efficiency and consistency, thereby addressing content gaps in areas that may have received insufficient human contributions. However, the impact of AI-assisted tools on human contributors as well as the technology designs to enhance human contributors' interaction, adoption, and satisfaction with AI-assisted tools in CBPP remain uncertain. In this proposed research, we will fill in this gap by co-designing an AI-assisted image tagging tool with contributors and users of Wikimedia Commons, with the goal of increasing adoption and satisfaction.
Wikimedia Commons is a WMF project that makes multimedia resources available for free copying, usage, and modification. However, a lack of structured, machine-readable metadata about media files has hindered its accessibility, searchability, usability, and multilingual support. Recently, WMF researchers attempted to introduce computer-aided image tagging (CAT). Unfortunately, our prior research revealed low adoption of the CAT tool by Commons contributors. Participants reported unsatisfactory usability and performance of the tool and resistance to changing their existing workflow of creating and maintaining the local category system. The CAT tool was deactivated in September 2023.
In this project, our aim is to study the previous WFM attempt to provide the CAT tool, understand the factors that led to its deactivation, and explore technology designs that enhance the adoption, usage, and satisfaction of AI-assisted tagging among Commons contributors. Our research questions are:
- How do AI-assisted image tagging and structured data on Commons affect the work and workflows of diverse contributors and user communities in related Wikimedia projects, such as Commons, different language versions of Wikipedia, and Wikidata?
- What are the perceptions, concerns, and preferences of Commons contributors and users regarding the quality of tags suggested by AI algorithms?
- What usability issues and challenges do contributors encounter when using the CAT tool on Commons?
- What technology designs can enhance the quality of suggested tags, identify appropriate tags for a Depicts statement, and improve the overall user experience with AI-assisted image tagging on Wikimedia Commons?
The successful completion of this project would advance the development of an AI-assisted image tagging tool on Wikimedia Commons, fostering greater adoption, usage, and satisfaction among contributors. This enhanced tool will provide Commons contributors with a more efficient, accurate, and user-friendly method of adding structured data to multimedia files. Improved structured data will enhance the searchability and usability of multimedia resources across WMF projects, promoting inclusivity among diverse language communities. Furthermore, the design insights from this study can be applied broadly to improve interaction and collaboration between human contributors and AI-powered tools across the Wikimedia tools ecosystem and other CBPP projects.
Methods[edit]
This study will comprise two stages. In the initial stage, we will conduct qualitative interviews with contributors and users of Wikimedia Commons to gather insights into their experiences with the deactivated WMF CAT tool. The outcomes of this first stage will inform the second stage of the study, where we will codesign solutions with contributors and users of Commons to enhance the adoption and satisfaction of AI-assisted image tagging tools.
Stage One - Learning from a Prior AI Tagging Project
In the initial phase of our study, we will first locate the discussions where the CAT tool was mentioned and analyze those discussions and posts to understand contributors' concerns and what they desired from the tool. Subsequently, we will broaden the dialogue beyond experienced Commons contributors by interviewing individuals with varying levels of experience, as well as those engaged in other Wikimedia projects supported by Commons, such as different language versions of Wikipedia and Wikidata. Through these interviews, we aim to delve deeper into the issues surrounding the deactivated CAT tool. The key areas of discussion include:
- Scope and Goal of Depicts Statements/Structured Data on Commons: We will guide participants in discussing how these elements influence and impact the current work and workflows of diverse contributors and user communities, as well as the collaborations between these communities.
- Quality of Tags Suggested by the CAT Tool: During each interview, we will present 3-5 example images and ask the interviewee to suggest Depicts statements. Following this, we will present and discuss tags suggested by the CAT tool with the interviewee to comprehend participants’ perceptions, concerns, and preferences regarding AI-assisted tagging. This process aims to identify cognitive distinctions, areas of agreement on AI-generated tags, and ways to enhance the suggestions.
- Usability of the CAT Tool: During each interview, we will showcase a live demo of the CAT tool, allowing participants to ask questions and provide feedback on the tool’s usability.
Stage Two - Co-Design for Enhanced Adoption and Satisfaction
In the second stage, we will actively involve members of the Commons community in a co- design study aimed at refining the CAT tool. Through participant interviews, we will collaboratively explore:
- Technology Designs: We will investigate technology designs that utilize current AI technologies, including human-in- the-loop image annotation [9] and tag refinement methods [10] based on human-generated category information and file descriptions. The goal is to enhance the quality of suggested tags.
- Tag Appropriateness and Filtering: We will explore ways to identify tags suitable for a Depicts statement and effectively filter out generic keywords or tags.
- Usability Enhancement Designs: We will work on designs to improve the overall user experience and address the usability issues identified in the stage one interviews.
Expected output[edit]
This project will produce three outputs:
- Research reports for the Wikimedia community.
- At least one paper manuscript to be submitted to leading HCI conferences such as CSCW or CHI.
- Presenting our research progress and outputs at events such as Wikimania 2024 and Wiki Workshop 2025 for Wikimedia contributors, developers, and interested community members.