Research:Community-centered Evaluation of AI Models on Wikipedia/Study Recruitment

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search


Are you someone who enjoys reviewing edits on the English Wikipedia? We could really use your assistance (with a maximum compensation of $150 USD)!

About Us:[edit]

We are a research team from Carnegie Mellon University, working together with the creator of ORES. Our research aims to help Wikipedians evaluate the quality of AI models on Wikipedia and identify areas for improvement. These models, such as ORES and Liftwing, power Huggle, SWViewer, recent change filters, etc., and aid patrollers in reviewing edits. We believe Wikipedians' voices are essential in evaluating AI models from their individual and collective perspectives, especially given the increasing use of AI models in Wikipedia nowadays.

Based on a formative study in our research, we developed Wikibench, a system that facilitates the AI evaluation process while centering Wikipedians' voices. We would be thrilled if you could participate in our pilot testing for this system!

The Studies[edit]

Our pilot testing involves two studies:

  1. Onboarding session: The first study consists of a 1-hour onboarding session to help you become familiar with our system. We will show you how the system work and gather your initial reaction and feedback on the system.
  2. Exit interview: After the first study, we kindly request that you continue using the system for one week while performing your patrolling work (with lightweight daily requirements that take less than 20 minutes per day). The second study takes place after one week, when we will learn about your experience with the system through a brief 30-minute exit interview.

Both studies will be conducted via Zoom (audio and screen-share only). We may also explore other options (e.g., text-based) if that works better for you. Please feel free to specify your preference in the screening survey below.


Upon successfully completing both studies, you will be eligible to receive a maximum compensation of $150 USD.

Specifically, we will provide $30 as compensation for the one-hour onboarding session, and $20 for the 30-minute interview. Additionally, if you meet the lightweight daily participation requirements for using our system during your one-week patrolling period, you will receive $20 per day, for up to 5 days.

How to participate[edit]

If you are interested in participating, please spend 30 seconds to complete a short screening survey (, which will help us determine your eligibility for the study. We will follow up with you via email within a few hours after receiving your response.

To receive updates about Wikibench and future study opportunities, please also sign your name here.

Lastly, please feel free to share this recruitment page with anyone who may be interested.

Contact us[edit]

If you have any questions, please don't hesitate to contact us. We greatly appreciate your time and consideration!