Research:Facilitating Public Deliberation of Algorithmic Decisions

From Meta, a Wikimedia project coordination wiki
04:03, 21 August 2020 (UTC)
Duration:  2020-August – 2021-May

This page is an incomplete draft of a research project.
Information is incomplete and is likely to change substantially before the project starts.

Introduction & Background[edit]

38 Wikipedia languages are currently using ML-based systems such as Objective Revision Evaluation Service, or ORES to make algorithmic predictions regarding edits quality. Past research has shown stakeholders often have different and competing values regarding these decisions. For example, there is a tension between improving the efficiency of reverting low-quality edits and treating newcomers nicely. We are proposing a deliberation-driven decision making process to address those competing values. For example, participants will be presented with different model cards representing different values and will discuss, debate, and negotiate about their models. Eventually, they will be asked to select a model and offer rationale behind their choice. We will also measure the process using metrics like quality of the decision, subjective satisfaction (legitimacy) and quality of the deliberation.

Value Table[edit]

Social/stakeholder group Values (priorities) System Criteria
Experienced editors Optimize experience of the experienced editors Optimize accuracy for experienced editors
Newcomers Optimize experience of newcomers Optimize accuracy for newcomers &

Maximize non-damaging predictions for newcomers

Wiki Community; Fairness advocates Fairness Keep similar accuracy, and/or false positive/negative rates for both experienced and newcomers
Patrollers Reducing overall efforts and maximize precision Minimize both damaging predictive rates & maximize fraction of actually damaging edits that are predicted as damaging
ML developers Accuracy of the system Maximize overall accuracy
Readers Content quality and maximize negative predictive value Minimize the proportion for damaging edits predicted as non-damaging

Process Documentation[edit]

Date Activities Participants
06/02/2020 Project initiated! Internal Group Meeting
06/10/2020 - 10/10/2020 Weekly group meeting for literature review and project formulation Internal Group Meeting
10/20/2020 Discussed with Dr. Amy Zhang and Dr. David Kargerabout adopting Wikum they designed and developed for Wikimedia for the deliberation process at CSCW virtual conference. Amy Zhang, David Karger, Wesley Deng.
10/21/2020 Conversation with Dr. Aaron Halfaker on the project plan and the best way to help the Wiki community through the project. Aaron Halfaker, Wesley Deng.
11/12/2020 Had a conversation with Dr. Aaron Halfaker concerning the project design and which wiki communities to collaborate with. Aaron offered his insights on the next step. Aaron Halfaker, Wesley Deng, Hong Shen.
11/13/2020 Presented the project plan for Dr. Loren Terveen and Dr. Mark Snyder from UMN for their advice on how to better design the study to benefit the community. Loren Terveen, Mark Snyder, Wesley Deng, Hong Shen, Haiyi Zhu.
12/01/2020 Check with Aaron about the recruitment messages and study design. Aaron Halfaker, Hong Shen
12/02/2020 New iteration for the interfaces that could better communicate the algorithmic decisions made by ORES. Internal Group Meeting
12/09/2020 Usability study for new iteration for the interface design. Re-design the pre-survey (Is it too long? Is the question vague?). Starting a Wikum page for pilot testing. Internal Group Meeting
01/06/2021 Deploy the interface online. Click here to check out the interface! Internal Group Meeting
01/07/2021 Insightful interface critique from enthusiastic Dutch Wiki community members at Dr. Aaron Halfaker's Office Hour! Aaron Halfaker, Leijie Wang, Wesley Deng.
01/15/2021 Informal user test for the wikum page and the interface with an enthusiastic English Wiki community member. Anonymous Wiki editor. Wesley Deng.
01/18/2021 Deploy the interface with the edits dataset from the Dutch Wiki community. Click here to check it out. Leijie Wang.
01/25/2021 Productive Meeting with two wonderful Dutch Community members on the future study plan. Group Meeting with Dutch Community Members.