Research:Facilitating Public Deliberation of Algorithmic Decisions
This page is an incomplete draft of a research project.
Information is incomplete and is likely to change substantially before the project starts.
Introduction & Background[edit]
38 Wikipedia languages are currently using ML-based systems such as Objective Revision Evaluation Service, or ORES to make algorithmic predictions regarding edits quality. Past research has shown stakeholders often have different and competing values regarding these decisions. For example, there is a tension between improving the efficiency of reverting low-quality edits and treating newcomers nicely. We are proposing a deliberation-driven decision making process to address those competing values. For example, participants will be presented with different model cards representing different values and will discuss, debate, and negotiate about their models. Eventually, they will be asked to select a model and offer rationale behind their choice. We will also measure the process using metrics like quality of the decision, subjective satisfaction (legitimacy) and quality of the deliberation.
Value Table[edit]
| Social/stakeholder group | Values (priorities) | System Criteria |
|---|---|---|
| Experienced editors | Optimize experience of the experienced editors | Optimize accuracy for experienced editors |
| Newcomers | Optimize experience of newcomers | Optimize accuracy for newcomers &
Maximize non-damaging predictions for newcomers |
| Wiki Community; Fairness advocates | Fairness | Keep similar accuracy, and/or false positive/negative rates for both experienced and newcomers |
| Patrollers | Reducing overall efforts and maximize precision | Minimize both damaging predictive rates & maximize fraction of actually damaging edits that are predicted as damaging |
| ML developers | Accuracy of the system | Maximize overall accuracy |
| Readers | Content quality and maximize negative predictive value | Minimize the proportion for damaging edits predicted as non-damaging |
Process Documentation[edit]
| Date | Activities | Participants |
|---|---|---|
| 06/02/2020 | Project initiated! | Internal Group Meeting |
| 06/10/2020 - 10/10/2020 | Weekly group meeting for literature review and project formulation | Internal Group Meeting |
| 10/20/2020 | Discussed with Dr. Amy Zhang and Dr. David Kargerabout adopting Wikum they designed and developed for Wikimedia for the deliberation process at CSCW virtual conference. | Amy Zhang, David Karger, Wesley Deng. |
| 10/21/2020 | Conversation with Dr. Aaron Halfaker on the project plan and the best way to help the Wiki community through the project. | Aaron Halfaker, Wesley Deng. |
| 11/12/2020 | Had a conversation with Dr. Aaron Halfaker concerning the project design and which wiki communities to collaborate with. Aaron offered his insights on the next step. | Aaron Halfaker, Wesley Deng, Hong Shen. |
| 11/13/2020 | Presented the project plan for Dr. Loren Terveen and Dr. Mark Snyder from UMN for their advice on how to better design the study to benefit the community. | Loren Terveen, Mark Snyder, Wesley Deng, Hong Shen, Haiyi Zhu. |
| 12/01/2020 | Check with Aaron about the recruitment messages and study design. | Aaron Halfaker, Hong Shen |
| 12/02/2020 | New iteration for the interfaces that could better communicate the algorithmic decisions made by ORES. | Internal Group Meeting |
| 12/09/2020 | Usability study for new iteration for the interface design. Re-design the pre-survey (Is it too long? Is the question vague?). Starting a Wikum page for pilot testing. | Internal Group Meeting |
| 01/06/2021 | Deploy the interface online. Click here to check out the interface! | Internal Group Meeting |
| 01/07/2021 | Insightful interface critique from enthusiastic Dutch Wiki community members at Dr. Aaron Halfaker's Office Hour! | Aaron Halfaker, Leijie Wang, Wesley Deng. |
| 01/15/2021 | Informal user test for the wikum page and the interface with an enthusiastic English Wiki community member. | Anonymous Wiki editor. Wesley Deng. |
| 01/18/2021 | Deploy the interface with the edits dataset from the Dutch Wiki community. Click here to check it out. | Leijie Wang. |
| 01/25/2021 | Productive Meeting with two wonderful Dutch Community members on the future study plan. | Group Meeting with Dutch Community Members. |