Research:Supporting deliberation and resolution on Wikipedia
This purpose of this study is to evaluate the utility of Wikum, a discussion summarization and decision support tool, for synthesizing input and determining consensus on large Wikimedia discussions, such as RfCs and WMF Community consultations.
Introduction[edit]
Discussions that happen within Wikipedia Talk pages often grow to tens or hundreds of comments. These discussions can be difficult for a reader to sift through because of their size as well as the many deep threads of back-and-forth conversation that arise. Beyond being difficult for readers, these discussions are an issue for editors of Wikipedia that need to come to a conclusion about a discussion, such as a Request for Comment (RfC). It may take a great deal of effort for a single Wikipedia editor to read through an entire RfC and then formally close it, while characterizing the overall arguments on either side and their final decision.
To help people quickly create a summary of a lengthy discussion, the tool Wikum was developed. This tool allows people to tag, group, and summarize small chunks or subthreads of discussion at a time, before going on to summarize larger and larger parts of the discussion that were already partially summarized. The tool also makes it easier for multiple people to work on the summarization process as it is clear what work has been done versus not, though it may be helpful for a single person summarizing a discussion as well. The tool was evaluated on three groups of paid recruited summarizers and compared to an alternative tool such as Google Docs. It was found that Wikum was overall faster for summarizing than Google Docs. More details can be found in the paper: "Wikum: Bridging Discussion Forums and Wikis Using Recursive Summarization", published at CSCW 2017.
In this research project, we aim to understand the current process of closing RfCs for Wikipedia editors as well as investigate the use of Wikum towards closing RfCs both in the individual case and as a group.
Research Questions[edit]
- Understand the current process by which RfCs get closed, including how many editors close them, how many RfCs get closed, and what methods editors use to close them.
- Release a comprehensive dataset of structured RfCs, and an analysis of their properties, including what characteristics predict whether an RfC will eventually be closed.
- Study the use of Wikum by a single editor towards closing an RfC and compare with their current method.
- Study the use of Wikum by multiple editors towards closing an RfC. Consider the assignment of roles, such as having more junior editors summarizing lower level subthreads or adding tags and more senior editors approving lower level summaries and summarizing at a higher level.
Methods[edit]
Initially, we conducted semi-structured interviews with a small number of editors, including frequent RfC closers and participants to understand their processes. 10 interviews were conducted.
Following that work, we collected a comprehensive dataset of RfCs into a structured format, so that closes as well as the entire discussion threaded structure is intact. We conducted an analysis of this data to understand what is predictive of closure. This dataset will be released publicly and linked here.
Following that work, we will conduct user studies to observe how editors close RfCs using their preferred method and Wikum. Frequent RfC closers will be approached to try out the tool, either to close an open RfC or an already-closed one, before providing feedback. Following the discussion that happened on the administrator's noticeboard, no participants will be compensated for their participation in this study.
Timeline[edit]
- March 2017: Develop interview questions
- April 2017: Interviewing commenced. In total 10 editors were interviewed.
- June 2017: Data collection of RfCs commenced. Over 7000 RfCs were collected from the beginning of English Wikipedia.
- Jan 2018: Data analysis of RfC dataset commenced. Building models to predict closure of RfCs from characteristics like tone and reciprocity of discussion, seniority of participants.
- Feb 2018: Continued feature development of Wikum in response to interviews and understanding of editors' current processes.
- April 2018: Data analysis concluded. This work was submitted and accepted to CSCW 2018. Link to completed paper will be posted here.
- August 2018: Finalize user study plans and begin recruitment and study of Wikum.
- April 2018: Analyze and write up data results.
Policy, Ethics and Human Subjects Research[edit]
The Wikum study has been approved by MIT's IRB to be exempt.
Results[edit]
Im, Jane; Zhang, Amy X.; Schilling, Christopher J.; Karger, David (2018-11-01). Deliberation and Resolution on Wikipedia: A Case Study of Requests for Comments 2 (CSCW). pp. 1–24. doi:10.1145/3274343.