Research:Discussion summarization and decision support with Wikum
This purpose of this study is to evaluate the utility of Wikum, a discussion summarization and decision support tool, for synthesizing input and determining consensus on large Wikimedia discussions, such as RfCs and WMF Community consultations.
Discussions that happen within Wikipedia Talk pages often grow to tens or hundreds of comments. These discussions can be difficult for a reader to sift through because of their size as well as the many deep threads of back-and-forth conversation that arise. Beyond being difficult for readers, these discussions are an issue for editors of Wikipedia that need to come to a conclusion about a discussion, such as a Request for Comment (RfC). It may take a great deal of effort for a single Wikipedia editor to read through an entire RfC and then formally close it, while characterizing the overall arguments on either side and their final decision.
To help people quickly create a summary of a lengthy discussion, the tool Wikum was developed. This tool allows people to tag, group, and summarize small chunks or subthreads of discussion at a time, before going on to summarize larger and larger parts of the discussion that were already partially summarized. The tool also makes it easier for multiple people to work on the summarization process as it is clear what work has been done versus not, though it may be helpful for a single person summarizing a discussion as well. The tool was evaluated on three groups of paid recruited summarizers and compared to an alternative tool such as Google Docs. It was found that Wikum was overall faster for summarizing than Google Docs. More details can be found in the paper: "Wikum: Bridging Discussion Forums and Wikis Using Recursive Summarization", published at CSCW 2017.
In this research project, we aim to understand the current process of closing RfCs for Wikipedia editors as well as investigate the use of Wikum towards closing RfCs both in the individual case and as a group.
- Understand the current process by which RfCs get closed, including how many editors close them, how many RfCs get closed, and what methods editors use to close them.
- Study the use of Wikum by a single editor towards closing an RfC and compare with their current method.
- Study the use of Wikum by multiple editors towards closing an RfC. Consider the assignment of roles, such as having more junior editors summarizing lower level subthreads or adding tags and more senior editors approving lower level summaries and summarizing at a higher level.
Initially, we will conduct semi-structured interviews with a small number of editors, followed by a wider deployment of a survey to understand the RfC closing process. Recruitment and sample size TBD.
Following that work, we will conduct user studies to observe how editors close RfCs using their preferred method and Wikum.
- March 2017: Develop interview and survey questions and begin data collection.
- April 2017: Continued feature development of Wikum in response to initial feedback and understanding of editors' current processes. Continue data collection.
- May 2017: Finalize user study plans and begin recruitment and study.
- July 2017: Analyze and write up data results.
Policy, Ethics and Human Subjects Research
The Wikum study has been approved by MIT's IRB to be exempt.