Research:Understanding Wikipedia Coordination Spaces and Costs
- Jodi Schneider
- Alexandre Passant
Understand conversation and coordination spaces in English Wikipedia, with an aim towards understanding the pain points and prototyping semantic enhancements. This is part of my Ph.D. work in DERI's Social Software Unit; we approach Computer Science with Semantic Web technologies. The goal of my Ph.D. is to provide support for conversation and argumentation in the social web using semantic web technology.
So far this has mainly involved study of article Talk pages and Articles for Deletion. My work aims to support decision-making and identify opportunities for socializing new community members.
- Key methodological features: Content analysis of Wikipedia pages; taxonomy development; interviews with editors; semantic enhancements.
- Scholarly papers at conferences
- Presentations to the community
Wikimedia Policies, Ethics, and Human Subjects Protection
Interview Study, 2012
We have studied the text of AfD discussions, and last year we conducted a round of interviews with experienced Wikipedians and adminstrators. Our overarching goal is to understand:
- What are the outcomes of deletion discussions, both for articles and for the community?
- What are the purposes and benefits of the current process? What are the "pain points"? What changes, improvements, or support tools could be suggested?
We have read, analyzed, and characterized discussions in two ways, based on decision factors and argument patterns. This helps us to understand research questions such as
- What factors contribute to the decision about whether to delete a given article?
- When multiple factors are given, what is the relative importance of those factors?
- What argument patterns are used in AfD discussions?
- Does experience with AfD change the argument patterns?
This helps us understand the process of arguing. Now we would like to better understand the outcomes of the process for the community.
We are particularly interested in understanding how people discover AfD, what their first experience of the process is like, and what they learn about through participating. We are also interested in learning about the problems experienced AfD participants see in the process, and how they determine whether an argument for keeping or deleting an article is a good one.
Other researchers have already noticed friction for newcomers to AfD: Geiger & Ford noticed that AfD participants are "overwhelmingly regulars to the process", and that it is relatively rare (18%) for "the creators of nominated articles [to participate] in that article's deletion discussion." (Participation in Wikipedia's Article Deletion Processes", Wikisym 2011). What we don't know is whether and to what extent this is a problem for the community.
We hope through interviews to find answers to questions like:
- What is the experience of participating in AfD like?
- What are the benefits to participating in AfD, for an individual? For the community?
- What are the costs and challenges of participating in AfD, for an individual? For the community? What are the problems and "pain points" with the current AfD process?
- What are the community norms for arguments? Would scaffolding arguments help new and infrequent participants?
This will help us understand the impact and outcomes of discussions, rather than just the discussion style and process.
Experienced Wikipedians and administrators
Previously, through using community spaces, I have recruited experienced Wikipedians and administrators for interviews about AfD. I am currently planning follow-up interviews with previous interviewees who gave me permission to contact them as my research progressed, to check my findings and to ask follow-up questions.
Novice Wikipedians/new users of AfD
I now need to interview Wikipedians who are occasional or new users of Articles for Deletion, to evaluate my findings and get their perspectives.
In terms of the sample, I propose to choose subjects from data that User:Staeiou collected for a WikiSym 2011 poster. (“G1: Patent nonsense”: Participation and Outcomes in Wikipedia’s Article Deletion Processes. R. Stuart Geiger and Heather Ford.).
I would like to interview 10 novices, so I would propose directly targeting subjects by email (I'm considering Talk page requests but this compromises anonymity.). Since I expect a low response rate (are there any figures on what's expected?), I would plan to randomly select 40 people to email, and then choose additional people as needed. (I would welcome suggestions for the content of the message as well; I would expect a brief description of the study, a link to this page, and a request to spend 30 min - 1 hour interview by skype, phone, or IRC.)
Benefits for the Wikimedia community
- Understand how community norms and policies are applied, at a very direct practical level
- Understand the differences in values and policy articulation between novice and experienced Wikipedians
- Suggest opportunities for streamlining decision-making while maintaining its quality
Ph.D. research from October 2009-October 2013.
Science Foundation Ireland SFI/09/CE/I1380 (Líon2)
“Identifying, Annotating, and Filtering Arguments and Opinions in Open Collaboration Systems” (PhD dissertation, defended October 1, 2013), which uses AfD as a use case; slides as PDF and on slideshare.
Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker. “Arguments about Deletion: Guiding New Wikipedians in Making Good Arguments” In Computer Supported Cooperative Work and Social Computing. San Antonio, TX, February 23-27, 2013. Slides also available (Slideshare), see writeup below
Jodi Schneider, Alexandre Passant, and Stefan Decker “Deletion Discussions in Wikipedia: Decision Factors and Outcomes.” In WikiSym2012. Linz, Austria, August 27-29, 2012. Slides also available (Slideshare), see writeup below
Jodi Schneider and Krystian Samp “Alternative Interfaces for Deletion Discussions in Wikipedia: Some Proposals Using Decision Factors. [Demo]” In WikiSym2012. Linz, Austria, August 27-29, 2012.
Jodi Schneider, Alexandre Passant, John G. Breslin, “Understanding and Improving Wikipedia Article Discussion Spaces.” In SAC 2011 (Web Track), TaiChung, Taiwan, March 21-25, 2011. Slides also available (Slideshare)
Jodi Schneider, Alexandre Passant, John G. Breslin, “Enhancing MediaWiki Talk pages with Semantics for Better Coordination - A Proposal.” In The Fifth Workshop on Semantic Wikis: Linking Data and People Workshop at 7th Extended Semantic Web Conference (ESWC), Crete, Greece, May 31, 2010. Slides also available (PDF)
Jodi Schneider, Alexandre Passant, John G. Breslin, April 26 & 27 2010. “A Content Analysis: How Wikipedia Talk Pages Are Used.” In WebSci2010, Web Science Conference. Raleigh, NC, 2010. Poster also available (Slideshare)
Overview of Work in Progress
Jodi Schneider “Building a Standpoints Web to Support Decision-Making in Wikipedia.” In Doctoral Colloquium, CSCW2012. Seattle, Washington, February 11-12, 2012. Format is extended abstract (PDF) Slides also available (Slideshare)poster
Jodi Schneider & Alexandre Passant, “Arguments about Deletion: Guiding New Users in Making Good Arguments.” In WebSci2011, ACM Web Science Conference. Koblenz, Germany, June 15-17, 2011. abstract (PDF), poster (PDF), poster (Slideshare)
Related Presentations to the Community
Writeups about my research
Research Newsletter (on WikiSym2012 paper & demo):
- "Deletion Discussions in Wikipedia: Decision Factors and Outcomes" found among other things that "69.5% of discussions and 91% of comments are well-represented by just four factors: Notability, Sources, Maintenance and Bias. The best way to avoid deletion is for readers to understand these criteria." One of the authors also co-presented a demo showing mock-ups of possible "alternative interfaces for deletion discussions in Wikipedia", which would highlight the prevalence of each type of argument (e.g. notability, sourcing...) in a deletion discussion more clearly.
Quote from Research Newsletter (on CSCW2013 paper):
- Winning and losing argument patterns in deletion debates: A paper subtitled "How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups" (presented at the CSCW'13 conference earlier this year) applied the argumentation theory of Doug Walton to classify comments in a corpus of deletion debates on 72 Wikipedia articles (all AfD that were initialized or relisted on January 29, 2011). The four Ireland-based authors emphasize that compared to previous related research which used simpler methods to classify deletion arguments, e.g. based on keywords or policy areas such as notability, their manual analysis is much more thorough and fine-grained, coding AfD comments into 17 categories based on Walton's classification. Among these, the "Rules" and "Evidence" categories are the most popular, making up 36% of AfD arguments. The papers's two other main results are that "familiarity with community norms correlates with [newbies'] ability to craft persuasive arguments" and that "acceptable arguments use community-appropriate rhetoric that demonstrate knowledge of policies and community values while problematic arguments are based on personal preference and inappropriate analogy to other cases" (drawing a direct comparison between Walton's list of problematic arguments and Wikipedia's list of deprecated AfD arguments, e.g. Walton's "Argument from Analogy" corresponds to WP:OTHER - "Other stuff exists").
Quote from Research Newsletter (on SAC 2011 paper):
- Shortcomings in the design of Wikipedia talk pages.
Also covered in Silicon Republic
- Schneider, J., Passant, A. and Decker, S. (2012) Deletion Discussions in Wikipedia: Decision Factors and Outcomes, WikiSym '12 PDF
- Jodi Schneider, Krystian Samp: Alternative Interfaces for Deletion Discussions in Wikipedia: Some Proposals Using Decision Factors. Demo, WikiSym'12, August 27–29, 2012, Linz, Austria. ACM 978-1-4503-1605-7/12/08. PDF
- Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker: Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups CSCW’13, February 23–27, 2013, San Antonio, Texas, USA PDF
- Schneider, Jodi, and Alexandre Passant (2011), Arguments about Deletion: Guiding New Users in Making Good Arguments. In Proceedings of the 2011 ACM Web Science Conference (WebSci '11). PDF
- Schneider, Jodi, Alexandre Passant, and John G. Breslin (2011). Understanding and improving Wikipedia article discussion spaces. In Proceedings of the 2011 ACM Symposium on Applied Computing (SAC '11), 808. New York, NY: ACM Press. DOI • PDF