Research:Understanding Wikipedia Coordination Spaces and Costs

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
Duration:  2010- — 2012-
GearRotate.svg

This page documents a research project in progress.
Information may be incomplete and change as the project progresses.


Key Personnel[edit]

  • Jodi Schneider
  • Alexandre Passant

Project Summary[edit]

Understand conversation and coordination spaces in English Wikipedia, with an aim towards understanding the pain points and prototyping semantic enhancements. This is part of my Ph.D. work in DERI's Social Software Unit; we approach Computer Science with Semantic Web technologies. The goal of my Ph.D. is to provide support for conversation and argumentation in the social web using semantic web technology.

So far this has mainly involved study of article Talk pages and Articles for Deletion. My work aims to support decision-making and identify opportunities for socializing new community members.

Methods[edit]

  • Key methodological features: Content analysis of Wikipedia pages; taxonomy development; interviews with editors; semantic enhancements.

Dissemination[edit]

  • Scholarly papers at conferences
  • Presentations to the community

Wikimedia Policies, Ethics, and Human Subjects Protection[edit]

Interview Study, 2012[edit]

Why[edit]

We have studied the text of AfD discussions, and last year we conducted a round of interviews with experienced Wikipedians and adminstrators. Our overarching goal is to understand:

  1. What are the outcomes of deletion discussions, both for articles and for the community?
  2. What are the purposes and benefits of the current process? What are the "pain points"? What changes, improvements, or support tools could be suggested?

We have read, analyzed, and characterized discussions in two ways, based on decision factors and argument patterns. This helps us to understand research questions such as

  1. What factors contribute to the decision about whether to delete a given article?
  2. When multiple factors are given, what is the relative importance of those factors?
  3. What argument patterns are used in AfD discussions?
  4. Does experience with AfD change the argument patterns?

This helps us understand the process of arguing. Now we would like to better understand the outcomes of the process for the community.

We are particularly interested in understanding how people discover AfD, what their first experience of the process is like, and what they learn about through participating. We are also interested in learning about the problems experienced AfD participants see in the process, and how they determine whether an argument for keeping or deleting an article is a good one.

Other researchers have already noticed friction for newcomers to AfD: Geiger & Ford noticed that AfD participants are "overwhelmingly regulars to the process", and that it is relatively rare (18%) for "the creators of nominated articles [to participate] in that article's deletion discussion." (Participation in Wikipedia's Article Deletion Processes", Wikisym 2011). What we don't know is whether and to what extent this is a problem for the community.

We hope through interviews to find answers to questions like:

  1. What is the experience of participating in AfD like?
  2. What are the benefits to participating in AfD, for an individual? For the community?
  3. What are the costs and challenges of participating in AfD, for an individual? For the community? What are the problems and "pain points" with the current AfD process?
  4. What are the community norms for arguments? Would scaffolding arguments help new and infrequent participants?

This will help us understand the impact and outcomes of discussions, rather than just the discussion style and process.

Who[edit]

Experienced Wikipedians and administrators[edit]

Previously, through using community spaces, I have recruited experienced Wikipedians and administrators for interviews about AfD. I am currently planning follow-up interviews with previous interviewees who gave me permission to contact them as my research progressed, to check my findings and to ask follow-up questions.

Novice Wikipedians/new users of AfD[edit]

I now need to interview Wikipedians who are occasional or new users of Articles for Deletion, to evaluate my findings and get their perspectives.

In terms of the sample, I propose to choose subjects from data that User:Staeiou collected for a WikiSym 2011 poster. (“G1: Patent nonsense”: Participation and Outcomes in Wikipedia’s Article Deletion Processes. R. Stuart Geiger and Heather Ford.).

I would like to interview 10 novices, so I would propose directly targeting subjects by email (I'm considering Talk page requests but this compromises anonymity.). Since I expect a low response rate (are there any figures on what's expected?), I would plan to randomly select 40 people to email, and then choose additional people as needed. (I would welcome suggestions for the content of the message as well; I would expect a brief description of the study, a link to this page, and a request to spend 30 min - 1 hour interview by skype, phone, or IRC.)

Benefits for the Wikimedia community[edit]

  • Understand how community norms and policies are applied, at a very direct practical level
  • Understand the differences in values and policy articulation between novice and experienced Wikipedians
  • Suggest opportunities for streamlining decision-making while maintaining its quality

Time Line[edit]

Ph.D. research from October 2009-October 2013.

Funding[edit]

Science Foundation Ireland SFI/09/CE/I1380 (Líon2)

References[edit]

Related Publications[edit]

“Identifying, Annotating, and Filtering Arguments and Opinions in Open Collaboration Systems” (PhD dissertation, defended October 1, 2013), which uses AfD as a use case; slides as PDF and on slideshare.

Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker. “Arguments about Deletion: Guiding New Wikipedians in Making Good Arguments” In Computer Supported Cooperative Work and Social Computing. San Antonio, TX, February 23-27, 2013. Slides also available (Slideshare), see writeup below

Jodi Schneider, Alexandre Passant, and Stefan Decker “Deletion Discussions in Wikipedia: Decision Factors and Outcomes.” In WikiSym2012. Linz, Austria, August 27-29, 2012. Slides also available (Slideshare), see writeup below

Jodi Schneider and Krystian Samp “Alternative Interfaces for Deletion Discussions in Wikipedia: Some Proposals Using Decision Factors. [Demo]” In WikiSym2012. Linz, Austria, August 27-29, 2012.

Jodi Schneider, Alexandre Passant, John G. Breslin, “Understanding and Improving Wikipedia Article Discussion Spaces.” In SAC 2011 (Web Track), TaiChung, Taiwan, March 21-25, 2011. Slides also available (Slideshare)

Jodi Schneider, Alexandre Passant, John G. Breslin, “Enhancing MediaWiki Talk pages with Semantics for Better Coordination - A Proposal.” In The Fifth Workshop on Semantic Wikis: Linking Data and People Workshop at 7th Extended Semantic Web Conference (ESWC), Crete, Greece, May 31, 2010. Slides also available (PDF)

Jodi Schneider, Alexandre Passant, John G. Breslin, April 26 & 27 2010. “A Content Analysis: How Wikipedia Talk Pages Are Used.” In WebSci2010, Web Science Conference. Raleigh, NC, 2010. Poster also available (Slideshare)

Overview of Work in Progress[edit]

Jodi Schneider “Building a Standpoints Web to Support Decision-Making in Wikipedia.” In Doctoral Colloquium, CSCW2012. Seattle, Washington, February 11-12, 2012. Format is extended abstract (PDF) Slides also available (Slideshare)poster

Jodi Schneider & Alexandre Passant, “Arguments about Deletion: Guiding New Users in Making Good Arguments.” In WebSci2011, ACM Web Science Conference. Koblenz, Germany, June 15-17, 2011. abstract (PDF), poster (PDF), poster (Slideshare)

Related Presentations to the Community[edit]

“Dynamics of Wikipedia Talk pages: serving the article, showing the community”,WikiMania 2010, Gdańsk, Poland, July 2010 Abstract, Slides (Slideshare)

Writeups about my research[edit]

Research Newsletter (on WikiSym2012 paper & demo):

  • "Deletion Discussions in Wikipedia: Decision Factors and Outcomes"[1] found among other things that "69.5% of discussions and 91% of comments are well-represented by just four factors: Notability, Sources, Maintenance and Bias. The best way to avoid deletion is for readers to understand these criteria." One of the authors also co-presented a demo showing mock-ups of possible "alternative interfaces for deletion discussions in Wikipedia"[2], which would highlight the prevalence of each type of argument (e.g. notability, sourcing...) in a deletion discussion more clearly.

Quote from Research Newsletter (on CSCW2013 paper):

  • Winning and losing argument patterns in deletion debates: A paper subtitled "How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups"[3] (presented at the CSCW'13 conference earlier this year) applied the argumentation theory of Doug Walton to classify comments in a corpus of deletion debates on 72 Wikipedia articles (all AfD that were initialized or relisted on January 29, 2011). The four Ireland-based authors emphasize that compared to previous related research which used simpler methods to classify deletion arguments, e.g. based on keywords or policy areas such as notability, their manual analysis is much more thorough and fine-grained, coding AfD comments into 17 categories based on Walton's classification. Among these, the "Rules" and "Evidence" categories are the most popular, making up 36% of AfD arguments. The papers's two other main results are that "familiarity with community norms correlates with [newbies'] ability to craft persuasive arguments" and that "acceptable arguments use community-appropriate rhetoric that demonstrate knowledge of policies and community values while problematic arguments are based on personal preference and inappropriate analogy to other cases" (drawing a direct comparison between Walton's list of problematic arguments and Wikipedia's list of deprecated AfD arguments, e.g. Walton's "Argument from Analogy" corresponds to WP:OTHER - "Other stuff exists").

Quote from Research Newsletter (on SAC 2011 paper):

  • Shortcomings in the design of Wikipedia talk pages.
    A diagram of deletion processes in the English Wikipedia, from Schneider et al., 2011 [4]
    Researchers from the National University of Ireland, Galway presented some work in progress from a project aimed at understanding Wikipedia coordination spaces and costs. In a paper presented earlier this year at SAC '11 the authors discuss the results of a small series of semi-structured user interviews with Wikipedia administrators and editors.[5] The results point at a number of drawbacks in the design of Wikipedia talk pages, suggesting that editors find it hard to keep up-to-date with temporally sparse discussions that are often scattered across multiple pages. The interviews also suggest that talk pages often become the target of support requests by new editors that go unnoticed. The lack of connection between discussions and the article itself (e.g. links between threads and specific sections or topics of the article) also emerges as one of the main weaknesses of Wikipedia talk pages. In the remainder of the paper the authors introduce a lightweight solution to allow effective categorization of comments posted on article talk pages by semantically enriching them with an RDF markup. This markup can then be exposed to end users with the aid of a JavaScript bookmarklet, manipulated and exported via SPARQL, and potentially used to generate granular notifications. In a poster presented last month at WebSci '11, the same team of researchers gives an overview of work in progress on AfD discussions and illustrates with a diagram the complexity of deletion discussions and procedures in the English Wikipedia. [4]


Also covered in Silicon Republic

Related Images[edit]

Deletion process on en-WP

External links[edit]

Related[edit]

Contacts[edit]

I welcome your input and feedback. Please feel free to leave a message on my Talk page or email me directly at jschneider@pobox.com.

References[edit]

  1. Schneider, J., Passant, A. and Decker, S. (2012) Deletion Discussions in Wikipedia: Decision Factors and Outcomes, WikiSym '12 PDF Open access
  2. Jodi Schneider, Krystian Samp: Alternative Interfaces for Deletion Discussions in Wikipedia: Some Proposals Using Decision Factors. Demo, WikiSym'12, August 27–29, 2012, Linz, Austria. ACM 978-1-4503-1605-7/12/08. PDF Open access
  3. Jodi Schneider, Krystian Samp, Alexandre Passant, Stefan Decker: Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups CSCW’13, February 23–27, 2013, San Antonio, Texas, USA PDF Open access
  4. a b Schneider, Jodi, and Alexandre Passant (2011), Arguments about Deletion: Guiding New Users in Making Good Arguments. In Proceedings of the 2011 ACM Web Science Conference (WebSci '11). PDF Open access
  5. Schneider, Jodi, Alexandre Passant, and John G. Breslin (2011). Understanding and improving Wikipedia article discussion spaces. In Proceedings of the 2011 ACM Symposium on Applied Computing (SAC '11), 808. New York, NY: ACM Press. DOIPDF Open access