Research:How role-specific rewards influence Wikipedia editors’ contribution

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search
Created
20:36, 25 October 2018 (UTC)
Duration:  2018-10 — 2019-2
Incomplete-document.svg

This page is an incomplete draft of a research project.
Information is incomplete and is likely to change substantially before the project starts.


Our research examines whether labeling and rewarding editors for their role-specific behaviors increase their positive behaviors on Wikipedia. Doing so could help Wikipediac retain editors and encourage them to do needed work. Previous experimental research has shown that when editors who contribute heavily to Wikipedia receive public, informal rewards, such as barnstars, they subsequently contribute more [1][2]. However, this prior research examined only the diffuse, motivational effects of the rewards, by getting editors to contribute more. But it did not examine its specific, cognitive effects, in which the label helps these editors decide what to do. Our previous research has shown that many editors specialize in the types of work that they do in Wikipedia, such as adding substantive content or performing copy-editing tasks, and that one can computationally identify the roles they occupy[3][4]. Our new, proposed research will examine whether labeling and rewarding them for the type of work they perform and the roles they occupy (e.g., as a substantive editor or copy editor) will have effects beyond simply telling them that they are good Wikipedians. If activity-specific barnstars encourage the actions that they reward, these barnstars provide Wikipedia a mechanism to encourage prolific editors to perform the distinct types of work they are most interested and skill in performing.

Specifically, we plan to a conduct random-assignment experiment in which editors who are eligible to receive an role-specific barnstar will either receive no barnstar, receive a generic barnstar that rewards them for contributing to Wikipedia generally or receive a barnstar that is specific to the type of work they do most frequently. Then we will investigate whether editors who receive barnstars show higher productivity and greater commitment to Wikipedia compared to those who receive no barnstars. In addition, we will examine whether editors who receive activity-specific barnstars subsequently specialize more in the activities for which they were rewarded compared to those who receive generic barnstars. This research will help us understand how informal labels sustain and direct volunteer effort in online production communities.

Methods[edit]

Our experiment will be conducted on the English Wikipedia. First we will run role-identification algorithms to identify approximately 1200 editors who heavily add substantive content to Wikipedia or who perform copy editing work. These comprise a pool of active editors who are eligible for receiving activity-specific barnstars on Wikipedia. Within each role type, we will randomly give 1/3 role-specific barnstars and give 1/3 of them generic barnstars by posting reward messages on their user talk pages on Wikipedia and we will not give barnstars to the remaining third.

Experimental design (N)
Barnstar condition/

Role type

Control

No barnstar

Generic

Barnstar

Role-specific

Barnstar

Copy editor 200 200 200
Substantive editor 200 200 200
Total 400 400 400

Inclusion criteria:

  • Editors who are active in the last three months
  • Editors in the top N% of those making edits adding substantive content or performing copy-editing actions

Exclusion criteria:

  • Remove users who had administrative or elevated privileges
  • Remove editors who have previously received a barnstar
  • Remove editors who are or control bots

We plan to observe the behavior changes among editors who receive the informal rewards, in order to investigate whether specific informal rewards positively influence editors' commitment to Wikipedia and production. To examine this, we will collect edits made by editors who received the different types of barnstars and compare their work with edits from editors who do not receive barnstars.

Our barnstar-giving random experiment will involve three conditions:

  1. Control group: 400 editors who are active contributors will not receive any barnstars
  2. Generic barnstar group: 400 editors who are active contributors will receive barnstars that are generic (e.g., The Original Barnstar).
  3. Role-specific barnstar: 400 editors who are active contributors will receive barnstars that label and reward them for the specific types of editing behavior they perform most frequently. The half who add substantive content to articles will receive the Tireless Contributor Barnstar and the half who make copy-editing edits will receive the Copyeditor's Barnstar.

Here is an example message that will be given to editors in Generic barnstar group:

{{subst:The Original's Barnstar|1=. You have been doing a great job of improving Wikipedia. In the last 3 months, you have made XX edits. Wikipedia needs your contributions. Keep up the good work! Diyiy (talk) 20:36, 25 October 2018 (UTC)}}

Here is an example message that will be given to editors in Role-specific group :

{{subst:The Copyeditor's Barnstar|1=. You have been doing a great job of improving Wikipedia. In the last 3 months, you have made XX edits and the majority of them were copy editing edits fixing errors. Wikipedia needs these kinds of contributions. Keep up the good work! Diyiy (talk) 20:36, 25 October 2018 (UTC)}}

Research by Restivo and Van De Rijt suggest that an N of 200 editors per cell will provide enough power to reliably distinguish the effects of barnstars on subsequent behavior [1][2].

Note that we used the account of Diyiy in the example message above. In the final deployment, we will seek help from some highly experienced Wikipedian editors to help post the barnstar message. We plan to post around 50 barnstar messages per week.

Timeline[edit]

  • Nov 1st - Nov 30st, 2018: Brainstorm project ideas/proposal, and applying for IRB approval and Wikipedia's approval.
  • Nov 1st - Nov 30th, 2018: Identify candidate editors and collecting their edits; applying our prior edit intention framework [3] to identify editors' skills.
  • Dec 1st, 2018: Give barnstars to editors who are assigned to different experimental conditions (receiving no barnstar, receiving generic barnstar and receiving skill-specific barnstar)
  • Dec 2nd, - Dec 31st, 2018: Observe editors' behavior changes
  • Jan 1st - Jan 15th, 2019: Analyze those editors' editing behaviors in Dec 2018, and compared it to their editing behaviors in Nov of 2018.
  • Jan 15th - Jan 31st, 2019: Summarize results as a link on the wikimedia Research:Projects page and wrap up the project
  • Jan 15th - March 10th, 2019: Write up the manuscript for external publication

Policy, Ethics and Human Subjects Research[edit]

We have submitted a protocol about this project to the institutional review board (IRB) at Carnegie Mellon University. It is currently under review..

Results[edit]

N/A

References[edit]

  1. a b Restivo, Michael, and Arnout van de Rijt. "No praise without effort: experimental evidence on how rewards affect Wikipedia's contributor community." Information, Communication & Society 17, no. 4 (2014): 451-462.
  2. a b Restivo, Michael, and Arnout Van De Rijt. "Experimental study of informal rewards in peer production." PloS one 7, no. 3 (2012): e34358.
  3. a b Yang, Diyi, Aaron Halfaker, Robert Kraut, and Eduard Hovy. "Identifying semantic edit intentions from revisions in wikipedia." In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2000-2010. 2017.
  4. D Yang, A Halfaker, RE Kraut, EH Hovy, "Who Did What: Editor Role Identification in Wikipedia." International Conference on Weblogs and Social Media (ICWSM), 2016, 446--455.

Contacts[edit]

Diyi Yang, Diyi Yang, diyiy@cs.cmu.edu

Robert E. Kraut, Robertekraut, robert.kraut@cmu.edu