Research talk:How role-specific rewards influence Wikipedia editors’ contribution

Why remove admins and people who previously received a barnstar?

I'm not sure I see that explained. --Halfak (WMF) (talk)

@Halfak (WMF) Good point. Here, we want to only focus on the effect of receiving the first reward. In this way, we can control for other factors and also make our study comparable to prior work that tested the effect of receiving the first barnstar. --Diyiy (talk)

It matters who gives you a barnstar

It looks like Diyiy will post the barnstar messages. When I receive a barnstar, I will certainly check out who awarded me the barnstar when interpreting the meaning to me. How do you expect the person awarding the barnstar to affect the results? Maybe people won't take Diyiy's barnstars seriously. --EpochFail (talk) 16:04, 3 December 2018 (UTC)

@Halfak (WMF) Yes, we won't post the messages using the account of Diyiy (talk). As we discussed over emails, we would like to collaborate with some highly experienced Wikipedia editors and ask them to help us post the barnstar messages. If you could point us to some editors or ways to recruit editors, that will be great! We're also considering posting the messages using Robert Kraut's account. --Diyiy (talk)

Giving barnstars to editors who deserve them

How will you ensure that the editors in fact deserve the barnstars you would award them? Can you produce a pilot sample of editors who might be identified for such rewards? How will you avoid vandals and other types of problematic editors? --EpochFail (talk) 16:05, 3 December 2018 (UTC)

@Halfak (WMF) We will only give the barnstars to editors who are among the top %N (N could be 1, 5) most productive editors in the last month. We have a list of editors who are candidates of such rewards - will share this with you soon. We selected editors not only based on their number of edits but also based on their type of edits. We used the semantic edit intention taxonomy to predict the semantic intention of editors' every edit. If there are any edits predicted as vandalism, we removed such users from our candidate list. We'll also double check whether an editor has any vandalism edits in the last month using Wikipedia Vandalism API. --Diyiy (talk)

800 barnstar awards could be disruptive

How will you ensure that your awards don't cause a big event? Can you spread out the posting of the awards over time? E.g. only post 50 per day for ~ a week? --EpochFail (talk) 16:07, 3 December 2018 (UTC)

@Halfak (WMF) This is a great suggestion - We will post around 50 per day for one or two weeks for this barnstar experiment. --Diyiy (talk)

These are two hypotheses with different premises

There are two studies that can be done here:

• You can analyse barnstars (and other awards) being handed out around the project and study their correlation with editor retention.
• You can also give out new barnstars. However, as many have already pointed out on this page and the related en.wp discussion, this will be an artificial situation, because the recipients will be aware that they are part of an experiment.

You might want to look out for the w:Lucas critique of interventions. Deryck C. 18:59, 20 December 2018 (UTC)

@Deryck Chan: We plan to conduct both a correlational study (your first suggestion) and an experiment study (by giving new barnstars). When giving barnstars to editors, we will not tell people that they are part of an experiment. Since we have posted this proposal with Wiki community, we did include one exclusion criteria - remove people who have been involved in any discussions around this proposed work. -Diyiy (talk) 02:24, 29 December 2018 (UTC)

@Diyiy: we will not tell people that they are part of an experiment.

I am glad to see that at least on this occasion, you specifically acknowledge that you are conducting a social experiment, rather than hiding behind the broader and more cuddly term of "research".

I am also pleased to see that you acknowledge that you plan is based on deceit Wikipedia editors, although you do not use that word.

I posted[1] at length about this on en.wp: here.

So I invite you to relabel your proposal, placing the words "social experiment" in the heading, and explicitly acknowledging in the lede both a) the lack of consent, and b) your sponsorship by Facebook. Your failure to prominently disclose that sponsorship is a very big breach of the principle of transparency, and I hope that you will remedy it promptly. --BrownHairedGirl (talk) 16:37, 30 December 2018 (UTC)

I think you're mis-defining copy-editing

Your "copy-editing" barnstar is going to say "the majority of them were copy editing edits fixing errors". That's only a tiny part of copy-editing Wikipedia articles. Speaking as someone who really got her start on Wikipedia as a copy editor (including copy-editing a lot of articles during or in preparation for featured or good article candidacies), the core of copy-editing is in things like:

• improving readability
• improving word selection
• arranging the article to flow most effectively
• considering the most beneficial use of reference sources
• improving formatting and layout
• ensuring that best practices are followed in accord with WP:MOS and other editorial policies and guidelines

It's not just fixing the typos and grammar mistakes - although most copy editors will start off there. But I have a feeling you're going to struggle to find enough editors meeting your criteria to come up with statistically significant results. Risker (talk) 01:55, 21 December 2018 (UTC)

@Risker: thanks for sharing this comprehensive definition for copy-editing. We propose to use "fixing the typos and grammar mistakes, improving the tone or punctuation" as a narrow definition for "copy-editing" for now. These are a subset of tasks that a copy-editor will do, but also relatively more characteristic tasks that copy editors do. Also, our proposed algorithms can predict fixing errors or improving punctuation with relatively higher accuracies, however, detecting edits like ensuring the best practices and guaranteeing the editorial policies are relatively challenging for current machine learning algorithms because of limited training corpus and dependence on external policies. --Diyiy (talk) 02:35, 29 December 2018 (UTC)