Research talk:How much do Wikipedians in the US value editing Wikipedia?

From Meta, a Wikimedia project coordination wiki

Survey questions?[edit]

Could you post a draft of your survey questions? That will help myself and others work out if you'll be collecting any sensitive information. --EpochFail (talk) 21:43, 24 October 2017 (UTC)[reply]

Thanks for the suggestion. We are working on finalizing the survey and we will post it here for the community to review before we launch the study. --avi_gan (talk) 22:58, 26 October 2017 (UTC)[reply]
Posted --avi_gan (talk) 20:33, 6 July 2018 (UTC)[reply]

Invitation[edit]

Please post a draft of the invitation you'd like to send to editors. I can help you work on this to make sure it communicates the project clearly. --EpochFail (talk) 21:43, 24 October 2017 (UTC)[reply]

Thanks for your help. We would love to get feedback on the invites before we send those out. We will update the article with the exact text of the invite. --avi_gan (talk) 22:58, 26 October 2017 (UTC)[reply]
Posted --avi_gan (talk) 20:33, 6 July 2018 (UTC)[reply]

Initial comments on design[edit]

  • Under the valuation questions you need to add a question about the valuation of not editing for one month if above $100. I don't know if this messes you up as far as IRB approval goes. For dedicated edit-every-day contributors, I don't think $100 is going to cut it. I was trying to imagine what I would say and figured it would have to be more in the $1,000 to $2,000 range for me even to seriously consider it. I edit probably 30-60 hours a week and as of now am the 1516th most active English Wikipedian by edit count. I realize that messes with your design as far as actually paying people, since you probably don't have the money to give me a thousand bucks, but it's the metric you're really going for, and you are way under valuing it for the comparatively small group of people who contribute most to the project.
  • For the valuation of not contributing, you probably also need to take into account whether editors are only active on one project, or multiple ones. If you pay me $1,500 not to edit the English Wikipedia, then I'm just going to take your money and go edit the English Wikiquote or Wikimedia Commons instead. I spent most of my day today on Commons and not actually doing very much on the English Wikipedia at all. The valuation for people active on multiple projects is likely to be lower.
  • You probably also need to ask how much time people spend on non-English-Wikipedia projects, because now that I've typed this, I realize that my time estimation above is all projects, and not just the English Wikipedia.
  • Please collect data on religion and ethnicity if you can. This really has nothing whatsoever to do you with your study design. But the WMF simply doesn't collect this information at all, and as far as I know (and I've done a bit of digging to try to find it) it simply doesn't exist currently. Even in a modest sample size.
  • No posting on talk pages has ethical concerns. Users may face sanctions if they are unresponsive. This is likely a nuance IRB did not consider.

Feel free to ask any follow up questions. Please ping me by copying and pasting {{ping|GreenMeansGo}} along with a new signature by adding ~~~~. I don't actively follow anything on meta.wiki, so I'm not likely to see a response otherwise. GMGtalk 22:12, 6 July 2018 (UTC)[reply]

All excellent points User:GreenMeansGo, thank you for the suggestions.
And yes, we cap our rewards at $100 due to budget reasons. I will edit the survey to add another question for those whose valuation is greater than $100. I will ask them to enter their $ valuation. However this figure will be hypothetical, but still will give us some information about how much we are underestimating for the most active Wikipedians.
We also want to measure the valuation for all projects and not just the English Wikipedia. I will edit the instructions to make this more clear. Same thing with the time, I will edit the question to include all projects and maybe add a separate question for English Wikipedia.
I will also add optional questions on religion and ethnicity at the end of the survey. All anonymized aggregate results will be posted on the project page, so you will be able to see the breakdown of editors who responded to our study by religion and ethnicity.
I will add instructions to say that if someone faces sanctions due to not posting on talk pages, they get an exception and will still get the reward.--avi_gan (talk) 02:27, 25 July 2018 (UTC)[reply]
To avoid overly generous interpretations of "facing sanctions", you'll probably want them to stop editing and contact you before posting on talk pages. There are very few discussions on wiki that are so urgent that they can't wait 24 hours for an editor to get online again, but if you're still editing elsewhere without replying to, say, ANI, then people will be annoyed. WhatamIdoing (talk) 15:24, 25 July 2018 (UTC)[reply]

What does it mean to stop?[edit]

This says, "In the main part of the survey, we will ask how much money we would have to give a participant so that they stop editing Wikipedia for 1 month."

I'd like to echo User:GreenMeansGo's question about "stop editing Wikipedia" means to you. Does this mean stop editing just the English Wikipedia? Or stop editing all projects? Does it 'count' if I spend the month drafting new articles offline? Do I have to stop clicking the 'Thank' button for people who do useful things? Do I have to stop reading the talk pages? A lot of the value for me comes from helping wiki-friends. Is it okay if I follow the discussions, and, if I see a wiki-friend answering a question, I send an e-mail message with my answer? All of which adds up to: Are you paying me to stop using a piece of software, or are you paying me to disconnect from the most important online community in the world? WhatamIdoing (talk) 01:21, 9 July 2018 (UTC)[reply]

Good point regarding the scope of the project. Ideally we want to measure how much value you get from being part of the community. So this includes editing all projects and not just the English wikipedia. I will edit the instructions of the study to make it clear.
However, there is no way for us to check if you draft articles offline for a month, emailed someone an answer etc. So although we would prefer that the subject not do this (to get their entire valuation for being part of the community), we cannot do much about it. Our $$ figures are therefore underestimated compared to the "true" estimates.--avi_gan (talk) 02:17, 25 July 2018 (UTC)[reply]
Thanks, that's clearer. From the POV of someone who has written too many obituaries, I ask that the instructions you send them permit one last edit – namely, to please post a note about taking a break for a while. w:en:Template:Attempting wikibreak might be a particularly appropriate method. We sometimes worry when our wiki-friends disappear without notice. You should probably also decide whether you need a policy on self-requested account blocks (either to encourage, discourage, or not care about them). WhatamIdoing (talk) 15:20, 25 July 2018 (UTC)[reply]
Excellent point. Will add an instruction to let selected editors post one last message before disappearing for a month.--avi_gan (talk) 18:17, 25 July 2018 (UTC)[reply]

Question about permanent deletion[edit]

The suggestion in the question that the user abandon their current account and form a new one without publicly linking the two is questionable in regard to Wikipedia's policies on sock puppetry. Compassionate727 (T·C) 22:00, 7 July 2018 (UTC)[reply]

Ewww. I just saw this item on the questionnaire. This is extremely ugly even in the best of circumstances. I would be quite unhappy to discover I wasted hours investigating someone who had been paid to make an undisclosed-new-account. Alsee (talk) 12:05, 8 July 2018 (UTC)[reply]

Will you permanently delete your current Wikipedia user profile in exchange for $x?

You will have to change your user name and request to delete your current user page as noted here. Your future contributions to Wikipedia will be recorded under your new user name. If selected to get the cash amount, you agree that you will not refer to your previous user name in your new user page. Note that this is a permanent change and not just for 1 month.

This item needs another look. For one thing, the w:en:Wikipedia:Changing username process means that all of my prior edits would be recorded under the new username, too. I think you're probably looking for the w:en:Wikipedia:Clean start process: stop using (but don't 'delete', which isn't possible) the old account, create a new one, following the private notification process but without publicly linking the two (just on the user page? So I could still tell all my friends on their user talk pages? Or at least tell them that I'm going to do this, so they'll know to be on the lookout for a new me? How different does my new username need to be?). WhatamIdoing (talk) 01:34, 9 July 2018 (UTC)[reply]
Sorry for not using the proper terminology. By permanently deleting, I did mean w:en:Wikipedia:Clean start. The idea behind this question is to understand how much of the editors motivation comes from their reputation acquired through their username. If someone is chosen to do this action, we will instruct them to create a new user name very different from their old one and also tell them that we prefer that they not tell their friends about their name change on talk pages. However, we have no way to verify this over a long period, so to make things more practical, we will wait for compliance for a month and give them the reward. Afterwards, they are free to post on their friends' talk pages. --avi_gan (talk) 02:11, 25 July 2018 (UTC)[reply]

Comments[edit]

The voucher value has to be discounted by some unknown/variable amount. I'd personally value the voucher below 50 cents on the dollar, and there is a rapidly declining marginal utility in getting multiples of items. If someone valued Wikimedia store items at list-price they could have bought them already.

The terms for the random draw do-not-edit winners are contradictory, confusing, or even potentially misleading. One sentence make a rather opaque statement that the "amount will be determined through the Becker–DeGroot–Marschak method", and later sentences plainly state 1 in 50 participants will be able to claim and receive the amount they listed for not editing. Taking the latter statement literally, it would be an invitation to simply put down a billion dollars. After studying the link to Becker–DeGroot–Marschak, it seems you're actually planning to cap the payout to some undisclosed amount. That makes far more sense, however it directly contradicts the other sentences. It means that possibly no one will actually be offered the claimed amount. I think this paragraph needs to be re-written for clarity.

  • Edit: I now see that the survey will limit the don't-edit value to $100. The description on the page here really should be improved.

I don't mean to make a big deal out of this last point, but I do question the appropriateness of approving a research project which pays people to not-edit. The impact may be limited at 20 people, but it's clearly negative. It's either new content not being created, or it's support work being dropped onto others. Alsee (talk) 11:04, 8 July 2018 (UTC)[reply]

Or it's a clever way to find out whether anyone's answers are accurate.
User:EpochFail, have you all talked about the effect of paying the $100 at the end of the month, rather than at the start? Losing something you've got ($100 already in your pocket) is generally a bigger deal than losing the possibility of getting something in the future. WhatamIdoing (talk) 01:38, 9 July 2018 (UTC)[reply]
WhatamIdoing, this is an interesting idea. I know that I'd discussed this idea with the Jana and Avi, but I can't remember why they didn't like it. I've pinged them IRL to get involved in this discussion so maybe they can remind me :) --EpochFail (talk) 16:00, 20 July 2018 (UTC)[reply]
Well, getting the money back later could be a significant barrier. It's one thing to hand me a $100 bill in person; it's another thing to send it to the PayPal account of a person you don't really know, and hope that you can still figure out how to reach me later if I violate the terms of our agreement (or "when I violate the terms", if we're talking about me; I seriously doubt that I could remember not to click the Edit button that long). WhatamIdoing (talk) 16:06, 20 July 2018 (UTC)[reply]
Thanks WhatamIdoing and User:EpochFail for the comments. Yes we will be paying the rewards at the end of the month and not at the start. Rewards will be given after we verify that the subject complied with the chosen selection.
And yes you are right, I need to improve the text of the instructions. 1 in 50 will be selected to be able to win the reward and for the selected person, one of their choices will be randomly chosen. If that choice is suppose: $35 and give up all editing for 1 month, we will contact them to inform them about the start date. At the end of the month, they will get the reward once we verify they have not edited during the month. I will edit the instructions to make it more clear.
We do this procedure to get "true" responses as opposed to hypothetical survey responses. My previous research shows that subjects reveal lower valuations when asked a hypothetical question, but if they actually have a chance to get real $, they reveal higher numbers which is probably closer to their true valuation.
To minimize disruption to the community (and also to keep budgets reasonable), we contact 1 in 50 subjects and even that subject has to give up editing Wikipedia only if they selected to give it up for $X in one of their responses that we randomly pick. --avi_gan (talk) 02:03, 25 July 2018 (UTC)[reply]

List[edit]

Is there a list I can sign my name to be considered for one of the 1000 editors? enL3X1 ¡‹delayed reaction›¡ 23:36, 8 July 2018 (UTC)[reply]

@L3X1: I doubt it, because allowing people to ask to participate would probably introduce some sort of sampling bias. Jc86035 (talk) 06:49, 13 July 2018 (UTC)[reply]
@L3X1: There is no such list. We will randomly pick editors to participate in the study in order to get a representative sample of the editor population. --avi_gan (talk) 02:04, 25 July 2018 (UTC)[reply]
@L3X1: We have now made the survey open to all US based editors. Please fill it out here if you are interested. Thanks! --avi_gan (talk) 01:34, 28 November 2018 (UTC)[reply]
Avi gan Thank you! enL3X1 ¡‹delayed reaction›¡ 03:19, 28 November 2018 (UTC)[reply]

Request[edit]

Just dropping a note that it would be much appreciated to get a courtesy ping whenever these results are published. GMGtalk 13:26, 7 November 2018 (UTC)[reply]

Yes noted, I am sending out the invites for the survey this week, results should be available in December/ January 2018. I will post a note here when they are updated on the meta page. --avi_gan (talk) 13:56, 7 November 2018 (UTC)[reply]

Survey now open to all US based Wikipedians[edit]

Hello US based Wikipedians

We're working to measure the value of Wikipedia in economic terms. We want to ask you some questions about how you value being able to edit Wikipedia.

Our survey should take about 10-15 minutes of your time. We hope that you will enjoy it and find the questions interesting. All answers will be kept strictly confidential and will be anonymized before the aggregate results are published. Regretfully, we can only accept responses from people who live in the US due to restrictions in our grant-based funding.

As a reward for your participation, we will randomly pick 1 out of every 5 participants and give them $25 worth of goods of their choice from the Wikipedia store (e.g. Wikipedia themed t-shirts). Note that we can only reward you if you are based in the US.

Click here to access the survey: https://mit.co1.qualtrics.com/jfe/form/SV_7PtejPtV7DFvev3

Thanks

Avi

Researcher, MIT Initiative on the Digital Economy --avi_gan (talk) 01:32, 28 November 2018 (UTC)[reply]

Suggest we close this survey[edit]

Firstly this survey seems to assume that people have only one account. Aside from the contentious issue of undeclared accounts there are lots of people who have multiple declared accounts. Most people who take part in outreach will have a clean account which follows the defaults and lacks the extra tabs that come with Twinkle or adminship. Secondly communication with others is not just a matter of talkpages. Email, edit summaries and the Thanks button are all communication methods that in combination means that one could largely step away from talkpages. OK you would have to leave blocking, welcoming and speedy deletion tagging to others - If I was offered such a deal I would consider giving up the tools for a month. Offering people the same range of money to do a cleanstart as to take a month's break is a tad insulting. But most seriously, the survey treats editing Wikipedia as a negative thing like alcoholism or obesity that you might bribe people to pause or give up on. I do not think it in the interests of the community to have such a survey run on Wikipedians. We have a problem with editor retention on this site, bribing editors to stop or pause their editing is not a good idea. WereSpielChequers (talk) 16:36, 28 November 2018 (UTC)[reply]

Thanks for the comments. We do not treat editing Wikipedia as a negative thing at all. In fact we are trying to show that it creates a huge positive externality to the society and the value it creates is not captured in standard economic measures. To put a $ figure on the economic surplus created by Wikipedia, we have to ask questions of the type we are asking in our survey. The survey is opt in, and only a small fraction of the respondents (1 in 50, maximum of 20) will be actually given the chance to get money to give up editing. They can choose to opt out at anytime. This is to ensure that the responses to the survey are not biased and are reflective of editors true valuations (previous research shows that if you just ask hypothetical questions without real incentives tied to responses, then results will be biased). Happy to discuss further. --avi_gan (talk) 00:18, 29 November 2018 (UTC)[reply]