Research talk:Dynamics of Online Interactions and Behavior

From Meta, a Wikimedia project coordination wiki

Links[edit]

Description[edit]

I think I understand most of this research page, but I'll admit to not getting all of it. Since I've got some experience of surveys both on and off wiki I suspect that if part of this goes above my head there will be many Wikipedians who are in the same boat. Now that might not matter if this was just being seen by a group of editors who are interested in research, but you want lots of people to read it. So my suggestion would be to re-write it so that it is understandable to most people. I.e. This read like a pitch to a university ethics committee; You are now pitching to a crowd that includes people with High school education and also many for whom English is not their native tongue. WereSpielChequers 19:59, 23 June 2011 (UTC)[reply]

Targetting[edit]

Who is this targeting? I assume the watchlist limits this to editors not readers, but is this for all Wikimedians, just the English language projects or just EN wiki? On the one hand the narrower the targetting the longer the banner would need to be displayed to get 2,000 responses, on the other hand we certainly can't have an English language survey broadcast to a Catalan or Korean wiki. WereSpielChequers 14:56, 26 June 2011 (UTC)[reply]

The target is only English Wikipedians, so any banner would only run on EN to logged in editors. Steven Walling at work 22:27, 27 June 2011 (UTC)[reply]
OK so we don't need to worry about translations, but if this is the trend and most surveys target EN then all the more reason to shift to the Omnibus strategy before editors get fed up of surveys. WereSpielChequers 16:46, 29 June 2011 (UTC)[reply]

Can you please specify clearly in the page what logged-in users are seeing the banner, for people who can't read the code? Thank you, Nemo 23:55, 8 December 2011 (UTC)[reply]

Omnibus surveys[edit]

My preference would be that instead of allowing assorted overlapping research programs to run this sort of survey, we instead turn our own survey into an omnibus survey that adds questions from researchers like this to the main survey, and then gives them an extract of the answers to their questions plus other core ones. That way we keep the spam down and only ask our editors once a year what their age, education, preferred operating system and so forth. I suspect that this would also give researcher the large and robust response that they want and most importantly would minimise spamming of editors. WereSpielChequers 14:56, 26 June 2011 (UTC)[reply]

(Copying here a note that I previously posted on RCom-l): regardless of the value of an omnibus survey as a general solution, it obviously would not apply to the Berkman study, which is mostly a behavioral game theory experiment: the post-experiment survey is only a part of the study, and not the most important. --DarTar 02:39, 16 December 2011 (UTC)[reply]
Actually the Berkman study would be conceptually easy to incorporate into an Omnibus survey. though some programming might well be required. Easy that is unless they required identifiable editor information such as username or editcount which wouldn't be available to them as part of an Omnibus Survey. On a broader note this does raise the issue of whether some surveys would still need to be standalone, though I'm not convinced that this would. I will amend my proposal to the effect that if we go down the route of an Omnibus Survey we might still have standalone surveys that didn't fit that, for example there may be a reason to run them immediately before and after a software upgrade, or we as a community might decide that we want to survey people now rather than weight several months for the next Omnibus survey. The key thing to my mind is that the Omnibus survey should be the default and any extra surveys would need to justify their status as additional ones, otherwise we need some other way to limit the surveying of the community to levels that almost all editors can tolerate. WereSpielChequers 13:01, 16 December 2011 (UTC)[reply]

Purposes[edit]

Currently the page says both that the data will only be used for research purposes and that the data will be available for any purpose. I'm assuming that the intent is that the anonymised statistical results will be available for any purpose, but the individual level data will only be used for research purposes. If so can this be clarified please. WereSpielChequers 16:43, 29 June 2011 (UTC)[reply]

The "only for research purposes" statement obviously concerns the use that we (as an academic research team) will make of the data. We are not speculating here about the innovative use that Wikimedians could make of the anonymised data once we release it on the public domain, although I must say that I don't see what kind of other purposes it could serve. SalimJah 17:38, 29 June 2011 (UTC)[reply]
In that case would you mind rephrasing it to say that? Currently we have two contradictory statements on one page. If the intent is that "The full set of questionnaire answers will only be used for research purposes, and to create an anonymised statistical data set that will be released on the Internet on an open licence" then may I suggest you say so? WereSpielChequers 22:41, 4 July 2011 (UTC)[reply]
Done! :) SalimJah 14:58, 6 July 2011 (UTC)[reply]

RCom review[edit]

Based on the previous community discussion and on feedback received by RCom members, WMF is happy to support this project by providing help with subject recruitment via a CentralNotice campaign. Steven Walling and Philippe Beaudette will be the main contact points for implementing and running the campaign. --DarTar 18:27, 29 June 2011 (UTC)[reply]

Yes, the parameters for the campaign are exactly:
  1. It will be run on English Wikipedia only
  2. Only to logged in users
  3. From July 5th, for a week maximum, depending on whether the subject recruitment quota gets met
  4. You can see the technical particulars at MediaWiki:Centralnotice-template-Harvard
Hopefully that clarifies what the CentralNotice campaign is. Steven Walling at work 23:59, 1 July 2011 (UTC)[reply]

6 month followup[edit]

Is the 6 month follow up planned to be of the same group of editors or would this be a new campaign with a new watchlist? I'm hoping it is the former, partly because a longitudinal study would be helpful, and partly because a second watchlist campaign for the same researchers would start to look a bit spammy. If you are hoping to do a follow up survey to the same 2,000 I would like to see the consent wording you are using, presumably this will be with a request for their Email? WereSpielChequers 16:27, 30 June 2011 (UTC)[reply]

Significant amount of money[edit]

Describing $50 as a significant amount of money makes the whole thing seem more commercial and less like a research project in my eyes, and up to $50 reminds me of the sort of weasel words that normally belong in my spam folder. Can you confirm that it is $50 if someone answers every question they are asked, or will some people answer every question but still not get $50 becasue they weren't in one of the groups who you gave the full set of questions to? WereSpielChequers 14:11, 4 July 2011 (UTC)[reply]

I've modified the page according to your comment. Note however that if it may make that research project look original, the fact that money is involved here does not alter neither its academic purpose nor its scientificity. Please be also reassured that the way the actual amount that each participant receives at the end of the survey is determined is explained extensively at the beginning of the survey when participants login. Thanks! SalimJah 16:33, 4 July 2011 (UTC)[reply]
Thanks for the change. I'm not objecting to you giving money to Wikimedians for answering your questions, though I'm intrigued as to how you can offer differential payments without some people giving answers that they think are high value if not entirely true. Would it be possible to give me a link to you current questionnaire? WereSpielChequers 22:59, 4 July 2011 (UTC)[reply]

Discussions about the banner[edit]

Nemo 23:47, 8 December 2011 (UTC)[reply]

And:

Make that 3: The Village Pump (technical) also uncollapsed. --Kim Bruning 03:22, 9 December 2011 (UTC)[reply]


We also have folks on irc.freenode.net , #wikimedia-rcom at this moment in time. --Kim Bruning 02:30, 9 December 2011 (UTC)[reply]

Foundation-l post(s): http://thread.gmane.org/gmane.org.wikimedia.foundation/52283

RFC: w:en:Wikipedia:Requests for comment/Central Notices and Requests for comment/Global banners

Open access license[edit]

What sort of license? Is it open access gold (free license) or not? Nemo 23:47, 8 December 2011 (UTC)[reply]

What do you mean by 'open access gold'? Chase me ladies, I'm the Cavalry 22:38, 10 December 2011 (UTC)[reply]
Nemo, we haven't enforced a specific green vs gold OA policy, but Daniel is working on that. Chase me ladies, I'm the Cavalry, for the green vs gold distinction see en:Open_access or these notes --DarTar 02:21, 16 December 2011 (UTC)[reply]
To play it fair and square, I'd expect the raw data to be accessible to all researchers under CC-BY-SA license as soon as they are available, so that the whole academic community could use it. This would be in Wikipedia spirit. Pundit 00:16, 16 December 2011 (UTC)[reply]
The full, anonymized dataset will be made available under an open license as per our policy, we haven't specified the actual license yet but we will recommend CC0 which is the license of choice for research data. --DarTar 02:21, 16 December 2011 (UTC)[reply]

Donation targets[edit]

I suspect that there's a reason why the participants are allowed to donate to both the Red Cross and the Wikimedia Foundation. ;-) Have you considered including other Wikimedia organizations? Some participants may be willing to donate to their local chapter but not (or less) to the WMF or Red Cross; it would be interesting to measure this, or anyway consider it in the results. Nemo 00:03, 9 December 2011 (UTC)[reply]

Is the information that you are allowed to donate your "winnings" to the Red Cross (and WMF, but who cares about them) made available to the participants at any point before the completion of the survey? If so I missed it. But if not, that's a pretty important part of the survey which can potentially change a person's choices to a significant degree. Basically, if you believe yourself to be more "charitable" than the average participant, then you should play the game much more "selfishly" (yes, it's a bit of a paradox), so that you can donate more at the end - though this varies by the portion of the survey.Volunteer Marek 03:59, 9 December 2011 (UTC)[reply]
It was stated at the begining. I just played selfishly and kept my winnings, but whatever. 68.180.80.227 04:30, 9 December 2011 (UTC)[reply]
If it was, it sneaked by. It's a completely different game if you're playing for the Red Cross then if you're playing for yourself. At least for some of us. And the annoying thing is I can't go back and check because their FAQ is unavailable if you already completed the survey.Volunteer Marek 08:11, 9 December 2011 (UTC)[reply]
Is it? It's just a way of trying to ensure the inducement is valuable to you.--Peter cohen 12:36, 9 December 2011 (UTC)[reply]
Of course if everyone plays like you that means the Red Cross get much less money then they could have. Although I agree with the IP that it was mentioned you could donate it (I can't remember if it specifically mentioned to who). Nil Einne 20:19, 10 December 2011 (UTC)[reply]
It's all explained. And the reason is, I think, exactly as you say, people who have made up their mind to donate (i.e. are stiveing for a true public good as opposed to a limited one). Rich Farmbrough 22:06 12 December 2011 (GMT).

Merging discussion to here[edit]

There was no specific page on enwiki for this, but this page existed on meta, so I've merged discussion to here. Feel free to second-guess me on this ;-) --Kim Bruning 02:28, 9 December 2011 (UTC)[reply]

Some Criticism[edit]

This is intended as constructive criticism:

  • There appears a disconnect between RCOM and the en community - any one of us could have explained this would be a controversial move and required discussion. Unfortunately this seems partially our fault by referring you to RCOM, however community discussion of this banner should have been taken given the reasonable opposition.
  • The "not ad" FAQ comment is severely undermined by the use of Logo's within the banner. This is, unfortunately, clear advertising - even if unintended - and could undermine WP's neutral position. I do not see why those logos are required.
  • Opening a new tab to a third party site, and automatically POSTing user information, is a very bad approach for a project that is supposed to be open and infer trust. A better approach would be to have the banner linking to a Wikipedia page which *then* contained the relevant link - with a full explanation of what would be passed over, where you were going, and the privacy measures implemented.

Just some initial thoughts. Wikipedia is primarily encyclopaedia, a place where the community may grant the privilege to survey the editorship. But then again may not. --ErrantX 10:06, 9 December 2011 (UTC)[reply]

Discussion is across three pages, which is annoying (for some reason people didn't want to centralize discussion). Summarizing here, it might be wise to take the following 3 steps asap :-)

  1. Remove the logos from the banner. (replace with text or leave out entirely). Check the AN/I thread for details.
  2. Add a link to this page from the banner (to help centralize discussion, and reduce complaints considerably)
  3. Update the FAQ with the answers to the privacy question, since many people have concerns about that. (It was answered on VP:T)

--Kim Bruning 10:31, 9 December 2011 (UTC)[reply]

I can understand at least not wanting to centralise discussion from WP to here - seeing as it is a community issue. In my experience WP discussions on these issues do take place in multiple forums :) Sometimes that can be beneficial because different places have different focus. --ErrantX 10:38, 9 December 2011 (UTC)[reply]
Surely it can come as no surprise that this banner would be seen as a free advert for some friendly institutions? I tried to handle this off-wiki last night but, frankly, have been given an obvious run around with the only advice being to write to OTRS after discussing with the other concerned trustees of my chapter and see if anything happened. Philippe has pointed me back to RCOM but as the most active discussion is at w:en:Administrators'_noticeboard/Incidents#A_note_about_the_legitimate_ad_concerns_expressed_by_some_folks_above. I suggest anyone with a view to express or information to add, do so there where it is likely to be read by far more people, rather than here. Note that WM-UK is not the only chapter with serious concerns about how this banner fails to meet our shared values. --Fæ 11:55, 9 December 2011 (UTC)[reply]
Technically, each of us (which includes you and me) as wikipedia editors has the power to take appropriate measures. However, since everyone has the same powers, we need consensus to actually start moving. --Kim Bruning 12:03, 9 December 2011 (UTC)[reply]
The logos need to be removed. If it looks like an ad then it is an ad. I can't understand why they have not been removed already if it is just an error of judgement - I can see no argument for them at all. Victuallers 12:36, 9 December 2011 (UTC)[reply]
I hope that the vocal criticism of a minority does not result in all those who have taken the option to make some money back for their involvement in Wikipedia being deprived of any future opportunities to do so. --Peter Cohen
Eh what? --Kim Bruning 19:45, 9 December 2011 (UTC)[reply]

[edit]

Banner has been temporarily disabled; I've set its weight down from 100% to 0%.

It sounds like this was meant to show to a "small percentage" of users, which it.... was not. --brion 12:46, 9 December 2011 (UTC)[reply]

Thanks Brion <3. I also created this page Meta:Central_notice_requests for any such future requests. Please feel free to direct any future discussions to global banner usage to the aforementioned page. Thanks. Theo10011 12:52, 9 December 2011 (UTC)[reply]
Not to sound sour or anything, but how many times do we get to miserably fail at writing proper central notices ? It seems about 25% over the past 4 years have been broken at initial launch. TheDJ 20:15, 9 December 2011 (UTC)[reply]
brion, the banner is actually as narrowly tailored as possible [1], i.e. it is coded to be displayed only to users eligible to participate in the study according to some editor metrics. Those are a balanced sample of around 10,000 experienced and 30,000 relatively "new" users. This feature is totally new and, in my view, a significant improvement upon the previous banner system. It was an important part of our design and we were happy to develop this code in coordination with WMF, so that banner overload (an issue the community has been complaining about lately [2]) could be reduced to a minimum for this and future campains. SalimJah 21:47, 9 December 2011 (UTC)[reply]
10,000 experienced users is a large enough proportion of the core community for the regulars to assume this was broadcast to us all. WereSpielChequers 11:12, 10 December 2011 (UTC)[reply]

Unhappy with the mandatory paypal-account[edit]

I canceled the survey when I got asked for either an existing paypal-account or a e-mail-address even before the questionnaire started. I do not have an account with them, nor do I want want or even want them to know one of my e-mail-addresses. Making it mandatory to give personal data to paypal in my eyes is not an acceptable condition for a survey what's o ever, and certainly not for Wikipedians. rgds --h-stt !? 16:55, 9 December 2011 (UTC)[reply]

You could have answered donate at wikimedia dot org. --Neo139 18:21, 9 December 2011 (UTC)[reply]
IIRC you couldn't not have a paypal account. --Kim Bruning 19:47, 9 December 2011 (UTC)[reply]
As others have said, no one said it had to be your email address. There are plenty of people and projects who publish their PayPal accounts online for donations. If you want something more personally, you could ask friends or family if they have a PayPal address. If KB is right, it worked even if you didn't even enter an address. If you really can't find anyone, feel free contact me by email :-P Nil Einne 20:17, 10 December 2011 (UTC)[reply]
Pesky double negatives: I can't access the pages anymore, but IIRC, you needed an email and paypal account. --Kim Bruning 19:15, 11 December 2011 (UTC)[reply]
I don't have a PayPal account either, and am not particularly keen on giving PayPal my email address. Nor do I want to donate directly to the WMF, or the American Red Cross. I think, in future, it would be a good idea to let donors have a different payment method... Chase me ladies, I'm the Cavalry 22:32, 10 December 2011 (UTC)[reply]
...I've also not actually received any emails about it from PayPal or the involvedresearch institutions, despite having completed the survey and having received $17... Chase me ladies, I'm the Cavalry 22:36, 10 December 2011 (UTC)[reply]

Methodology questions[edit]

1) What currency are people being offered $50 in? 50 AUD = 51.08 USD, 50 USD = 48.94 AUD, 50 NZD = 38.77 USD, 50 USD = 64.48 NZD, 50 CAD = 49.10 USD, 50 JMD = 0.57 USD. The lack of a clearly stated value for what currency appears important. During the methodology design process, was this considered? What are the assumptions regarding currency understanding for participants from outside the US? Did you get data to show that New Zealanders and Jamaicans are likely to assume the currency is American and respond according to that? Or is there a correction implemented for Jamaican an New Zealand participants that assumes they will be operating using a much lower incentive point than their US based counter parts?

2) What sort of research was done before selecting PayPal? The penetration in some countries for PayPal is much lower and it is harder for people to use this service when compared to local options, like BPAY for Australians. Beyond that, in some areas, PayPal can charge you for changing the currency over to another currency. Does the $50 come in the local currency and if it is American, will non-American users be compensated for any fees incurred as a result of currency conversion? In some cases, the only way to avoid fees is to buy goods in American currency and have them shipped locally.

3) In relation to questions one and two, during the methodology development process, what sort of corrections were made to address these points when joined together with the issue of non-American dollar currencies and the inability to access a payment system favoured by Americans? Can you better articulate exactly how this will be addressed to account for margin of error amongst participants?

4) Will this research speak to ALL English Speaking Wikipedians or will it be broken down by national groups and other demographic groups? Is this information being requested? If not, how does the methodology, other than size of the sample, correct for having a representative sample?

--LauraHale 23:19, 10 December 2011 (UTC)[reply]

en.wiki[edit]

Since the survey only affects en.wiki editors, I find that the complaints that it was insufficiently discussed on English Wikipedia have a point. Should we move the discussion page there, make an announcement, and have one of the researchers answering questions over there? Or may be such page already exists, and I am just too slow?--Ymblanter 09:06, 11 December 2011 (UTC)[reply]

There's also the problem of getting the researchers to discuss this on any wiki format. My reading of this section is that an EN wiki editor opposed it here in June and there was no response. Also having seen this survey for the first time last week I think it was mistake to target it at EN wiki - without translation it needed an English speaking audience, but it did not need to be targeted at the most surveyed part of the community. So if there were consensus to resume in some form I would suggest targeting more of the editors who rarely see research surveys and haven't started to complain about excess research. WereSpielChequers 09:59, 11 December 2011 (UTC)[reply]
I am not sure I fully understand you. The research targets En wiki editors, at least in the form it has been designed. Some editors complained, as you know, that the action seriously affects en.wp should be discussed on en.wp and not on meta. This is why I was suggesting to open a dedicated discussion on en.wp. I did not suggest to change anything else (though of course it could be considered if needed). My understanding is that now most of the complaints are not about the survey itself, methodology etc, but about lack of communication. I think this could be the most logical first step to improve communication.--Ymblanter 10:39, 11 December 2011 (UTC)[reply]
Having done this survey I see no reason other than the cost of translation why it should be targeted at EN wiki as opposed to other Wikimedia sites. One of our problems in coming up with a throttle to limit research surveys to a level that the community might find acceptable is that the researchers usually want to target the same group - the most active users on EN wiki. As this research has no obvious connection to EN wiki it seems sensible to me that if this were done it should be done to groups of editors who are less frequently surveyed than EN wiki editors. WereSpielChequers 00:03, 12 December 2011 (UTC)[reply]
Well, actually I think there is a problem. Translation is not sufficient, we also for every project which is going to be involved need a dedicated person (preferably involved in the project) who will be able to negotiate with the project, and this has to be done in the language of the project. And we do not have so many languages spoken in RCom, much less by the researchges running the survey. This is surmountable though and I think it could be done with some planning.--Ymblanter 10:14, 12 December 2011 (UTC)[reply]
Agreed. When I said "if it were done" I didn't want to imply that this was the only issue. But in my opinion it would be much easier to sell this to the community if it was one annual research survey or at least there were some mechanisms to prevent the researchers always approaching the same individuals. WereSpielChequers 14:41, 12 December 2011 (UTC)[reply]
Do you mean that YOU were the EN wiki editor who opposed it?--Ymblanter 19:50, 11 December 2011 (UTC)[reply]
Yes WereSpielChequers 00:03, 12 December 2011 (UTC)[reply]
I think now I get it. Well, I hope the researchers will discuss your suggestions here at some point, especially since the banner is disabled now.--Ymblanter 06:43, 12 December 2011 (UTC)[reply]
Actually, I moved that; we already had an open request for comment, so the best place is en:Wikipedia:Requests for comment/Central Notices Chzz 08:04, 19 December 2011 (UTC)[reply]

Sample size[edit]

Did the researchers get their required sample? Rich Farmbrough 23:19 12 December 2011 (GMT).

Community discussion - 2[edit]

Jerome started the second discussion which has taken the suggestions into account: en:Administrators'_noticeboard/Incidents#Berkman_Sciences_Po_banner_taken_down--Ymblanter 08:22, 17 December 2011 (UTC)[reply]

Moved to en:Wikipedia:Requests for comment/Central Notices Chzz 08:05, 19 December 2011 (UTC)[reply]

No PayPal payment?[edit]

I'm curious about whether the PayPal payment promised to survey participants really works? I completed the survey in the beginning of December 2011 and earned $15 USD, out of which $5 was to be donated to Wikipedia, and $10 was to be transferred to my PayPal account. It's been over a month since the survey was completed, but I received neither the PayPal transfer (which was promised to arrive within 10 days), nor any other confirmations or notifications to provided email. Can anyone provide some update on this please? Thanks. -- Nazar 20:22, 17 January 2012 (UTC)[reply]

Hi Nazar! I've been checking: looks like your PayPal transfer has been executed: out of your $15, $5 have been donated to Wikimedia (so that's fine), but your personal $10 transfer has been rejected for some reason. I send you an e-mail so that we can find a way to fix this! :) SalimJah 15:03, 20 January 2012 (UTC)[reply]
Ok. Got your email and sent you my answers (via return email) with possible options to fix it. Looking forward to your reaction. Thanks. -- Nazar 17:08, 25 January 2012 (UTC)[reply]
Retrieved from "https://meta.wikimedia.org/w/index.php?title=Research_talk:Dynamics_of_Online_Interactions_and_Behavior&oldid=3311803"