Research talk:Wikipedia Dispute Resolution

From Meta, a Wikimedia project coordination wiki

Initial review[edit]

Steven, thanks for posting your proposal, please find below some comments and requests.

Purpose of the project
Even if you don't have specific hypotheses you would like to test at this stage, it would be helpful if you articulated the questions that you are trying to answer with this survey. Asking participants about their "perceptions of dispute resolution" or "gauging interest in participating in on and off wiki training" are legitimate goals, but can you spend a few more words about the actual problems with the current dispute resolution process that you would like to better understand? For example, can you give us pointers to existing discussions on this process suggesting issues that need to be investigated or fixed? What would be an ideal actionable outcome of this study (e.g. a proposal for policy change? A set of feature requests?).
Draft of the survey
When reviewing SR requests we usually ask proponents to share a draft of the survey if it's available, to give us a better idea of the kind of questions you would like to ask and give you our input.
Subject recruitment
Posting a recruitment message via a bot for 2K participants among active editors sounds problematic: that's a pretty large sample and before giving you the green light I would like to hear more about how many completed responses would allow you to complete the analysis. If a few hundreds completed responses are sufficient to answer your questions (and depending on the response rate that you expect, which is presumably a function of the length of the survey and the interest in this topic among your participants), there is no probably need to recruit 2K participants. In the past we've supported requests to recruit participants in multiple batches until the target number of responses is reached, I would rather follow this approach after hearing how many responses you need than allowing a blanket recruitment targeting thousands of editors.
Data retention
Thanks for the very detailed section on data retention, I don't have any comment to add on this.
Privacy
I'd like to make sure that you use an appropriate privacy statement on the landing page and that you treat the data you collect accordingly. The Legal and community advocacy team can provide you some guidance on how to implement this. You can contact Philippe Beaudette to obtain support with this.
Type of WMF support
I changed this request to SR as it seems to be the only type of support that applies, let me know if you have any questions.
Hi Dario, thanks for al your comments. I think I have addressed all your comments with the changes I have now made to the proposal. Jonathan Morgan, another researcher, helped me with the privacy statement which is on the survey top) so let me know what you think. Steven Zhang (talk) 20:05, 21 February 2012 (UTC)[reply]

Follow up[edit]

Landing page
if possible there should be a short landing page prior to the start of the survey, where you also state your identity (for the purpose of data collection)
I may be able to create a landing page of sorts, which I could do on-wiki somewhere. It can't really be done within Google Forms as far as I know. In regards to identifying myself, do you mean something like "Hi, I'm Steven Zhang, which is also my username across Wikimedia. I am an active participant in dispute resolution on the English Wikipedia, and am conducting this survey to get a better understanding on the community's perspective of dispute resolution."
you can just add a new page using Google form, can't you? The only kind of information that you would need to have on the landing page is the one that is currently at the top of the screen. Saying something like "This survey is run by Steven Zhang, an active participant in dispute reolution on the English Wikipedia. For any question on this survey please contact me at: Steven Zhang <cro0016@gmail.com>". Stating clearly who runs the survey is important if participants have questions about privacy or data retention.
OK. I can do that.
Effort
you should add information on this page on the number of questions and expected duration
Can do
Sweet
Mandatory vs optional questions
making all questions required is the best way to ensure a very low response rate: are you sure that they all fields should be mandatory?
I would welcome suggestions on making questions required, but I made some adjustments to ensure not all are mandatory. I felt that making all of them optional would make responses kinda useless (having prefer not to say in everything etc)
it's your call but best practices in survey design suggest that you should be able to skip every question or at least add an option for participants to say don't know/won't want to answer. Seriously consider this issue before you go live as I expect a high drop off rate if participants get frustrated by too many mandatory fields.
I'll see if I can tweak it a bit.
Recruitment
I'd like to iterate my suggestion that you recruit people in batches (say 200 per batch) and then you only post a new batch of requests until you hit the target response rate. Can you make an explicit recruitment plan of this kind in the proposal?
If it was December or January I'd tend to agree. Based on numbers I've been told, approx 10% of a number offered a survey actually respond to it. Additionally, I'd want to wait a month before I start collating results. With the survey offered at the start of April, ending at the start of May, I'd see it take another month to collate the results and put it into a readable format (like the results of the New Page Patrol survey. Then I'd use some of that info to prepare the proposed Wikimania presentation. If only 20 responded, it'd be a relatively useless sample, and I wouldn't have time to offer it again (or would be cutting it very fine).
I think it's reasonable to expect that most people you contact will respond within one week (especially if these are active community members). You should be able to send the first batch after the first week as needed. 2K recruitment messages is really beyond what we've ever supported and I cannot exclude that this will trigger negative reactions from the community or jeopardize the possibility of other surveys.
I got a lot of feedback and support from Oliver Keyes who ran the NPP survey, and looking at the NPP survey, almost 4,000 were offered the survey. That's kinda why I think 2,000 is a good size, I'd hope for 500 responses which would make the results useful. I'd be happy to start with the top 1,000 if that works for you.
Rationale
I still don't see any examples of potential problems or documentation on why dispute resolution does not function as expected, which this survey should help shed light on. This is something I asked you and that would make your proposal much more solid.
The main problems that the core group that work in dispute resolution have observed (there are about a dozen of us that have been in discussions) is a low amount of active users in dispute resolution (compare it to very high doctor-patient ratios, we can't look after everyone well) and inexperienced volunteers (a first year med student doing open heart surgery). This has led to people leaving the project. I have a small handful of examples in my talk page archives but I have seen examples of people leaving, and the fallout from a dispute due to an inexperienced mediator. I just haven't collected results of it. No study on DR has been done in the past, and that's kinda the whole point of this. We as a collective have our views, concerns and ideas, but we want to get the views of the community on the matter.
Would it be ok to post this on the project page or do you have reasons to believe this would bias the results? This is the kind of documentation that we typically ask to people who submit support requests.
Yeah, I can post the above to the project page.

Closing the review[edit]

Many thanks for making the changes that Aaron and I requested, I am happy to mark this project as reviewed. As for the recruitment, please consider contacting only the top 1K participants and report on the response rate before proceeding any further. --DarTar (talk) 20:19, 3 April 2012 (UTC)[reply]

Sure, can do. Steven Zhang (talk) 20:29, 3 April 2012 (UTC)[reply]

Problem participating[edit]

The link on your invitation on my talk page didn't work. I also got logged out when I came to this page. Ghostofnemo (talk) 00:36, 10 April 2012 (UTC) Your message was on my Wikipedia talk page, not on this wiki. Ghostofnemo (talk) 00:37, 10 April 2012 (UTC)[reply]

The survey link is here. Steven Zhang (talk) 00:46, 10 April 2012 (UTC)[reply]
Still not working for me. I use Ubuntu as my OS, if that makes a difference. I also have anti-java script software, but it's not notifying me of a problem. 180.30.220.50 13:25, 12 April 2012 (UTC)[reply]

Scam?[edit]

Your link directed me to a page outside Wikipedia. This is completely unacceptable as it could be used for a scam. I advise others users not to take part in your survey. Xxanthippe (talk) 23:03, 11 April 2012 (UTC).[reply]

All surveys are done off Wikipedia. If the survey was hosted on Wikipedia (which isn't even technically possible) the information that links answers to a user, such as demographic info, would be available for all to see. The survey is hardly a scam. Steven Zhang (talk) 23:47, 11 April 2012 (UTC)[reply]
Ditto – all surveys we run (including community surveys, studies by external researchers as well as WMF surveys) are hosted on external servers. If any editor has doubts about the credentials of the person running this survey s/he can look up this page where the project and the terms under which the data is collected are documented. --DarTar (talk) 01:26, 12 April 2012 (UTC)[reply]

survey[edit]

I wanna start a survey on vandals and interview them. Any ideas where I can start.--Deathlaser (talk) 20:18, 5 May 2012 (UTC)[reply]

At the nearest school? --78.35.244.186 20:53, 18 June 2012 (UTC)[reply]

Some problems and solutions concerning editor retention and dispute resolution[edit]

Please see en:User:Timeshifter/More articles and less editors. It summarizes succinctly some of my analysis of problems and some solutions concerning editor retention, dispute resolution, etc.. --Timeshifter (talk) 14:33, 7 May 2012 (UTC)[reply]