Research:Voice and exit in a voluntary work environment/Team work effect:a viability test

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search

During the first half of FY17-18, the team on this project did a series a brainstorming sessions to figure out the fastest way to experiment whether the direction we're moving towards is going to work at all. We initially started by developing the following framework:

  1. The user joins a Wikimedia project.
  2. The user is exposed (somehow) to a series of questions that can help us put the user in a specific cluster in terms of their interests and characteristics. We first target users have have just completed their first edit (to ensure that they have some interest in editing). We may relax this constraint later on depending on how users engage with the survey.
  3. Users in the same cluster who have opted in to be matched (via the survey) will be automatically paired (in a controlled experiment) and able to communicate with each other. In the first stage of the project, users will be paired via e-mail (for simplicity). They may later be paired directly on-wiki.

Doing all the steps above in a systematic way requires a lot of engineering, design, user experience, and research work to get it right. While we started working on specific aspects of the above (for example the clustering part), we decided to push in parallel the idea of testing whether this is the right direction at all, i.e., Can we see any evidence that pairing up people in teams can help them stay longer?

This document captures the result of this second iteration of brainstorming to arrive at a direction to test and conclude about the direction more quickly.

Overview[edit]

About 80% of individuals who register a Wikipedia account provide their e-mail. We intend to contact new users (who have accepted to receive email from other users) via e-mail to run a survey and derive users’ areas of expertise and interest for our “matching” treatment.

For respondents (i) who opt-in for a match, and (ii) for whom a match is found, we randomize in 3 treatments:

  1. CONTROL: send an e-mail to explain that we couldn't process their matching request, but that we still hope they will contribute.
  2. BASELINE: same as CONTROL + suggest a page to contribute to, according to their interests. (This suggestion may have to be made manually in the first step, according to some consistent procedure TBD.)
  3. TREATMENT: send an e-mail to explain that we were able to process their request, provide the Wikipedia username and e-mail address of the match, and suggest a page to contribute to (according to users’ common interests).

Newcomer survey[edit]

Below, we describe the current rough templates we have arrived at for using in this test. Please note that these are not finalized templates and feedback is welcome.

Email[edit]

The following email template is created assuming that the test will be done on Wikipedia.

Dear [USERNAME],

We want to know how to make Wikipedia better for new editors like you. Please help us by answering this short online survey! 

The survey has 11 items and will take you about 4 minutes to complete. [LINK TO PRIVACY STATEMENT FOR THE SURVEY + NO QUESTION IS MANDATORY] 

Thank you for for joining Wikipedia! 

The Research team at Wikimedia Foundation
[WIKIMEDIA FOUNDATION INFO]

Survey[edit]

Below is the list of survey questions which are mentioned in the above email. In the absence of an on-wiki survey tool, we continue to use Google Forms as the means for collecting this data, similar to the previous research in this field. Of course, the privacy statement will clearly state this to make sure the users who do not want to interact with tools outside of Wikimedia be aware before clicking on the link to survey.


I. AVAILABLE FREE TIME
How many hours per week would you say you can typically dedicate to activities of your own choosing? Think of this as your available “free time”, excluding working hours as well as any other “constrained” activity. 
[Range]

II. TECHNICAL SKILLS
Please rate your level of understanding of the following Internet-related terms. 
Please choose one number on this scale, where 1 means that you “do not understand what the term means” and 5 means that you “fully understand what the term means”.
Advanced search
PDF
Spyware
Wiki
Cache 
Phishing

III. CONFIDENCE IN ABILITY TO CONTRIBUTE CONTENT
Please rate your level of agreement with the following statement: “I would feel confident changing information on a Wikipedia entry”.
Please choose one number on this scale, where 1 means that you “strongly disagree” and 5 means that you “strongly agree”.

IV. PREVIOUS EXPERIENCE WITH ONLINE COMMUNITIES
Think of your past interactions with online communities (through social media, internet user groups, open source development platforms, or other media), would you say that you experience mostly positive or mostly negative forms of interaction with such communities? 
Please choose one number on this scale, where 1 means that you "mostly experience negative interactions'" and 5 means that you "mostly experience positive interactions". 

How often do you typically experience online harassment, cyberbullying or 'trolling' over the Internet? 
Please choose one number on this scale, where 1 means that you "almost never experience online harassment, cyberbullying or 'trolling'" and 5 means that you "experience online harassment, cyberbullying or 'trolling' on a regular basis". 

V. PREEXISTING EXPECTATIONS ABOUT WIKIPEDIA COMMUNITY
Please rate your level of agreement with the following statement: “In general, I expect the Wikipedia community to be friendly and inclusive towards newcomers”.  
Please choose one number on this scale, where 1 means that you “strongly disagree” and 5 means that you “strongly agree”.

VI. TASTE FOR WIKIPEDIA AS A COMPETITIVE ENVIRONMENT
How much would you say you enjoy arguing with others? 
Please choose one number on this scale, where 1 means that you generally “avoid contentious debates with others” and 5 means that you generally “enjoy entering into contentious debates with others”.

VII. RISK AVERSION
How do you see yourself: are you generally a person who is fully prepared to take risks or do you try to avoid taking risks?
Please choose one number on this scale, where 1 means that you are generally "unwilling to take risks" and 5 means that you are generally "fully willing to take risks".

VIII. DEMOGRAPHIC INFORMATION
Your age:
[Range]

Your gender: 
Male
Female
Other

What is the highest level of education that you have completed?
Less than High School
High School/GED
Some College
2 years College Degree (Associates)
4 years College Degree (BA, BS)
Master’s Degree
Professional Degree (MD, JD)
Doctoral Degree (PhD)
Don't know

IX. INTERESTS AND EXPERTISE
Please tell us about the topical areas that are of most personal interest to you and about which you have some knowledge. 

It is likely that there are other recently registered Wikipedia contributors out there who share (some of) your personal interests. Would you be interested in us putting you in touch with one of them via e-mail, so that you could help each other out while contributing to Wikipedia? 
PLEASE NOTE: we typically receive more requests of this type than we can actually process. If you are interested in this proposal, we cannot guarantee that we will be able to process your request.  

Note that the question in "Technical Skills" section is based on Based on http://www.webuse.org/pdf/HargittaiHsieh-WebSkillMeasures2012.pdf and the one in "Confidence in Ability to Contribute Content" from http://www.tandfonline.com/doi/full/10.1080/1369118X.2014.957711 .

Follow-up emails[edit]

Control[edit]

Dear [USERNAME],

We are coming back to you regarding your stated interest in “teaming-up” with another recently registered Wikipedia editor to contribute to articles of common interest.

This proposal has been very popular among the contributors who took part in our survey, and because we currently process those requests manually, we could only manage a subset of them.

In order to ensure fairness in our selection procedure, we randomly selected the subset of editors for whom we would try to find an appropriate match. Your request did not end-up in this randomly selected group, so that we were unfortunately not able to meet it. (Given its popularity, we may try to develop automated ways to manage such requests moving forward…)

Needless to say, Wikipedia needs you! Anything you do to improve its content will benefit thousands of people around the world.

Please help us bring free knowledge to everyone!

Sincerely,

The Research team at Wikimedia Foundation

[WIKIMEDIA FOUNDATION INFO]


Baseline[edit]

Dear [USERNAME],

We are coming back to you regarding your stated interest in “teaming-up” with another recently registered Wikipedia editor to contribute to articles of common interest.

This proposal has been very popular among the contributors who took part in our survey, and because we currently process those requests manually, we could only manage a subset of them.

In order to ensure fairness in our selection procedure, we randomly selected the subset of editors for whom we would try to find an appropriate match. Your request did not end-up in this randomly selected group, so that we were unfortunately not able to meet it. (Given its popularity, we may try to develop automated ways to manage such requests moving forward…)

Needless to say, Wikipedia needs you! Anything you do to improve its content will benefit thousands of people around the world.

According to the interests you stated, we would like to suggest that you expand the lead section of this underdeveloped Wikipedia article.

Please help us bring free knowledge to everyone!

Sincerely,

The Research team at Wikimedia Foundation

[WIKIMEDIA FOUNDATION INFO]


Note: Expanding the lead section may not be the best choice of task. Let's discuss this more with the community this experiment will run in.

Treatment[edit]

Dear [USERNAME] and [USERNAME],

We are coming back to you regarding your stated interest in “teaming-up” with another recently registered Wikipedia editor to contribute to articles of common interest.

This proposal has been very popular among the contributors who took part in our survey, and because we currently process those requests manually, we could only manage a subset of them.

In order to ensure fairness in our selection procedure, we randomly selected the subset of editors for whom we would try to find an appropriate match. Your request ended-up in this randomly selected group, so that we were able to try to meet it. (Given its popularity, we may try to develop automated ways to manage such requests moving forward…)

We have therefore matched the two of you according to what you have indicated to us about your interests, and we hope you will get in touch with each other to discuss ways of working together as a team.

Needless to say, Wikipedia needs you! Anything you do to improve its content will benefit thousands of people around the world.

According to the interests you stated, we would like to suggest that you collaborate to expand the lead section of this underdeveloped Wikipedia article.

Please help us bring free knowledge to everyone!

Sincerely,

The Research team at Wikimedia Foundation

[WIKIMEDIA FOUNDATION INFO]