Research:Teahouse/Metrics

Add links
From Meta, a Wikimedia project coordination wiki


Read more: More detailed and updated metrics are available in the Phase 2 Metrics Report.

Teahouse:metrics

» findings on editor participation, engagement and retention from the pilot period

WP:Teahouse was designed to introduce new editors to the Wikipedia community in a friendly and engaging way. Between February 27th and May 27th, almost 600 editors participated in Teahouse, 68% of them new editors. Guests asked more than 500 questions on the Teahouse Q&A board, and created over 200 guest profiles. More than 40 editors contributed as Teahouse hosts.

In this report we assess the impact of the Teahouse during this pilot period. We report data on the participation, engagement and retention of new editors gathered from surveys and edit logs. We also report feedback from Teahouse hosts and other Wikipedian participants in order to understand the impact of Teahouse on the wider editing community.

Executive summary[edit]

  • New & experienced editors alike rated all aspects of their Teahouse experience favorably by a wide margin
  • Most questions asked on the Q&A board came from new editors
  • Many new editors returned to ask or answer additional questions
  • 28% of Teahouse guests were women, higher than the Wikipedia average of 9%
  • Teahouse guests contributed more article content than other new editors, and that content survived longer
  • More Teahouse guests were still editing at the end of the pilot period
  • Only 4.5% of invitees visited Teahouse, but some recruitment strategies yielded higher success rates
  • Survey responses reflect perennial challenges for new editors, as well as new directions

/Archive

WP:Teahouse[edit]

Teahouse was launched on February 27th 2012, and the pilot period ran for approximately three months, through May 27th. In that time 442 registered editors participated (made at least one edit) on the Teahouse Q&A board and 514 questions were asked. Approximately 215 new profiles were created on the Guestbook page. 586 editors (68% of them new editors ) participated in all.

Methods[edit]

Surveys[edit]

To assess editors’ perceptions of Teahouse and their experiences we surveyed Teahouse guests twice during the pilot period, once during week 5 and once during week 11. We also surveyed Teahouse hosts during week 11, asking them a related set of questions. Our response rates for these surveys were relatively high: 25% (68/267) of new editors and 56% (71/127) of experienced Wikipedians who were offered the survey responded.

Below we present the results from the surveys conducted in week 11, near the end of the pilot period. We are still working on integrating the results from the week 5 survey of Teahouse guests into these totals. In the meantime, preliminary results from that survey can be viewed here.

Edit logs[edit]

In order to assess activity on Teahouse we tracked edits to Teahouse pages by new editors and veterans on a live ‘mirror’ of en.wikipedia.org. We tracked transclusions of Teahouse invitation templates and (aggregated) editing activities of Teahouse guests with MySQL queries and Python scripts.

High level findings[edit]

Participation and engagement : new editors[edit]

68% of Teahouse participants (those who edited either the Q&A board or the Guests page) were new editors.

» New editors give the Teahouse high marks[edit]

71% of new editors surveyed said that they were "Satisfied" or "Very satisfied" with their Teahouse experience, versus only 5% who said they were "Dissatisfied" or "Very dissatisfied" .

Guest overall satisfaction with WP:Teahouse

When asked to describe what in particular they liked about their experience, new editors cited a range of factors, from the promptness and quality of the answers they received to the friendly atmosphere and the ease of use.

Only one respondent answered the related question on dislikes, saying that there was a lot of talk and not much action at Teahouse.

Survey responents also provided valuable feedback, most of it positive, in their answers to a follow-up question Is there anything else you'd like to share with us? If so, you can let us know here.

» Female newcomers participate in Teahouse at a higher rate[edit]

In all, 28% of survey respondents self-identified as female. This is a noticeably higher rate than figures cited in previous studies, which range between 8% and 13%, suggesting that Teahouse appealed to female editors.

Female newcomers did not rate their overall experience with Teahouse substantially higher or lower than respondents who identified as male. However, female editors' satisfaction ratings for the answers the received on the Q&A board (while still positive) were noticably lower on average than the satisfaction ratings given by their male counterparts. While it should be noted that these averages are based on a small sample size and are therefore sensitive to outliers (one or two editors who recorded particularly good or particularly bad experiences), we intend to investigate why gender seemed to be a factor in editors' satisfaction with Teahouse Q&A.

Participation and engagement : Wikipedians[edit]

Experienced Wikipedians (defined as editors whose accounts were created before 1/1/2012 or who had made more than 100 edits at the time they first participated) who had edited Teahouse/Questions or Teahouse/Guests during the pilot period and who were not blocked were also invited to participate in a survey. 73% of these survey respondents indicated that they were not current or former Teahouse hosts.

» Experienced Wikipedians give the Teahouse high marks[edit]

70% of Wikipedians surveyed said that they were "Satisfied" or "Very satisfied" with their Teahouse experience, versus 5% for "Very dissatisfied" or "Dissatisfied", which mirrors the responses of new editors.

Wikipedian overall satisfaction with WP:Teahouse
Statistic Value
Min Value 2
Max Value 5
Mean 4.02
Variance 0.82
Standard Deviation 0.91
Total Responses 58

» Many experienced Wikipedians believe the Teahouse has a positive impact on the experience of new editors[edit]

When asked to describe what in particular they they liked about their experience on the Teahouse, experienced editors cited many of the same features as new editors: the promptness and quality of the answers they read or received, the friendly atmosphere, etc. Many responents also indicated that they believed that the Teahouse was having a positive impact on newcomers, primarily by functioning as a friendly "safe zone" for new editors, and that the presence of the Teahouse was beneficial to the community as a whole. As one editor eloquently put it: "There is nothing extraordinary about it. It just needs to exist and is fundamental to the Wikipedia experience."

No respondents stated any specific dislikes in response to the question Was there anything in particular you liked about your Teahouse experience?.

Experienced editors also provided valuable feedback, most of it positive but with some concerns and criticisms, in their answers to four follow-up questions:

  1. What was the most rewarding thing about participating in Teahouse?
  2. What was the most frustrating thing about participating in Teahouse?
  3. What was the most surprising thing about participating in Teahouse?
  4. Is there anything else you'd like to share with us?

Responses to these questions are included in the four tables below.

» Most experienced Wikipedians intend to participate again[edit]

In response to the question Do you think you will participate in the Teahouse in the future? 79% of Wikipedians surveyed responded "Yes, frequently" or "Maybe once in a while".


Likelihood of future Teahouse participation by Wikipedians
Statistic Value
Min Value 1
Max Value 4
Mean 2
Variance 1.09
Standard Deviation 1.04
Total Responses 56

» Wikipedians see plenty of room for improvement[edit]

Experienced Wikipedians also provided helpful suggestions on new features to further support Teahouse guests, listed in the table below.

Features suggested by Wikipedians to support Teahouse guests

Experienced Wikipedians also provided helpful suggestions on new features to improve the experience of participating as a Teahouse host, listed in the table below.

Q&A board[edit]

The Q&A board experienced a relatively high volume of activity, which was sustained throughout the pilot period. There were an average of 39 revisions and 5.6 questions per day (for comparison, the Wikipedia Help Desk averaged 90.3 revisions and 16.9 questions per day over the same period).

Revisions per day to WP:Teahouse/Questions, 2/27/2012 - 5/27/2012
Questions asked per week on WP:Teahouse/Questions, 2/27/2012 - 5/27/2012

Findings[edit]

Participation and engagement : new editors[edit]

68% of participants on the Q&A board were new editors. 25% of new editors asked more than one question, with an average of 1.6 questions per new editor. Q&A board participation has been relatively steady during the pilot period, after an initial lull.

» New editors were very satisfied with the answers they received[edit]

Guest satisfaction with answers to questions asked on Teahouse/Questions
Statistic Value
Min Value 1
Max Value 5
Mean 3.86
Variance 0.8
Standard Deviation 0.9
Total Responses 63

The Q&A board seems to be a very stimulating and interactive place for new editors. 87% of respondents stated that they were 'satisfied' or 'very satisfied' with the quality of the answers on the Q&A board. 97% of editors stated that they received a follow up message from a Teahouse host after they asked their question (generally a Teahouse talkback template,and/ or a personal message).

When asked to describe what in particular they they liked about the answer they received, 22 new editors responded. Positive features they highlighted included the friendliness and promptness of the reply, as well as the personal detail of their answers and the effort that the answerer (or multiple answerers) made to clarify complex issues. One respondents also cited a negative impression that they experienced more discussion than action on their specific problem Their responses are included in table below.

» Many new editors were hesitant to answer questions themselves[edit]

While 61% of new editors surveyed indicated that they had asked a question on the Q&A board, only 11% said they had answered one. This is a substantially lower percentage than we had hoped for.

When asked to describe what in particular they they liked about their experience, new editors cited a range of factors, from the promptness and quality of the answers they received to the friendly atmosphere and the ease of use. One respondents also cited dislikes, in particular that there was a lot of talk and not much action.

Reasons given by Teahouse guests for not answering a question
Statistic Value
Min Value 1
Max Value 5
Mean 3.4
Variance 1.49
Standard Deviation 1.22
Total Responses 57

Participation and engagement : Wikipedians[edit]

» Even experienced Wikipedians found the Q&A board a good place to get help[edit]

Experienced Wikipedians who came to the Q&A board for help were also highly satisfied with their answers, in roughly the same proportion as new editors.

Wikipedian satisfaction with Teahouse answers
Statistic Value
Min Value 1
Max Value 5
Mean 4.11
Variance 1.21
Standard Deviation 1.1
Total Responses 19
Reasons Wikipedians gave for asking a question on Teahouse

» Half of Wikipedians who participated don't intend to participate regularly[edit]

Many Wikipedians who participated in the Q&A board did not sign up as Teahouse hosts. Survey responses indicate that more than half of the Wikipedians who didn't sign up indicated that they did not intend to participate regularly, either because they were just 'checking it out' or because they didn't have time to participate further. A small percentage of respondents (~9%) indicated that they did not know that they were allowed to sign up as hosts, or did not feel experienced enough to do so.

Reasons Wikipedians gave for not signing up as Teahouse hosts

Guestbook[edit]

Summary[edit]

The Teahouse Guests page drew in fewer editors overall than the Q&A board, and unlike Q&A board participation we saw a gradual decline in the creation of new Guest profiles over the course of the pilot.

Findings[edit]

Participation and engagement : new editors[edit]

Roughly half of new editors we surveyed at the end of the pilot said that they had created a guest profile.

» New editors enjoyed browsing other guests' profiles and creating their own[edit]

Guests' levels of satisfaction with the profile creation process was comparable to the level of satisfaction reported for the Teahouse as a whole, though substantially lower than the level of satisfaction that guests reported with the answers they received on the Q&A board. This indicates that guests may have found the Q&A board more valuable overall than the guest profiles.

Guests highlighted the ease of creating a profile and the ability to browse other profiles as particularly beneficial. No guests identified any aspects of Guest profiles that they especially disliked.

Guest satisfaction with the process of creating a profile on Teahouse/Guests
Statistic Value
Min Value 3
Max Value 5
Mean 3.80
Variance 0.44
Standard Deviation 0.66
Total Responses 30

» Profile creation declined late in the pilot period[edit]

The number of profiles created per week was initially quite high, but the rate of creation began to decline noticeably around the middle of the pilot period. We believe this was due to a glut of profiles on the page. We have since changed the Teahouse/Guests page so that it only shows the most recently-created profiles, with a complete list of Teahouse guests and their introductions available on a separate "Guestbook" page.

Guest profiles created per week on WP:Teahouse/Guests, 2/27/2012 - 5/17/2012

» Many new editors did not know they could create a profile[edit]

Of those guests who did not create a profile, the most common reason they cited was that they "didn't know it was there". This suggests that a more visibly prominent call-to-action could increase the proportion of guests who introduce themselves.


Q: Why didn't you introduce yourself?

Reasons given by Teahouse guests for not creating a profile
Statistic Value
Min Value 1
Max Value 6
Mean 3.00
Variance 1.39
Standard Deviation 1.18
Total Responses 34

Participation and engagement : Wikipedians[edit]

Invitations[edit]

Summary[edit]

A general workflow for inviting new users from a database report

Thousands of new Wikipedia accounts are created every day, but most of them are never used for more than a couple of edits before they are abandoned. This presents both challenges and opportunites for outreach to new editors: on the one hand, focusing on only editors who highly active early on yields a small enough sample that a significant proportion of promising new editors can be personally invited by a handfull of dedicated hosts. On the other hand, since our goal was to reach out to those editors who were likely to give up quickly—whether because of Wikipedia’s lack of usability and sociability or in response to early, negative experiences—we did not want to set the bar for invitation too high and risk ignoring the very editors we set out to recruit.

Following previous research on early editing patterns associated with an increased probability of retention, we created an automated daily report of new editors who had made at least 10 edits within their first 24 hour period or who had made at least 20 edits over at least three editing sessions within their first four days. We excluded editors whose accounts were currently blocked from editing for vandalism or disruptive behavior. Teahouse hosts viewed the contribution histories of these editors and selected which ones to invite. Hosts were also encouraged to invite new editors they encountered in the course of their daily editing activities, through the feedback dashboard, the new editor contribs filter, and at offline meetup events.

Findings[edit]

» Overall response rate from direct invitations is fairly low[edit]

Hosts sent out a total of 7339 invites over the course of the pilot period, and 321 (4.5%) invitees subsequently participated in the Teahouse (asked or answered a question on the Q&A board or created a Guestbook profile). This low response rate reflects the challenge of trying to reach out to newcomers very early, before the natural ‘winnowing’ process takes effect.

» Different invitation strategies yield different response rates[edit]

Most channels from through which we invited newcomers to Teahouse yielded roughly the same response rate. However, we do see some indications that different recruitment strategies, such as inviting newcomers through particular channels and emailing, can yield slightly higher rates of response than inviting all good faith newcomers.

Inviting authors of failed AfC candidates. We see this most notably with invitations sent to new editors whose articles had been rejected by AfC. Early in the pilot, in collaboration with several Teahouse hosts, we designed a second invite template for editors who had been rejected by AfC, an especially frustrating event which can makes editors feel like they have performed a great deal of wasted work. This templatetook a sympathetic stance on the rejected editor’s plight and explicitly welcomed them to visit the Teahouse Q&A board for feedback and advice on next steps. In all, 758 new editors were invited using this template, of which 55 (7.3%) subsequently visited Teahouse.

Inviting new editors over email. Another strategy that seemed to yield a slightly higher response rate was emailing guests. Hosts were asked to send email invites to guests who had 'Email this user' enabled, in addition to posting a templated invite on the user's talk page. In a sample of 2244 Teahouse invitees who had this feature enabled, 833 received an emailed invitation to Teahouse, in addition to a welcome template, while 1411 invitees received a template only. We find that 61 (7.3%) emailed guests subsequently visited the Teahouse, compared with 72 (5.1%) templated guests. These preliminary results, while only suggestive, may indicate that reaching out to very new users through external channels such as email can enhance the 'stickiness' of community projects or Wikipedia editing in general.

» Drop-in guests and invited guests participate in different ways[edit]

Interestingly, we saw a stark differences in participation patterns of invited guests and ‘drop in’ guests (newcomers who found Teahouse without a specific invitation). For instance, out of a total of 191 guests profiles on our Guestbook page, 92% were created by invitees and only 8% by drop-in guests. By contrast, 40% of newcomers who participated on the Q&A board (by either asking a question or answering one) were drop-ins. While we can't say anything for sure, we suspect that this disparity exists because guests who had not been specifically invited may have felt relatively more comfortable performing lower-stakes activities (asking questions) than actions that signalled group identification (creating a profile).

New editor retention[edit]

In order to determine whether participating in Teahouse has a positive impact on an editors likelihood of continuing to edit Wikipedia or on their editing behaviors, we analyzed the outcomes for three groups of editors in three different 'conditions'--two different control groups and an experimental group--drawn from editors who joined between February 25th and May 19th.

  • Control A: 192 new editors who met the requirements to be included on the Teahouse invitee report but whose usernames were not included on that report, who were not invited through any other forum, and who did not find their way to the Teahouse on their own.
  • Control B: 200 new editors who were invited through the feedback dashboard, the Teahouse invitee reports and the new editor contribs filter, but did not ask a question on the Teahouse Q&A page or create an introduction box on the Teahouse/Guests page.
  • Experimental: 190 new editors who were invited through the feedback dashboard, the Teahouse invitee reports and the new editor contribs filter and who did edit the Teahouse/Questions and/or Teahouse/Guests pages.


Summary[edit]

We measured the level and type of activities these editors performed after the date they were invited to Teahouse (or, in the case of Control A, the date they would have been invited). The tables below present our complete data on these three samples (a 'table key' is provided below each table). We highlight particularly notable findings with charts in the bulleted list of findings below.


Table 1: editing patterns

group ns0 bytes added # editors with >0 NS0 edits total global edits avg global edits (by editor) total articles created/edited avg articles created/edited (by editor) significant bytes survived per edit survival metric per edits
Control A 1571464 141 7506 53.23404255 1319 9.4 95.18228921 181.1
Control B 5633766 158 20558 130.1139241 3177 20.1 109.5534449 230.3
Teahouse Guests 16901194 188 53072 282.2978723 17473 92.9 170.2108615 316.3

Table 1 key terms

  • ns0: the article namespace
  • # editors with >0 ns0 edits: number of editors in the group who made at least 1 edit to the article namespace after the date of invitation
  • significant bytes: number of bytes (excluding whitespace) added by
  • bytes survived: number of bytes added that had not been deleted or reverted at the time of analysis
  • survival metric: an average 'score' of significant bytes added per editor to the article namespace which survive across multiple subsequent revisions. Content that survives multiple revisions is considered to be higher quality or more trusted than content that only survives one subsequent revision. The more subsequent revisions an edit's content survives, the higher that edit's survival metric.


Table 2: recent activity

group total editors # recently active % recently active avg recent edits avg recent article edits # blocked % blocked
Control A 192 17 9% 19.4 11.5 13 7%
Control B 200 21 11% 67.4 64.5 10 5%
Teahouse Guests 190 63 33% 121.0 92.1 12 6%

Table 2 key terms

  • ns0: the article namespace
  • recently active: number of editors in the group who made at least 1 edit to any namespace between May 23rd and June 6th
  • recent edits: average number of edits, per editor, between May 23rd and June 6th
  • blocked: editors blocked (at any time and for any duration) during the pilot period

Findings[edit]

» Teahouse guests make more article edits, and add more article content[edit]

Amount of content (in bytes) added by Teahouse guests vs. control groups

Teahouse guests added much more content to the article namespace than members of either of the control groups. This trend was seen both in the total content bytes added by all group members, and in the per-editor averages.

avg (by editor) ns0
Control A 11145.1
Control B 35656.7
Teahouse Guests 89900.0

Further, our results show that

  • more Teahouse guests made at least one edit to the article namespace
  • Teahouse guests edited more articles, in total and on average
  • Teahouse guests made more global edits (to all namespaces), in total and on average

These findings indicate that guests engaged in more editing activity than editors who did not visit the Teahouse. Interestingly, we find that Teahouse guests also made comparatively more edits to other namespaces (such as the article and user talk namespaces).

NS0 edit ratio
Control A 0.68
Control B 0.83
Teahouse Guests 0.65

This indicates that although Teahouse guests performed more editing work on average, they also performed comparatively more work in non-article spaces (for instance, article_talk:, user_talk: and Wikipedia) than members of the control groups. This may mean that Teahouse guests participated more in talk page discussions than other editors, suggesting more community interaction.

» Contributions by Teahouse guests are more likely to survive[edit]

Average survival (in significant bytes) of content added by Teahouse guests vs. control groups

Not only do Teahouse guests create more article content, but our findings indicate that the content they add may be of higher quality. The graph above shows the amount of content added by members of the three groups in significant bytes (bytes of text excluding whitespace) that had not yet been deleted or reverted through the end of the pilot period.

However, this measure on its own could be misleading in some cases. For instance if Teahouse guests tended to edit many more articles that were less actively edited or watched, they would be less likely to be reverted whether or not their content was higher quality. In order to account for this possibility we compute an averaged survival metric for the three groups. This metric, developed by data analysts in the Wikimedia Foundation's Global Development department, accounts for the number of subsequent revisions that the content of each edit survives. This metric gives higher weight to content that survives through many subsequent revisions by other editors, assuming that a piece of article content has been viewed many times by other editors and has not been removed can be "trusted" to be good quality with more confidence than a piece of content that has been seen by fewer times.

We find that edits by Teahouse guests also show a higher average survival metric than those made by editors in Control groups A and B.

Survival Metric per edits
Control A 181.1
Control B 230.3
Teahouse Guests 316.3

In order to ascertain whether editors in our control groups had made fewer contributions because they had been blocked from editing, we also analyzed how many editors in each group had been blocked (for any duration) during the preceding three months. We found no significant difference between the number of editors in each group who had been blocked.

num_blocked % blocked
Control A 13 7%
Control B 10 5%
Teahouse Guests 12 6%

» More Teahouse guests are still editing[edit]

The metrics presented so far do not take into account whether or not Teahouse guests are still more active than control editors. In order to find out whether or not Teahouse guests were more likely to continue editing longer than other editors, we measured the editing activity of editors in our three groups in the days immediately preceding and following the pilot period, May 23rd through June 6th.

We found that 33% of the guests in our sample had made at least 1 edit during this period (versus 9% and 11% for groups A and B, respectively), and that they had made more recent edits on average.

Although this analysis only measures short-term retention, it is an encouraging sign given that most editors who stop editing very early in their Wikipedia careers do not edit again. We intend to measure retention of members of these groups at 3, 6, and 9 months to analyze the impact of Teahouse on long term editor retention.

Percentage of sample of Teahouse guests vs. control groups who made at least one edit during the final weeks of the pilot period
Number of edits made by sample of Teahouse guests vs. control groups in the final weeks of the pilot period

New editor experience: Insights[edit]

Summary[edit]

Features that new editors requested for WP:Teahouse

Findings[edit]

» New editors want to find collaborators and things to do[edit]

» New editors want high quality help documentation[edit]

» New editors still struggle with the Wikipedia interface[edit]

Unfortunately, new editors in 2012 struggle with many of the same problems as previous generations of newcomers.

Features that new editors requested for WP:Teahouse

» New editors are excited about contributing to Wikipedia[edit]

New editors in 2012 are excited about becoming Wikipedians. Two thirds of our new editor survey respondents gave an answer to the question What are some things you are excited about doing on Wikipedia? That this is the highest response rate we received to any open-ended question (by far) only underscores the enthusiasm and energy of these new Wikipedians.

Retrieved from "https://meta.wikimedia.org/w/index.php?title=Research:Teahouse/Metrics&oldid=24879334"