Research:Teahouse/Phase 2 report/Metrics

From Meta, a Wikimedia project coordination wiki


Teahouse:metrics

» findings on editor participation, engagement and retention from February 27th through October 11th, 2012

WP:Teahouse was designed to introduce new editors to the Wikipedia community in a friendly and engaging way. Between February 27th and October 11th, 1,512 editors participated in Teahouse. 73% of these participants (1,098) were new editors. Guests asked 1,381 questions on the Teahouse Q&A board, and created 420 profiles. 77 editors signed up to be Teahouse hosts.

We report on Teahouse activity and impact during this roughly seven-month period. We focus our analysis on experiments we ran and new features we added during Phase 2 of the project. Data specific to the Teahouse pilot period (February through May 2012) can be found in the pilot metrics report.

Contents


Executive summary[edit]

  • Participation by new editors has increased 26%. 21% more questions are being asked and 34% more profiles created.
  • 29% of Teahouse guests are women, 3 times higher than the Wikipedia average of 9%.
  • Teahouse guests edit 3 times more articles. They make 4 times more edits, and edit 2 times longer than other new editors.
  • Automating the invite process has increased traffic and decreased host workload.
  • 80% of guests rate their Teahouse experience as above average.

Overview[edit]

Methods[edit]

Surveys

To assess editors’ perceptions of Teahouse and their experiences during the post-pilot phase of the project, we surveyed 400 Teahouse guests who visited Teahouse between July 1st and October 1st. Our response rate was fairly high: 22% (89/400) of new editors who were offered the survey responded.

Edit logs

In order to assess activity on Teahouse we tracked edits to Teahouse pages by new editors and veterans on a live mirror database of en.wikipedia.org. We tracked invitations sent to guests by HostBot and aggregated editing activities of Teahouse guests with MySQL queries and Python scripts.

Page views

We aggregated pageview data from stats.grok.se for the Teahouse main page.

Glossary of Research Terms[edit]

New editor. A registered editor who either a) had made fewer than 100 edits at the time of their invitation and/or first visit to Teahouse, or b) created their account in 2012. Similar to the E1-99 editor category used by the Wikimedia Foundation for research purposes.

Host. An experienced Wikipedian who participates in the Teahouse. Most hosts have created profiles for themselves on the Teahouse Hosts page.

Guest. An editor (registered or unregistered) who makes at least 1 edit to the Guests page, the Questions page, or the Teahouse talk page.

Visit. An edit by a non-host to the Guests page, the Questions page, the Teahouse talk page, or the Host Lounge talk page.

Automated Invites. Invitations to a specific set of new editors sent automatically on a daily basis by HostBot. First implemented on 7/23/2012.

Response rate. Percentage of editors invited through a particular mechanism (such as a personalized HostBot invite) who subsequently visited the Teahouse.


WP:Teahouse[edit]

We analyzed guest and host activity patterns over time, and asked a sample of recent guests to rate and describe their overall experience on the Teahouse. We found that participation by guests and hosts has increased over time, that most new editors enjoy their experience, and that women editors participate more than expected.

It was nice that it existed or else I wouldn't have created the article. I don't know how you would ever get new editors without it. It's great and necessary.

Findings[edit]

» More new editors are participating[edit]

The total number of new editors participating in the Teahouse Q&A board and/or the Guests profile page has increased by 26% over the pilot period, from 34 new editors per week 42. The greatest increase has been seen in the three months that automated invites have been sent out.

Total new editors participating in Teahouse/Questions and Teahouse/Guests pages, per week. Feb27 - Oct11 2012
New editor participation
total new editors participating 1098
total new editors participating per week 34.2
during pilot 32.9
since automated invites 41.5


» New editors give the Teahouse high marks[edit]

68% of new editors surveyed said that they were "Satisfied" or "Very satisfied" with their Teahouse experience, versus only 5% who said they were "Dissatisfied" or "Very dissatisfied". 78% of new editors surveyed indicated that they would return to the Teahouse.

How satisfied were you with your overall experience on Teahouse?
Answer # responses %
Very Dissatisfied 2 2%
Dissatisfied 3 3%
Neutral 24 27%
Satisfied 30 34%
Very Satisfied 30 34%


Do you think you will participate in the Teahouse in the future?
Answer Response %
Yes, frequently 15 17%
Maybe once in a while 54 61%
No, definitely not 4 4%
I'm not sure/I haven't decided 16 18%
Total 89 100%


When asked to describe what in particular they liked about their experience, new editors cited a range of factors, from the promptness and quality of the answers they received to the friendly atmosphere and the ease of use.

As mentioned previously, the speed of the responses. Sometimes, particularly if you're concerned about something, it's nice to know there is a way of getting an (almost) immediate response. It makes it feel as if there IS someone to talk to about things.


It was not that interesting that I could go back again. There was really nothing to do.


just grateful you guys are there so that I know I have somewhere to go in times of need and that the response is fast and friendly. Thank you.


» Female newcomers participate in Teahouse at a higher rate[edit]

30% of respondents to our most recent survey identified themselves as women. This figure is in line with the results from our two previous surveys, in which 22% (13/59) and 35% (17/48) respondents identified as women. The overall rate of female participation, as measured by these survey responses, was 29%.

If you're comfortable telling us, what is your gender?
I am female 27 30%
I am male 51 57%
I'd rather not say 11 12%
Total 89 100%
Cumulative survey results: Female participation
Survey Date Female Respondents Total Responents % Female
Survey 1 April 13 59 22%
Survey 2 June 17 48 35%
Survey 3 October 27 89 30%
Total 57 196 29%

This is a noticeably higher rate than figures for female participation in Wikipedia as a whole, which ranges between 8% and 13%. This finding strongly suggests that the Teahouse is making progress toward one of the project's original goals of creating an appealing space for new female editors to get support, by offering a new approach to help on Wikipedia.

These survey numbers are likely to accurately reflect the true proportion of female Teahouse participants. New editors who received one survey were not asked to participate in subsequent surveys, whether or not they had chosen to fill out the first. And although it is possible that during the pilot period some female editors were invited to participate by Teahouse hosts specifically because they self-identified as women on their Wikipedia, the majority of respondents to the third survey were new editors who had been invited automatically by HostBot, with no consideration for their gender.


» Most guests either ask a question or create a profile[edit]

We were surprised to find that most new editors who visit the Teahouse only participate in one activity. While 70% of new editors ask a question and 40% create a profile, only 10% do both.

Relative proportion of newcomers participating in Teahouse/Questions vs Teahouse/Guests pages, Feb27 - Oct11 2012


New editor participation by page
% of new editors who participated in Teahouse/Questions 70%
% of new editors who created a Teahouse/Guests profile 40%
% of new editors who participated in both /Questions and /Guests 10%

Since most guests visit the Teahouse for particular reasons, suggesting that one strategy for increasing participation beyond the current level would be to offer additional calls to action or new ways to participate. Some possibilities might be new calls to action around suggesting editing tasks for newcomers, creating new discussion forums not focused on Q&A, and better surfacing of WikiProjects that would welcome new members.


» More hosts are participating[edit]

The total number of hosts participating in the Teahouse (by answering questions on the Q&A board, building pages, and participating in talk page discussions) has increased by 19% over the pilot period weekly average, from 21 hosts per week to 25.

Hosts participating per week on WP:Teahouse, February - October 2012
Host participation
total hosts participating 77
average hosts participating per week 21.4
during pilot 20.7
during phase 2 24.7

Q&A board[edit]

The Teahouse passed the 1,000 questions milestone during Phase 2, and the Q&A board continues to be the most active area on the Teahouse. Most questions are asked by new editors, and most are answered by Teahouse hosts, although many experienced Wikipedians also participate in Q&A by answering questions as 'unofficial hosts'. Questions span a wide variety of topics: and while some are relatively simple and easily 'resolved', many spark detailed, animated discussions.

Teahouse/Questions
total questions asked 1381
average questions per guest 1.6
% of questioners who asked more than one question 23%
% of questioners who respond in their own question thread 48.6%
average responses per question 2.9
median time to first response 33:20 minutes
questions per day 6
during pilot 5.6
since automated invites 6.8
questions per week 42.2
during pilot 39.5
since automated invites 47.7


Findings[edit]

The Q&A board seems to be a very stimulating and interactive place for new editors. 80% of respondents stated that they were 'satisfied' or 'very satisfied' with the quality of the answers on the Q&A board. 88% of editors stated that they received a follow up message from a Teahouse host after they asked their question (generally a Teahouse talkback template, and/or a personal message). However, most guests are still hesitant to answer questions.


» Question volume has increased[edit]

The Q&A board was experiencing a lull in activity at the start of phase 2, but activity increase noticeably after we implemented automated invites on July 23rd and have remained high ever since. Current Q&A activity (48 questions/week) is 20% higher than during the pilot period (40/week) and 45% higher than during the transition period (33/week) between the end of the pilot and the deployment of automated invites.

Questions per week on WP:Teahouse/Questions, February - October 2012


» Many questions receive a response in 30 minutes or less[edit]

Teahouse hosts answer guests' questions rapidly. The median time between when a question is asked and its first response is 33 minutes and 20 seconds. Average response time has been shrinking week by week, possibly as a result of increasing host participation.

Median response time to questions on WP:Teahouse/Questions, by week, February - October 2012


» Each questions receives around 3 answers[edit]

Teahouse guests can expect several answers to their question. Each question on the Q&A board receives an average of 2.93 replies from one or more other editors. However, the number of answers per question has decreased since the end of the pilot period.

Mean number of answers to questions on WP:Teahouse/Questions, by week, February - October 2012


» New editors were very satisfied with the answers they received[edit]

Did you ask a question on the Teahouse Q&A board?
Answer # responses %
Yes 69 74%
No 24 26%
Total 93 100%


How satisfied were you with the quality of the answer you received?
Answer Response %
Very Dissatisfied 2 3%
Dissatisfied 3 5%
Neutral 8 12%
Satisfied 26 39%
Very Satisfied 27 41%
Total 66 100%


When asked to describe what in particular they they liked about the answers they received, new editors cited a range of factors, from the promptness and number of answers they received to the friendly tone and the level of detail.

They just seemed like they wanted me (a rank beginner) to succeed.


I didn't like the way the person explained. I have heard from better Wikipedians. They needed to explain properly.


» New editors are still hesitant to answer questions[edit]

Our findings here are similar to those from the pilot metrics report. While 74% of new editors surveyed indicated that they had asked a question on the Q&A board, only 7% said they had answered one. If the Q&A board is meant to be a peer-support space, this is still room for improvement in this area.

Why haven't you answered any questions on the Q&A board?
Answer Response %
I couldn't figure out how to answer a question 6 7%
I didn't know I was allowed to answer questions 23 28%
I didn't see any questions I knew the answer to 20 24%
I didn't feel like it 5 6%
some other reason (please describe) 29 35%
Total 83 100%


When asked to describe in their own words why they had not answered a question on the Q&A board, many guests said that they did not feel they had the expertise to answer a question, or that they were concerned about making mistakes or giving incorrect answers.

I didn't know I was allowed to answer questions as given above but I also don't feel I'm sufficiently experienced with the intricacies of Wikipedia!


I was just lurking but I saw a question that was obvious and that I knew with strong certainty

Guests page[edit]


Findings[edit]

» New editors enjoyed browsing other guests' profiles and creating their own[edit]

Around 40% of survey respondents stated that they had created a profile. Guests' levels of satisfaction with the profile creation process was comparable to the level of satisfaction reported during the pilot period, with 68% of respondents saying they were 'satisfied' or 'very satisfied'.


Did you introduce yourself on the Guests page?
Answer # responses %
Yes 38 43%
No 51 57%
Total 89 100%


How satisfied were you with the experience of creating your guest page introduction?
Answer # responses %
Very Dissatisfied 0 0%
Dissatisfied 1 3%
Neutral 11 29%
Satisfied 19 50%
Very Satisfied 7 18%
Total 38 100%


Guests highlighted the ease of creating a profile and the ability to browse other profiles as particularly beneficial. No guests identified any aspects of Guest profiles that they especially disliked. The majority (55%) of guests who did not create a profile said that they didn't know it was there. Others said that they could not figure out how to create a profile and cited lack of familiarity with markup. One guest said they weren't comfortable personal information on the internet, and another said "I am only interesting in posting information that should/could be of interest to others. my mission here had nothing to do with me."

I get to put myself out there as a part of a thing far greater than myself, a thing that all of us share, because it's not just people's Wikipedia, but it's also mine


Why didn't you introduce yourself?
Answer # responses %
didn't feel like it 5 10%
couldn't figure out how 6 12%
didn't know it was there 28 55%
didn't want to be visible to others 2 4%
didn't want to be associated with the Teahouse 0 0%
some other reason (please describe) 10 20%
Total 51 100%


As I'm unsure of the etiquette within Wiki, which I feel is the reason I ended up embroiled in the situation I was asking advice about, I'm fairly weary of drawing undue attention; does that sound right or just plain silly?


» More new editors are creating profiles[edit]

Although profile creation declined over the course of the pilot period, it increased substantially over the course of Phase 2. Between July 23rd and October 11, an average of 19.3 new profiles were created per week, compared with 14.4 new profiles per week during the pilot period, a 34% increase.

Number of new editors creating profiles on Teahouse/Guests page, per week. Feb27 - Oct11 2012


Teahouse/Guests
total profiles created 482
profiles per day 2
during pilot 2.1
since automated invites 2.76
profiles per week 14.2
during pilot 14.4
since automated invites 19.3

We highlight two factors in particular which likely contributed to this increase:

  • The implementation of an 'archiving' script that limited the number of profiles on the main Guests page to 10-20 of the most recent profiles. Profiles of previous guests were archived to the Teahouse 'guestbook'. This change was intended to increase the perceived value of creating a profile by making new profiles more visible.
  • The implementation of automated invites allowed us to reach out to more new editors each day. These new editors visited the Teahouse at the same rate as those guests who were invited manually during the pilot period. However, where many manual invites were sent to guests who a host had judged as needing help, automated invites were sent to guests regardless of whether or not they currently needed assistance. We believe that since fewer of these guests had a questions in mind when they visited, many more of them created profiles instead.

Invitations[edit]

New editors find out about the Teahouse in a variety of ways. During the pilot period, manual invitations were the primary mechanism. During Phase 2, several other mechanisms were developed.

Manual Invitations

During the pilot period, all invitations to visit the Teahouse were sent manually by Teahouse hosts, with two hosts (Rosiestep and SarahStierch) sending 80% of all tracked invitations.

While this process was effective at bringing new editors to the Teahouse, the response rate was quite low: only around 4.5% of those who received an invite subsequently visited Teahouse. Anecdotally, hosts who sent many invites expressed frustration at this low rate of return, considering how labor intensive the process of manually inviting editors was. Many editors continued to manually invite new editors to the Teahouse during Phase 2.

External links

Because invitations are crucial for driving traffic to the Teahouse, but manual invitations alone was not a sustainable strategy, we decided to experiment with two other mechanisms for making new editors aware of the Teahouse: automated invitations and a custom Article Feedback call to action.

The Wikipedia Help Desk also added a link to the Teahouse, and a special message to new editors, to their page header. The community has also added Teahouse links to many other help spaces, such as:

We have not explicitly tracked inbound traffic or the response rate from these sources.

Automated invites

We developed a Python script that automatically invited new editors who met our basic invitee criteria.

On July 23rd, HostBot was approved by the Bot Approvals Group to begin sending automated invitations to all new editors listed on the invitee reports. While the goal of the Teahouse is to be as inclusive as possible, after conversation with community members we decided to program HostBot to not invite users who had received level 4 warnings, or who had been accused of sockpuppetry.

In evaluating the long-term effectiveness of automated invites, we were particularly interested in answering the following questions:

  • Are new editors more or less likely to respond to an invitation sent by a bot than an invite sent by human?
  • Are new editors more likely to respond to a personalized invitation or a generic one?
Article Feedback v5 call to action

We worked with the Wikimedia Foundation's Editor Engagement team to implement a Teahouse Call to Action (CtA) for the article feedback tool, which would be displayed to 10% of registered editors who submitted article feedback. The Teahouse CtA was rolled out on August 21st, and has (as of October 14th) been displayed to 881 editors who filled out Article Feedback.

Findings[edit]

» Direct invitations continue to drive traffic to Teahouse[edit]

During the pilot period, we found that the majority of Teahouse guests were invited guests. This pattern continued in Phase 2, with 44% of survey respondents stating that they were invited to the Teahouse via a talk page invitation.

How did you find out about Teahouse?
Answer # response %
Someone invited me on my own talk page. 41 44%
Someone emailed me an invitation. 6 6%
Another editor referred me to Teahouse. 17 18%
I am a member of a class, and my instructor told me about Teahouse 0 0%
I read about Teahouse on Wikipedia. 22 23%
Other (please describe below) 8 9%
Total 94 100%
I was looking for help/advice and reached it through the help pages.


» Automated invites yield the same response rate as manual invites [edit]

HostBot sent out a total of 4268 invites between July 23rd and October 11th, and 165 (3.9%) of invitees subsequently participated in the Teahouse (asked or answered a question and/or created a profile). This response rate is not statistically different from the response rate (4.27%) for manual invitations sent to invitee report candidates during the pilot period, as determined by a 2-tailed t-test (t=0.68, p > .05, 2-tailed).

Despite some initial concerns that removing editorial discretion from the invitation process might lead to an increase in vandalism or disruptive editing on the Teahouse, we have not observed any such increase in the time that automated invites have been running.

Given the findings above, and the facts that automated invitations are much less labor-intensive than manual invites and have contributed to an increase in new editor participation, we view automated invitations as a key component of the Teahouse's continued success.


» Personalized messages did not yield significantly different results from generic invitations[edit]

Previous research has determined that adding a degree of personalization to templated talk page messages (for instance, by using active voice and calling out the messager as a fellow editor) can yield better responses in some scenarios than generic messages. We tested the effectiveness of personalization in automated invites by including a personal signature from a Teahouse host in roughly 50% of messages, while leaving the other 50% generic.

We found that adding personalization to an automated invitation does not affect response rate. A sample of 2435 automated invites yielded response rates of 3.8% for both personalized and generic invitations. Findings and a full discussion of these results can be read on the invites experiment page.

Given this result, we decided to make all invitations personalized. A personalized invitation gives the invitee a personal point of contact that they can message directly if they choose to, and may help new users feel like they are a part of a community. We currently have no plans to continue these experiments.


» Few AFT respondents subsequently visited the Teahouse[edit]

Out of the 881 registered editors who received the Teahouse call to action after submitting their article feedback, 7 (0.08%) have visited the Teahouse so far.

Page traffic[edit]

The main Teahouse page was viewed 39,678 times between February 27th and October 13th, for an average of 173 (non-unique) pageviews per day. The graph at right shows page views per day during that period.

Unique page views, by day, for WP:Teahouse. February 27th - October 14th 2012.

» Page traffic has increased over time[edit]

The graph above shows that page traffic has increased in recent months. There were an average of 155 pageviews per day during the pilot period, versus 204 views per day since the introduction of automated invites. The dramatic spike in pageviews around July 15th corresponds to Wikimania 2012, during which the Teahouse was discussed in several presentations and panels.

Teahouse pageviews
total page views 39,678
page views per day 155
during pilot 154
since automated invites 204

» Most page requests come from the US and UK[edit]

An analysis of a 10% sample of all pageviews for WP:Teahouse and its subpages shows visitors from 101 different countries. Over 65% of all page requests come from within the United States, followed by the United Kingdom with 10% of all page requests. Not surprisingly, this distribution of page requests by country closely parallels that of en.wikipedia.org.

pageviews of WP:Teahouse and its sub-pages between 4/05/2012 and 10/01/2012, by country of origin

New editor retention[edit]

Sample: 149 Teahouse visitors and 1954 non-vistors invited between 2/27 and 7/25. Edits to Teahouse pages not included in counts. Only non-visitors who made at least 1 edit to Wikipedia after the date of their invite are included.

In order to determine whether participating in the Teahouse has a positive effect on an editor's likelihood of continuing to edit Wikipedia and on their level of editing activity, we analyzed the editing behaviors of a sample of editors who were invited to the Teahouse by a host between February 27th and July 25th. Editors in this sample were invited (manually) to the Teahouse because their names appeared in the daily Invitee report or their submission was rejected by Articles for Creation.

To control for the possibility that new editors who did not visit had simply not seen the invitation, we excluded from our sample those who did not make at least one edit to Wikipedia after the date of invitation. This left us with a sample of 1,948 editors, 149 of these editors visited the Teahouse. For the editors who visited Teahouse, any edits they made to WP:Teahouse (and its sub-pages) were ignored in this analysis. We also ignored any edits made by editors in our sample prior to their invitation.

Our main findings from this analysis are highlighted below. Because our edit data are not normally distributed (in both groups, a small number of editors made hundreds or thousands of edits while most made from a few to a few dozen) we present both mean and median averages, where appropriate.

Editing activity by Teahouse visitors vs. non-visitors
visitors non-visitors
mean post-invite edits 241 75
median post-invite edits 29 12
mean weeks with edits 5.75 3.65
median weeks with edits 4 2
mean articles edited 76.5 21.2
median articles edited 5 2
% with article edits 74% 81%
% edits to talk namespaces 35% 20%

These findings show that Teahouse visitors are more active than new editors who do not visit Teahouse, and that they exhibit certain positive behavioral characteristics (such as editing more than one article, or participating in talk page discussions) to a greater degree than editors in the control group. While the findings presented below do not prove definitively that participating in the Teahouse causes editors to exhibit these behaviors, that is a plausible interpretation, given our findings on editor activity and satisfaction, presented above. For instance, it is easy to imagine how getting a helpful answer from a friendly Wikipedian at the Teahouse could make a new editor feel more comfortable and confident participating in talk page discussions. However, even taking the position that the Teahouse tends to attract editors who are most likely to become active, productive contributors anyway, we believe that the Teahouse provides these editors with better and earlier assistance and more positive community interaction than they would likely have received otherwise.

They gave me some specific pointers, but more than that they made me feel welcome and like I had someone to turn to for help.

Findings[edit]

» Teahouse guests make more edits to more articles[edit]

On average, Teahouse guests made more edits (241 versus 75), and edited more articles (76.5 versus 21.2). This is encouraging since many new editors start out with the intent to create or contribute to a single article. Encouraging new editors to branch out and edit more articles may lead to higher long-term retention.

Average cumulative edits by Teahouse visitors vs. invited non-visitors, February - October 2012.
Average number of articles edited by Teahouse visitors vs. invited non-visitors, February - October 2012.

» Teahouse guests edit more each week, for more weeks[edit]

The Teahouse visitors in our sample edited roughly twice as long on average as non-visitors, measured by the number of weeks after being invited in which they made at least 1 edit to Wikipedia. They also edited more on average during weeks when they did edit. We analyzed the subsequent editing activity of 1,655 editors (133 invited visitors, 1,522 invited non-visitors) from the pilot period (February 27th - May 27th).

Teahouse visitors made 7 edits per week on average, while invited non-visitors made 2.1 Teahouse visitors also made more mean edits in 21/21 (100%) of all weeks, and more median edits in 17/21 (81%) of weeks.* The first chart below (left) shows mean and median edits-per-week for the two groups during the subsequent 21 weeks (May 28th - October 18th).

The second chart below (right) shows the average increase in cumulative edits over time for these two groups. Teahouse visitors invited during the pilot period had over twice as many edits on October 21st as they had on May 28th (an 115% increase), while the non-visitor group increased their total edit counts by only 77% over that 5 month period.

Mean and median edits per week by Teahouse visitors vs. non-visitors who were invited during pilot, between May 28 - October 18 2012.
Increase in average total edits per editor between May 28 - October 21, for Teahouse visitors and non-visitors invited during the pilot period.
*The per-week medians above only count editors who made at least 1 edit in that week. Because a high proportion of editors in both groups will have made no edits in each given week, the traditional per-week median for both groups would be 0 for all weeks.

» Teahouse guests communicate more with other editors[edit]

Teahouse guests also made roughly 75% more edits to the primary talk namespaces (User_talk, Article_talk and Wikipedia_talk) than other new editors. This suggests that Teahouse guests participate more in talk page discussions and may be more involved in community processes than new editors who chose not to not participate.