Research:Teahouse/Phase 2 plan

The Teahouse pilot period is over, but the Teahouse project continues! As we shift focus from concept experimentation to deepening the proven concept's roots on English Wikipedia, this page will serve as a planning space to scope the work that lies ahead for Phase 2 of the Teahouse project.

We need your help!

The WMF project team continues to be actively engaged in Teahouse during phase 2, but we would like to encourage even more participation from community volunteers during this second phase.

Please review the plan and add your name in the space provided below if you're interested in participating in some capacity as work moves forward in each priority area - we will need your help in the form of feedback, testing, development, process consulting, expert leadership, and more.

Goals

Phase 2 of the Teahouse project will focus on improvements and continued experimentation in order to grow the project on English Wikipedia, with the following goals in mind:

• Scale traffic and participation - Teahouse is a many-to-many support system, and should allow as many editors to participate as possible
• Move towards sustainability - Wikipedia is and should be sustained by the efforts of volunteers, and Teahouse should have efficient, proven systems and processes in place that allow volunteers to keep the project running for years to come
• Increase opportunities for peer support and engagement - Teahouse is a space intended to help each other become better Wikipedians, so work should continue on features that increase opportunities for editor support (Note: by peers here we also mean guests supporting guests and hosts supporting hosts, in addition to continuing the trend of hosts supporting new guests)
• Demonstrate longer-term impact of participation on new editor retention - Teahouse is thought to support increased engagement and retention of new editors, and regular measurement and comparison of participants' editing patterns against a control should help prove or disprove this hypothesis

Targets

To measure phase 2 progress towards stated goals, we've set the following targets:

• Double the number of active hosts per week, from 15 to 30
• Triple the number of new editors participating in Teahouse each week (measured by intro creations and edits made to Q&A forum), from 40 to 120
• Decrease project team time spent on daily/weekly maintenance tasks by 50%, and demonstrate an increase in the amount of daily/weekly project contributions made by non-staff participants, particularly for activities outside of answering questions
• Continued demonstration of measurable increase in Teahouse guests' Wikipedia editing activity compared with control over time

Metrics

Monthly project metrics will continue to be run, in order to measure progress towards targets. We will focus on measuring two kinds of metrics: participation metrics (who is participating in Teahouse, how, how many are participating, and how often) and retention metrics (the subsequent editing patterns and attrition rate of Teahouse guests). Specific measures will include:

• Participation
  • Teahouse/Questions
    • How many hosts and guests participate per week
    • How many questions are asked
      • Per week
      • Broken out by new editors vs. established editors
    • How many questions per guest
    • Average response time per question
    • Average responses per question

• • Teahouse/Guests
    • How many guest intros are created per week

• Retention
  • Comparison of editing activities of Teahouse guests vs. 'control' groups of new editors
  • Comparison of attrition rate of Teahouse guests vs. 'control' groups of new editors

Priority work

To accomplish the goals of Teahouse phase 2, some aspects of the project require further automation, some need stabilization, some processes need to be further developed or streamlined, and some feature enhancement is also required.

• 7 key areas have been identified as priorities for continued project development
• Tasks have been broken down within each area, as well as motivations behind the selection of these tasks
• Challenges outline the research question or issue the task is intended to resolve
• Work sprints will be organized to cluster work being completed in each area

For background on how these priorities emerged and supporting data, please see the Teahouse pilot report.

Invites

This part of phase 2 is in progress! Please see the documentation page and get involved!

Task - Experimenting with bot-sent invites Done

• Motivation - Using talk page templates to invite new editors has proven effective. However, manually inviting new editors is time-consuming, and a relatively high volume of invites (50-100 a day) will be necessary to grow participation.
• Challenge - Are bot-sent invites as successful for bringing in new guests as host-sent invites? Will this discourage emailing new editors, which is more effective than templates alone?

Task - Experimenting with invite language Done

• Motivation - Many messages left on talk pages do not have a very personal or welcoming tone, and many messages are confusing and too long. We have kept the Teahouse invitations short and sweet, but we haven't experimented with different types of messages yet.
• Challenge - Does what the invite says make a difference?

Task - Re-working personal invite process Done

• Motivation - personal invitations are more engaging than generic ones, and human judgement is also useful for identifying new editors who appear to be especially motivated, or in need of extra help.
• Challenge - Can we encourage some Wikipedians to continue to personally invite new editors? Can we encourage more email invites, which have demonstrated more success at reaching new editors than talk pages?

• I'm supportive of work being done on invites and may want to help, please keep me updated when the time comes:

1. Rosiestep (talk) 05:53, 28 June 2012 (UTC)
2. Ryan Vesey (talk) 23:14, 29 June 2012 (UTC) (You'll probably need to leave a comment on my en-wiki page)
3. Doctree (talk) 03:36, 5 July 2012 (UTC) (Please leave message on my en-Wikipedia page)
4. Hajatvrc (talk) 19:18, 2 August 2012 (UTC) (Same for my talk page at en-wiki)
5. Your name here!

Hosting

This part of phase 2 is in progress! Please see the documentation page and get involved!

Task - Host intro creation. Call to action for all experienced Wikipedians to enter Teahouse as hosts, incorporate Host Tips into intro creation process to encourage use of the Teahouse response style. Done

Challenge - Can we make hosting a more lightweight process that gets more Wikipedians to host while still preserving the warm Teahouse experience for guests?

Task - Updated and simplification of host lounge and your hosts pages. Clear calls to action so that everyone can participate in Teahouse in whatever capacity they can.

Challenge - Can we make hosting a more lightweight process that gets more Wikipedians to host while still preserving the warm Teahouse experience for guests?

Task - Host activity surfacing. Lightweight check-in process for hosts arriving and automated archiving of host intros to the breakroom after 1 week inactivity.

Challenge - Can we make it easy for guests to see which experienced Wikipedians are active in the Teahouse in any given week, without making more work for hosts?

Task - Surface talk-back template process/scripts for easier use by anyone answering questions

Challenge - As more editors drop in to answer questions, can we still ensure guests are notified of responses and encouraged to return to Teahouse again?

• I'm supportive of work being done on hosting and may want to help, please keep me updated when the time comes:

1. Your name here!

Guest Intros

Task - Simplify intro-creation process Done

Challenge - Can we reduce user-error rate in profiles created, and encourage more guests to create profiles?

Task - Archiving system for guest intros Done

Challenge - Can we automate archiving to the guestbook with pagination etc?

Task - Add interactive call to action for welcoming guests who create intros, along with a tracker in the host lounge for those who want to welcome new guests

Challenge - Can we encourage some hosts to welcome new guests more often?

Task - Build follow-up activities into intro-creation process, by placing clearer calls to action for what to do next on success landing page Done

Challenge - Can guests who create intros be encouraged to do a second activity?

• I'm supportive of work being done on guest introductions and may want to help, please keep me updated when the time comes:

1. Your name here!

Q&A

Task - Make replying to questions a clear call to action

Challenge - Can more new editors be encouraged to reply to questions by replacing the edit button with a clearer call to action?

Task - Highlight topics or areas specifically for new-on-new discussions or non-expert type questions

Challenge - Can simple interface changes encourage more new-to-new discussions, creating new opportunities for peer support while keeping the encyclopedia focus?

Task - Usability test on top-posting questions

Challenge - Can we demonstrate that top-posting does create a better experience for new editors, and solve this open question once and for all?

• I'm supportive of work being done on Q&A and may want to help, please keep me updated when the time comes:

1. Your name here!

Acknowledgement

Task - Build acknowledgement into daily Teahouse activities - easy ways for thanking hosts and guests, regularly acknowledge and reward each other for contributions, perhaps experiment with badges?

Challenge: Can more participation be encouraged through more early and often acknowledgment? Does this demonstrate any impact on the gender gap?

• I'm supportive of work being done on acknowledging contributions and may want to help, please keep me updated when the time comes:

1. FloNight (talk) 16:41, 4 July 2012 (UTC)
2. Your name here!

Outside Connections

Task - Experiment with at least 1 new hook in another new-editor-facing feature Done

Challenge - Are there places in the Wikipedia interface where a link to Teahouse will dramatically increase traffic of new good faith editors, but not flood the space with reader questions, vandals, or other off-topic posts?

Task - More incorporation with other help pages/processes Done

Challenge - Do links to the Teahouse from the help desk, Help:Contents, or other help pages/processes encourage new editors to get the help they need, without interfering with what's working well in those other systems? Do links to the most useful set of help documentation or tutorials enhance the experience for new editors?

Task - Mechanisms to route new editors to active WikiProjects based on interests

Challenge - Can we encourage new editor retention by pointing guests toward active topic-based community projects?

• I'm supportive of work being done to interlink Teahouse with other projects and may want to help, please keep me updated when the time comes:

1. Your name here!

Maintenance

Task - Template protection Done

Challenge - Can we ensure vandalism doesn't scale along with traffic, but still allow easy editing of pages that need to be edited by all?

Task - Teahouse Maitre d' role creation

This part of the phase is in process, please visit this documentation page to learn more!

Challenge - Can we organize volunteers to take on administration-type tasks of the Teahouse, including moderation and enforcement of house rules, invite checking, etc, to maintain a clean and friendly space for new editors

Task - Metrics automation

Challenge - Can we make monthly metrics a plug-and-play system so hosts and the project team continue to have a good understanding of Teahouse's basic health levels in terms of invites, traffic, activity, and other important stats?

• I'm supportive of maintenance work being done and may want to help, please keep me updated when the time comes:

1. Rosiestep (talk) 16:54, 30 June 2012 (UTC), This interests me, especially the Teahouse Maitre d' role.
2. FloNight (talk) 16:38, 4 July 2012 (UTC)
3. Hajatvrc (talk) 20:57, 10 September 2012 (UTC)

Out of scope

Some things we'd really like to do, but don't have resources yet to accomplish:

• Adding profile image thumbnails to signatures in Q&A forum, to tie the 2 areas together and further personalize Q&A interactions
• Integrating real forum software into the Q&A forum, to resolve edit conflict issues and further scale the feature
• Connect guests based on interests, tags in intros, etc

We hope some of these will be tackled in a future phase of the project!

Timeline

Phase 2 of the EN:WP Teahouse project will run for 4 months, from June 11 through October 11 2012. Below is an estimated timeline of when each area of work will be tackled.

June

• Communicate pilot learnings
• Plan and prioritize Phase 2 work
• Investigate and secure available resources for Phase 2

July

• Invites
• Hosting
• Guest Intros

Aug

• Maintenance
• Outside Connections
• Q&A

Sept

• Acknowledgement

Oct

• Phase 2 metrics, project assessment and reporting