Research:Teahouse/Phase 2 report/Metrics
|See the Phase 2 report for more information on project goals, activities and outcomes.|
» findings on editor participation, engagement and retention from February 27th through October 11th, 2012
WP:Teahouse was designed to introduce new editors to the Wikipedia community in a friendly and engaging way. Between February 27th and October 11th, 1,512 editors participated in Teahouse. 73% of these participants (1,098) were new editors. Guests asked 1,381 questions on the Teahouse Q&A board, and created 420 profiles. 77 editors signed up to be Teahouse hosts.
We report on Teahouse activity and impact during this roughly seven-month period. We focus our analysis on experiments we ran and new features we added during Phase 2 of the project. Data specific to the Teahouse pilot period (February through May 2012) can be found in the pilot metrics report.
To assess editors’ perceptions of Teahouse and their experiences during the post-pilot phase of the project, we surveyed 400 Teahouse guests who visited Teahouse between July 1st and October 1st. Our response rate was fairly high: 22% (89/400) of new editors who were offered the survey responded.
- Edit logs
In order to assess activity on Teahouse we tracked edits to Teahouse pages by new editors and veterans on a live mirror database of en.wikipedia.org. We tracked invitations sent to guests by HostBot and aggregated editing activities of Teahouse guests with MySQL queries and Python scripts.
- Page views
Glossary of Research Terms
New editor. A registered editor who either a) had made fewer than 100 edits at the time of their invitation and/or first visit to Teahouse, or b) created their account in 2012. Similar to the E1-99 editor category used by the Wikimedia Foundation for research purposes.
Host. An experienced Wikipedian who participates in the Teahouse. Most hosts have created profiles for themselves on the Teahouse Hosts page.
Response rate. Percentage of editors invited through a particular mechanism (such as a personalized HostBot invite) who subsequently visited the Teahouse.
We analyzed guest and host activity patterns over time, and asked a sample of recent guests to rate and describe their overall experience on the Teahouse. We found that participation by guests and hosts has increased over time, that most new editors enjoy their experience, and that women editors participate more than expected.
|It was nice that it existed or else I wouldn't have created the article. I don't know how you would ever get new editors without it. It's great and necessary.||
» More new editors are participating
The total number of new editors participating in the Teahouse Q&A board and/or the Guests profile page has increased by 26% over the pilot period, from 34 new editors per week 42. The greatest increase has been seen in the three months that automated invites have been sent out.
|New editor participation|
|total new editors participating||1098|
|total new editors participating per week||34.2|
|since automated invites||41.5|
» New editors give the Teahouse high marks
68% of new editors surveyed said that they were "Satisfied" or "Very satisfied" with their Teahouse experience, versus only 5% who said they were "Dissatisfied" or "Very dissatisfied". 78% of new editors surveyed indicated that they would return to the Teahouse.
|How satisfied were you with your overall experience on Teahouse?|
|Do you think you will participate in the Teahouse in the future?|
|Maybe once in a while||54||61%|
|No, definitely not||4||4%|
|I'm not sure/I haven't decided||16||18%|
When asked to describe what in particular they liked about their experience, new editors cited a range of factors, from the promptness and quality of the answers they received to the friendly atmosphere and the ease of use.
|Was there anything in particular you liked about your Teahouse experience? Click 'Show' to view all responses|
|Tea House is a very good for new users because new users can learn many things at it.|
|It's very pretty compared to the rest of Wikipedia. I enjoy the tea house decor in real life and it was refreshing to see something like it on Wikipedia.|
|Quick and firm responce.|
|friendly helpful people|
|its a good way for newbies to get used to Wikipedia style and prtotcols|
|as of yet, nothing in particular, the welcome was warm enough, perhaps a follow up would have been nice.|
|It was somehwere to get soem help which is great|
|It's a great place to get started (for new users).|
|My questions were answered very quickly.|
|The fact that it exists and was helpful to me.|
|The fact that there is somebody "out there", that there is a sincere community, gives a professional and safe feeling about Wikipedia|
|As mentioned previously, the speed of the responses. Sometimes, particularly if you're concerned about something, it's nice to know there is a way of getting an (almost) immediate response. It makes it feel as if there IS someone to talk to about things.|
|The editors are very friendly and patient, which is great when compared to the rest of Wikipedia in how new editors are treated.|
|I got my question answered. It has a nice friendly non-techie look to it.|
|I received answers to my questions regarding editing|
|Questions were answered quickly, the atmosphere felt very supportive, and most importantly, I received very helpful and clear answers to questions.|
|Yes, I liked that you can still edit things if you make any mistakes.|
|I think it is a very organized system and an improvement for the Wikipedia community.|
|All hosts were very welcoming and understanding! The Teahouse has greatly helped me to overcome being timid about editing! Thank you all sooo much! I have so much appreciation for you.|
|In spite of feeling pretty lost as far as how to create a Wikipedia page, everyone was very friendly and accepting.|
|I liked meeting other wikipedians. It's very interesting to know about them.|
|Senior editors like Sarah, NtheP and one other his moniker is hard to remember :-) are all prompt in providing answers so I can move forward.|
|I like how I could meet more experienced editors, and I felt very welcome when somebody invited me to the Teahouse.|
|It was nice that it existed or else I wouldn't have created the article. I don't know how you would ever get new editors without it. Its great and necessary.|
|got a quick and helpful answer|
|Yes! It is superb and very friendly. Love!|
|Heather was very kind and supportive after my terrible blunder. I just wish the Teahouse was more visible especially when someone creates a new account.|
|As mentioned previously, the speed of the responses. Sometimes, particularly if you're concerned about something, it's nice to know there is a way of getting an (almost) immediate response. It makes it feel as if there IS someone to talk to about things.||
|Was there anything in particular you disliked about your Teahouse experience? Click 'Show' to view all responses|
|The people were rude.|
|It was not that interesting that I could go back again. There was really nothing to do.|
|See comments above.|
|There is nothing to do except look at people's profile. I thought it would be more excited and creative.|
|It was not that interesting that I could go back again. There was really nothing to do.||
|Is there anything else you'd like to share with us? Click 'Show' to view all responses|
|Overall WMF projects are really a neat idea for the sharing of educational and especially free-based information. The teahouse is definately on the right track. I am sure I will return periodically to see how it develops and after this survey may even participate in question answering or other things needing mentioning.|
|Just that I have tons of admiration for the collaborative nature of the your hard work and want to be involved more|
|Nothing else, except thank you for giving me this survey! I was glad to give some feedback.|
|I don't function very well in team sports.|
|No but it was helpful.|
|first, thanks very much to those that helped me. The answers were not particularly helpful but they made me realize that this is a huge trial and error system in which the smallest details mean a lot. I don't even know yet how to ask a question so that it can be answered.|
|That last question was not very helpful. 'Female' is not a gender, it is a sex. As such, you can have technically 'female' men and 'male' women. It's also rather demeaning to refer to women as 'females' because it is the way you'd speak about an animal.|
|I don't understand why the photos are on a site called Wikimedia Commons, rather than Wikipedia Commons. I can't see any link from the Wikipedia site to the Commons site.|
|this is an amazing, frustrating site. it has all of the disadvantages of being totally anarchic. and all of the advantages of nice people helping other nice people. figure out the solution to that and you make it heavenly. thanks bpolk|
|Thanks for your help. Wikipedia is very hard for beginners and your help was vital for me, So THANKS guys!|
|This survey is all about nothing...|
|Wiki is a terrific resource and I would like to think by doing minor edits when I come across for instance typos, citations I can add or extend an article in some way, I'm able to put something back in return.|
|I have found tagging images daunting. So many editors with different views. Could use an experienced editor to assist and counsel.|
|Thank you, Teahouse folks!|
|My responses should be considered null because I don't really have any valid experience with Teahouse nor really know it's purpose Things are very complex - overly so - on Wikipedia.|
|Although I was ultimately satisfied with the Teahouse, I initially thought it was a mistake to try it. The first contact from an Editor was confrontational and referred to editing techniques not known to me as a beginner. Editors should remember that, most likely, a person's asking a question is less knowledgeable of Wiki's functionality (ie editing) and rules.|
|Nthep answered my question, and my question got answered perfectly!|
|I think a little edit I did needs attention but I couldn't get anyone's attention so I gave up.|
|where do we show the proof that the page we created is True to keep the page alive|
|I would like to share about our articles. There should be a Wikipedia project that is creative and expresses Wikipedians.|
|See above. (I despair of getting anything helpful. I expect to be told of a response somewhere locked in your world, but that won't help me.)|
|Nothing, except maybe you should change the part "encyclopedia anyone can edit", because all sorts of idiots are misinforming us with the power of words.|
|just grateful you guys are there so that I know I have somewhere to go in times of need and that the response is fast and friendly. Thank you.|
|I would like to share about my editing. I have only edited twice because of all your rules, but I've figured out EVERYTHING thanks to Trypofish. He was very helpful to me and my edits. If you wish to contact me, I'm RAIDENRULES123|
|Thanks for organizing the Teahouse - it's been a great help to me!|
|I'm just glad to see there are people willing to help, especially if you can't pay someone to help you. The article I created had to exist and was overdue and without help Wikipedia would be worse off for not having it, same with any other article contribution. Just very happy to see people are there to help people like me who are inexperienced so they too can contribute.|
|Well, nothing. Thanks!|
|just grateful you guys are there so that I know I have somewhere to go in times of need and that the response is fast and friendly. Thank you.||
» Female newcomers participate in Teahouse at a higher rate
|See the female participation section of the pilot metrics report for earlier findings.|
30% of respondents to our most recent survey identified themselves as women. This figure is in line with the results from our two previous surveys, in which 22% (13/59) and 35% (17/48) respondents identified as women. The overall rate of female participation, as measured by these survey responses, was 29%.
|If you're comfortable telling us, what is your gender?|
|I am female||27||30%|
|I am male||51||57%|
|I'd rather not say||11||12%|
|Cumulative survey results: Female participation|
|Survey||Date||Female Respondents||Total Responents||% Female|
This is a noticeably higher rate than figures for female participation in Wikipedia as a whole, which ranges between 8% and 13%. This finding strongly suggests that the Teahouse is making progress toward one of the project's original goals of creating an appealing space for new female editors to get support, by offering a new approach to help on Wikipedia.
These survey numbers are likely to accurately reflect the true proportion of female Teahouse participants. New editors who received one survey were not asked to participate in subsequent surveys, whether or not they had chosen to fill out the first. And although it is possible that during the pilot period some female editors were invited to participate by Teahouse hosts specifically because they self-identified as women on their Wikipedia, the majority of respondents to the third survey were new editors who had been invited automatically by HostBot, with no consideration for their gender.
» Most guests either ask a question or create a profile
We were surprised to find that most new editors who visit the Teahouse only participate in one activity. While 70% of new editors ask a question and 40% create a profile, only 10% do both.
|New editor participation by page|
|% of new editors who participated in Teahouse/Questions||70%|
|% of new editors who created a Teahouse/Guests profile||40%|
|% of new editors who participated in both /Questions and /Guests||10%|
Since most guests visit the Teahouse for particular reasons, suggesting that one strategy for increasing participation beyond the current level would be to offer additional calls to action or new ways to participate. Some possibilities might be new calls to action around suggesting editing tasks for newcomers, creating new discussion forums not focused on Q&A, and better surfacing of WikiProjects that would welcome new members.
» More hosts are participating
The total number of hosts participating in the Teahouse (by answering questions on the Q&A board, building pages, and participating in talk page discussions) has increased by 19% over the pilot period weekly average, from 21 hosts per week to 25.
|total hosts participating||77|
|average hosts participating per week||21.4|
|during phase 2||24.7|
|See the Q&A section of the pilot metrics report for previous findings and survey results.|
The Teahouse passed the 1,000 questions milestone during Phase 2, and the Q&A board continues to be the most active area on the Teahouse. Most questions are asked by new editors, and most are answered by Teahouse hosts, although many experienced Wikipedians also participate in Q&A by answering questions as 'unofficial hosts'. Questions span a wide variety of topics: and while some are relatively simple and easily 'resolved', many spark detailed, animated discussions.
|total questions asked||1381|
|average questions per guest||1.6|
|% of questioners who asked more than one question||23%|
|% of questioners who respond in their own question thread||48.6%|
|average responses per question||2.9|
|median time to first response||33:20 minutes|
|questions per day||6|
|since automated invites||6.8|
|questions per week||42.2|
|since automated invites||47.7|
The Q&A board seems to be a very stimulating and interactive place for new editors. 80% of respondents stated that they were 'satisfied' or 'very satisfied' with the quality of the answers on the Q&A board. 88% of editors stated that they received a follow up message from a Teahouse host after they asked their question (generally a Teahouse talkback template, and/or a personal message). However, most guests are still hesitant to answer questions.
» Question volume has increased
The Q&A board was experiencing a lull in activity at the start of phase 2, but activity increase noticeably after we implemented automated invites on July 23rd and have remained high ever since. Current Q&A activity (48 questions/week) is 20% higher than during the pilot period (40/week) and 45% higher than during the transition period (33/week) between the end of the pilot and the deployment of automated invites.
» Many questions receive a response in 30 minutes or less
Teahouse hosts answer guests' questions rapidly. The median time between when a question is asked and its first response is 33 minutes and 20 seconds. Average response time has been shrinking week by week, possibly as a result of increasing host participation.
» Each questions receives around 3 answers
Teahouse guests can expect several answers to their question. Each question on the Q&A board receives an average of 2.93 replies from one or more other editors. However, the number of answers per question has decreased since the end of the pilot period.
» New editors were very satisfied with the answers they received
|Did you ask a question on the Teahouse Q&A board?|
|How satisfied were you with the quality of the answer you received?|
When asked to describe what in particular they they liked about the answers they received, new editors cited a range of factors, from the promptness and number of answers they received to the friendly tone and the level of detail.
|Was there anything in particular you liked about the answer your received? Click 'Show' to view all responses|
|Charles was great - other responses were less informative. Specific answers were the best.|
|Gentle tone to a new user|
|The two who answered my questions were clearly into helping people learn the ropes on Wikipedia|
|It was moderately informative|
|A warmest and friendly answering the question|
|They just seemed like they wanted me (a rank beginner) to succeed.|
|I have asked several questions and the answers ranged from mediocre to excellent. I especially appreciated learning that mostly all editors are equal|
|The person who answered my question was really helpful. They gave me some specific pointers, but more than that they made me feel welcome and like I had someone to turn to for help.|
|It partially fixed the problem|
|well, it still took me a few hours/days/pulling the last remnants of my hair out but i eventually persevered to figure out how to do what i needed. the results i got totally were determined by the patience and insight of the individual i encountered. sarah stierch was wonderful. but she was one part of a long chain and, had she not been the one to give me a hint or an answer, I would have been stymied. answers on wiki seem to totally depend on the nurturing nature of the individual who responds. that's somewhat scary.|
|My submission was declined twice, I then updated it again and submitted it but didn't hear anything for days. I then posted a message on tealeaf and had an almost immediate reply letting me know that he/she will have a look at my page and also told me that it was looking good for submission. The page was then allowed to go through.|
|There were two answers and one proposed to help me set up the fact box, which was a great help|
|I really appreciated the speed of the responses and their advice helped ease some of my concerns.|
|how do you put pics and videos on here.|
|It was comprehensive and addressed both the abstract, general answer to my question and the more personal, specific-to-me answer.|
|Exactly what I was looking for.|
|clear and consice|
|When the editor understood I was a beginner, she was helpful with specifics|
|Concise but very specific and helpful!|
|They explained it very well and were specific. Specific is the key to me.|
|Extremely detailed. Plainly a lot of effort had been put into answering my question.|
|nothing like that. but it made me easy to get|
|I have visited twice with a total of 3 different questions. All answers were informative, easy to understand, and thoroughly explained. I particularly appreciated that one host's answer built upon the previous one's - giving me a complete picture. I'd like to mention the host dalahäst who went above and beyond in giving much additional quidence on his talk page when I asked him questions there.|
|Message was very friendly and positive.|
|as I'm a novice any help is appreciated. I found the response fairly simple to understand and resolved the issue I had.|
|Sarah's input help tremendously in making a direct translation of a person's military career into a life of a real person. Interesting indeed. Am motivated to do more of this type of writing. Hope Sarah will continue to provide feedback for continuous improvement. Much, much obliged and mucho gracias indeed. CHHistory|
|I like how descriptive and detailed the editors were, and one even went the extra mile and advised me about something on my user page.|
|It was accurate and onpoint|
|I liked that it was answered, but, it took months to answer, and I am unclear of the answer and where it is,|
|I initially asked if I could pay someone to help me create the page but was told that is a no-no, so I took whatever help I could get and that help ended up being very good and came from a few different people which was great.|
|to the point with helpful links and encouraging|
|Quick, multiple replies gave me a lot of needed information.|
|After my blunder as a Newbie I got slammed by some other editors. I was kindly taken in by Heather at the Teahouse|
|They just seemed like they wanted me (a rank beginner) to succeed.||
|Was there anything in particular you disliked about the answer your received? Click 'Show' to view all responses|
|I have found becoming a new editor pretty difficult. Worse yet, i find when asking questions people tend to not be too helpful simply linking to generic pages. I read the pages, thats why i was requesting more details :)|
|The person who responded to me became condescending and aggressive.|
|Yes, I am completely lost at trying to be involved with Wikipedia. I admit that I am not a young geek, but I am reasonably sophisticated (Ph.D. in math from Duke, J.D. from Yale, Fortran programmer). But I find everything about Wikipedia completely inaccessible.|
|I didn't like the way the person explained. I have heard from better Wikipedians. They needed to explain properly.|
|I didn't like the way the person explained. I have heard from better Wikipedians. They needed to explain properly.||
» New editors are still hesitant to answer questions
Our findings here are similar to those from the pilot metrics report. While 74% of new editors surveyed indicated that they had asked a question on the Q&A board, only 7% said they had answered one. If the Q&A board is meant to be a peer-support space, this is still room for improvement in this area.
|Why haven't you answered any questions on the Q&A board?|
|I couldn't figure out how to answer a question||6||7%|
|I didn't know I was allowed to answer questions||23||28%|
|I didn't see any questions I knew the answer to||20||24%|
|I didn't feel like it||5||6%|
|some other reason (please describe)||29||35%|
When asked to describe in their own words why they had not answered a question on the Q&A board, many guests said that they did not feel they had the expertise to answer a question, or that they were concerned about making mistakes or giving incorrect answers.
|Why haven't you answered any questions on the Q&A board? Click 'Show' to view all responses|
|I'm very new and I'm just going to be quiet and learn.|
|I've programmed for 25 years but wiki'd for only 25 hours|
|I'm a total Wikipedia "newbie" .|
|I am less active on Tea House. In last days I was busy in my first two wikipedia articles i.e. Lottery short story by premchand and Gajar ka Halwa. That's the reason. Thank you.|
|I have been mostly very busy trying to update the few pages that I have become involved in. I do not feel at all qualified yet to answer a question so mostly I don't even read them|
|I'm confused about what is the Q&A board and the Teahouse|
|i was fighting my own battles as a newbie|
|I have been busy trying to set up my company's own page and do not feel "expert" enough to help others|
|Didn't know it was there.|
|I didn't know I was allowed to answer questions as given above but I also don't feel I'm sufficiently experienced with the intricacies of Wikipedia!|
|I am new to Wikipedia|
|I had such a bad experience, I don't think that this is the best thing to do.|
|I haven't looked at other questions and doubt that I would be of any help to anyone since I am new at editing Wikipedia.|
|Have not really gone there|
|I'm new and doing more learning than teaching|
|I do not have such answers or knowledge|
|I'm not yet competent enough to help someone else|
|I have no interest in doing so. Editing articles that are in my interest is enough for me right now. Plus, I don't feel I am qualified to answer any questions yet.|
|Some day I hope to! But I feel I'm too new right now.|
|wikipedia is not an easy site to understand, let alone quickly and easy do stuff (create pages, reply to questions) unless you have done some serious learning (and that learn curve is steep)|
|I was going to visit it, but i hadn't enough time.|
|Am to new to this. Once I have completed an article that is good enough for posting live, than I will help others the best I know how.|
|I don't really like to answer questions that other people ask because I might get it wrong, and most Wikipedians are very rude.|
|There are others who are much more qualified to do that than I am.|
|I am still a noob myself and senior WP authors will know better answers|
|I haven't had a chance to answer the questions (not logged in for a while)|
|I don't feel like I know enough about Wikipedia and all the formatting|
|I didn't know I was allowed to answer questions as given above but I also don't feel I'm sufficiently experienced with the intricacies of Wikipedia!||
|What made you decide to answer a question? Click 'Show' to view all responses|
|I was just lurking but I saw a question that was obvious and that I knew with strong certainty|
|The popup (after writing a profile) told me to check the questions page, and I saw a few questions that I thought I could answer.|
|A statement was made with which I - very mildly - disagreed.|
|Think I said THANKS!|
|I was just lurking but I saw a question that was obvious and that I knew with strong certainty||
|See the Guests page section of the pilot metrics report for previous findings and survey results.|
» New editors enjoyed browsing other guests' profiles and creating their own
Around 40% of survey respondents stated that they had created a profile. Guests' levels of satisfaction with the profile creation process was comparable to the level of satisfaction reported during the pilot period, with 68% of respondents saying they were 'satisfied' or 'very satisfied'.
|Did you introduce yourself on the Guests page?|
|How satisfied were you with the experience of creating your guest page introduction?|
Guests highlighted the ease of creating a profile and the ability to browse other profiles as particularly beneficial. No guests identified any aspects of Guest profiles that they especially disliked. The majority (55%) of guests who did not create a profile said that they didn't know it was there. Others said that they could not figure out how to create a profile and cited lack of familiarity with markup. One guest said they weren't comfortable personal information on the internet, and another said "I am only interesting in posting information that should/could be of interest to others. my mission here had nothing to do with me."
|Was there anything in particular you liked about the experience of creating an introduction? Click 'Show' to view all responses|
|To be quite honest, my initial immersion into Wikipedia was so mind boggling, I can't remember if I really did an intro or not.|
|I get to put myself out there as a part of a thing far greater than myself, a thing that all of us share, because it's not just people's Wikipedia, but it's also mine.|
|Picking my own image from commons|
|Remembering my favorite quote by Mark Twain|
|Not really anything in particular.|
|This question has made more aware of the possible value of the introduction. I never really occured to me that someone would actually take the time to read it.|
|read the intros, found new people with interesting stories to tell on how they started with wiki|
|Yes, now people know about me.|
|I should go to the Teahouse pages because I'm a bit confused about this and the past 2 questions. I'm unsure whether or not I did a "guest introduction" - I simply posted my questions under a new section.|
|I mainly liked telling about myself.|
|I liked how you can explain about yourself and discover about others. This is what I liked in particular.|
|I just like typing, because it is like writing (my best school subject), only without the hand-aches (not headaches).|
|I like how we got to choose a picture from Wikimedia Commons.|
|Yes! I loved it.|
|I liked the Syntax method of writing and uploading pictures.|
|I get to put myself out there as a part of a thing far greater than myself, a thing that all of us share, because it's not just people's Wikipedia, but it's also mine||
|Why didn't you introduce yourself?|
|didn't feel like it||5||10%|
|couldn't figure out how||6||12%|
|didn't know it was there||28||55%|
|didn't want to be visible to others||2||4%|
|didn't want to be associated with the Teahouse||0||0%|
|some other reason (please describe)||10||20%|
|Why didn't you introduce yourself? Click 'Show' to view all responses|
|Not that I can remember but you may be able to clarify|
|i guess (and this, i suppose, could be taken as a criticism - but isn't meant to be) that i was and am only interesting in posting information that should/could be of interest to others. my mission here had nothing to do with me.|
|As I'm unsure of the etiquette within Wiki, which I feel is the reason I ended up embroiled in the situation I was asking advice about, I'm fairly weary of drawing undue attention; does that sound right or just plain silly?|
|I had no idea that I was supposed to do thtat.|
|I will in the near future.. when i get the time|
|Didn't know I was a guest.|
|a) couldnt find it, b) couldnt figure out how, c) didnt know it was there - again i state, wikipedia is not an easy system to learn/understand|
|too new, let me gain some experience first.|
|do not like to give personal information on the internet|
|As stated earlier feel I don't know enough about formatting etc.|
|As I'm unsure of the etiquette within Wiki, which I feel is the reason I ended up embroiled in the situation I was asking advice about, I'm fairly weary of drawing undue attention; does that sound right or just plain silly?||
» More new editors are creating profiles
Although profile creation declined over the course of the pilot period, it increased substantially over the course of Phase 2. Between July 23rd and October 11, an average of 19.3 new profiles were created per week, compared with 14.4 new profiles per week during the pilot period, a 34% increase.
|total profiles created||482|
|profiles per day||2|
|since automated invites||2.76|
|profiles per week||14.2|
|since automated invites||19.3|
We highlight two factors in particular which likely contributed to this increase:
- The implementation of an 'archiving' script that limited the number of profiles on the main Guests page to 10-20 of the most recent profiles. Profiles of previous guests were archived to the Teahouse 'guestbook'. This change was intended to increase the perceived value of creating a profile by making new profiles more visible.
- The implementation of automated invites allowed us to reach out to more new editors each day. These new editors visited the Teahouse at the same rate as those guests who were invited manually during the pilot period. However, where many manual invites were sent to guests who a host had judged as needing help, automated invites were sent to guests regardless of whether or not they currently needed assistance. We believe that since fewer of these guests had a questions in mind when they visited, many more of them created profiles instead.
New editors find out about the Teahouse in a variety of ways. During the pilot period, manual invitations were the primary mechanism. During Phase 2, several other mechanisms were developed.
- Manual Invitations
During the pilot period, all invitations to visit the Teahouse were sent manually by Teahouse hosts, with two hosts (Rosiestep and SarahStierch) sending 80% of all tracked invitations.
While this process was effective at bringing new editors to the Teahouse, the response rate was quite low: only around 4.5% of those who received an invite subsequently visited Teahouse. Anecdotally, hosts who sent many invites expressed frustration at this low rate of return, considering how labor intensive the process of manually inviting editors was. Many editors continued to manually invite new editors to the Teahouse during Phase 2.
- External links
Because invitations are crucial for driving traffic to the Teahouse, but manual invitations alone was not a sustainable strategy, we decided to experiment with two other mechanisms for making new editors aware of the Teahouse: automated invitations and a custom Article Feedback call to action.
The Wikipedia Help Desk also added a link to the Teahouse, and a special message to new editors, to their page header. The community has also added Teahouse links to many other help spaces, such as:
We have not explicitly tracked inbound traffic or the response rate from these sources.
- Automated invites
|See the invite experiment page on Meta for more information about automated invites.|
We developed a Python script that automatically invited new editors who met our basic invitee criteria.
On July 23rd, HostBot was approved by the Bot Approvals Group to begin sending automated invitations to all new editors listed on the invitee reports. While the goal of the Teahouse is to be as inclusive as possible, after conversation with community members we decided to program HostBot to not invite users who had received level 4 warnings, or who had been accused of sockpuppetry.
In evaluating the long-term effectiveness of automated invites, we were particularly interested in answering the following questions:
- Are new editors more or less likely to respond to an invitation sent by a bot than an invite sent by human?
- Are new editors more likely to respond to a personalized invitation or a generic one?
- Article Feedback v5 call to action
We worked with the Wikimedia Foundation's Editor Engagement team to implement a Teahouse Call to Action (CtA) for the article feedback tool, which would be displayed to 10% of registered editors who submitted article feedback. The Teahouse CtA was rolled out on August 21st, and has (as of October 14th) been displayed to 881 editors who filled out Article Feedback.
» Direct invitations continue to drive traffic to Teahouse
During the pilot period, we found that the majority of Teahouse guests were invited guests. This pattern continued in Phase 2, with 44% of survey respondents stating that they were invited to the Teahouse via a talk page invitation.
|How did you find out about Teahouse?|
|Someone invited me on my own talk page.||41||44%|
|Someone emailed me an invitation.||6||6%|
|Another editor referred me to Teahouse.||17||18%|
|I am a member of a class, and my instructor told me about Teahouse||0||0%|
|I read about Teahouse on Wikipedia.||22||23%|
|Other (please describe below)||8||9%|
|How did you find out about the Teahouse? Click 'Show' to view all responses|
|I don't know. I think I arrived there by hit and miss|
|A bot invited me on my own talk page.|
|I was looking for help/advice and reached it through the help pages.|
|I found it|
|asked to join|
|i asked a question and Teahouse forum was among the solutions|
|I was looking to discover all of the features of Wikepedia and found it.|
|I was looking for help/advice and reached it through the help pages.||
» Automated invites yield the same response rate as manual invites 
HostBot sent out a total of 4268 invites between July 23rd and October 11th, and 165 (3.9%) of invitees subsequently participated in the Teahouse (asked or answered a question and/or created a profile). This response rate is not statistically different from the response rate (4.27%) for manual invitations sent to invitee report candidates during the pilot period, as determined by a 2-tailed t-test (t=0.68, p > .05, 2-tailed).
Despite some initial concerns that removing editorial discretion from the invitation process might lead to an increase in vandalism or disruptive editing on the Teahouse, we have not observed any such increase in the time that automated invites have been running.
Given the findings above, and the facts that automated invitations are much less labor-intensive than manual invites and have contributed to an increase in new editor participation, we view automated invitations as a key component of the Teahouse's continued success.
» Personalized messages did not yield significantly different results from generic invitations
Previous research has determined that adding a degree of personalization to templated talk page messages (for instance, by using active voice and calling out the messager as a fellow editor) can yield better responses in some scenarios than generic messages. We tested the effectiveness of personalization in automated invites by including a personal signature from a Teahouse host in roughly 50% of messages, while leaving the other 50% generic.
We found that adding personalization to an automated invitation does not affect response rate. A sample of 2435 automated invites yielded response rates of 3.8% for both personalized and generic invitations. Findings and a full discussion of these results can be read on the invites experiment page.
Given this result, we decided to make all invitations personalized. A personalized invitation gives the invitee a personal point of contact that they can message directly if they choose to, and may help new users feel like they are a part of a community. We currently have no plans to continue these experiments.
» Few AFT respondents subsequently visited the Teahouse
Out of the 881 registered editors who received the Teahouse call to action after submitting their article feedback, 7 (0.08%) have visited the Teahouse so far.
The main Teahouse page was viewed 39,678 times between February 27th and October 13th, for an average of 173 (non-unique) pageviews per day. The graph at right shows page views per day during that period.
» Page traffic has increased over time
The graph above shows that page traffic has increased in recent months. There were an average of 155 pageviews per day during the pilot period, versus 204 views per day since the introduction of automated invites. The dramatic spike in pageviews around July 15th corresponds to Wikimania 2012, during which the Teahouse was discussed in several presentations and panels.
|total page views||39,678|
|page views per day||155|
|since automated invites||204|
» Most page requests come from the US and UK
An analysis of a 10% sample of all pageviews for WP:Teahouse and its subpages shows visitors from 101 different countries. Over 65% of all page requests come from within the United States, followed by the United Kingdom with 10% of all page requests. Not surprisingly, this distribution of page requests by country closely parallels that of en.wikipedia.org.
|WP:Teahouse/* pageviews by country (click 'show' to view)|
|United Arab Emirates||28|
|Iran Islamic Republic of||20|
|Korea Republic of||13|
|Syrian Arab Republic||5|
|Isle of Man||4|
|Moldova Republic of||1|
New editor retention
Sample: 149 Teahouse visitors and 1954 non-vistors invited between 2/27 and 7/25. Edits to Teahouse pages not included in counts. Only non-visitors who made at least 1 edit to Wikipedia after the date of their invite are included.
In order to determine whether participating in the Teahouse has a positive effect on an editor's likelihood of continuing to edit Wikipedia and on their level of editing activity, we analyzed the editing behaviors of a sample of editors who were invited to the Teahouse by a host between February 27th and July 25th. Editors in this sample were invited (manually) to the Teahouse because their names appeared in the daily Invitee report or their submission was rejected by Articles for Creation.
To control for the possibility that new editors who did not visit had simply not seen the invitation, we excluded from our sample those who did not make at least one edit to Wikipedia after the date of invitation. This left us with a sample of 1,948 editors, 149 of these editors visited the Teahouse. For the editors who visited Teahouse, any edits they made to WP:Teahouse (and its sub-pages) were ignored in this analysis. We also ignored any edits made by editors in our sample prior to their invitation.
Our main findings from this analysis are highlighted below. Because our edit data are not normally distributed (in both groups, a small number of editors made hundreds or thousands of edits while most made from a few to a few dozen) we present both mean and median averages, where appropriate.
- Editing activity by Teahouse visitors vs. non-visitors
|mean post-invite edits||241||75|
|median post-invite edits||29||12|
|mean weeks with edits||5.75||3.65|
|median weeks with edits||4||2|
|mean articles edited||76.5||21.2|
|median articles edited||5||2|
|% with article edits||74%||81%|
|% edits to talk namespaces||35%||20%|
These findings show that Teahouse visitors are more active than new editors who do not visit Teahouse, and that they exhibit certain positive behavioral characteristics (such as editing more than one article, or participating in talk page discussions) to a greater degree than editors in the control group. While the findings presented below do not prove definitively that participating in the Teahouse causes editors to exhibit these behaviors, that is a plausible interpretation, given our findings on editor activity and satisfaction, presented above. For instance, it is easy to imagine how getting a helpful answer from a friendly Wikipedian at the Teahouse could make a new editor feel more comfortable and confident participating in talk page discussions. However, even taking the position that the Teahouse tends to attract editors who are most likely to become active, productive contributors anyway, we believe that the Teahouse provides these editors with better and earlier assistance and more positive community interaction than they would likely have received otherwise.
|They gave me some specific pointers, but more than that they made me feel welcome and like I had someone to turn to for help.||
» Teahouse guests make more edits to more articles
On average, Teahouse guests made more edits (241 versus 75), and edited more articles (76.5 versus 21.2). This is encouraging since many new editors start out with the intent to create or contribute to a single article. Encouraging new editors to branch out and edit more articles may lead to higher long-term retention.
» Teahouse guests edit more each week, for more weeks
The Teahouse visitors in our sample edited roughly twice as long on average as non-visitors, measured by the number of weeks after being invited in which they made at least 1 edit to Wikipedia. They also edited more on average during weeks when they did edit. We analyzed the subsequent editing activity of 1,655 editors (133 invited visitors, 1,522 invited non-visitors) from the pilot period (February 27th - May 27th).
Teahouse visitors made 7 edits per week on average, while invited non-visitors made 2.1 Teahouse visitors also made more mean edits in 21/21 (100%) of all weeks, and more median edits in 17/21 (81%) of weeks.* The first chart below (left) shows mean and median edits-per-week for the two groups during the subsequent 21 weeks (May 28th - October 18th).
The second chart below (right) shows the average increase in cumulative edits over time for these two groups. Teahouse visitors invited during the pilot period had over twice as many edits on October 21st as they had on May 28th (an 115% increase), while the non-visitor group increased their total edit counts by only 77% over that 5 month period.
» Teahouse guests communicate more with other editors
Teahouse guests also made roughly 75% more edits to the primary talk namespaces (User_talk, Article_talk and Wikipedia_talk) than other new editors. This suggests that Teahouse guests participate more in talk page discussions and may be more involved in community processes than new editors who chose not to not participate.