Research:Teahouse/Metrics/Archive

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
Historical This research page is kept for historical interest. Any policies mentioned may be obsolete.
Please visit the research homepage for current research-related contents and discussions.
You can review the most recent Teahouse metrics report at Research:Teahouse/Metrics.

Category:Archives/Research

Wikimedia data.jpg

August Metrics[edit]

Cumulative activity metrics for the period February 27th through August 26th are presented below. The metrics report from the pilot period is available here.

Highlights[edit]

  • 741 new editors have participated in Teahouse since February 27th
  • 50% more new editors are participating every week since automated invites began. Q&A board participation is up by 27%, and profile-creation by more than 50%
  • 45% more hosts are participating each week
  • The Q&A board is as lively and interactive as ever. Each question receives around 3 answers, and are answered within about 30 minutes. Over 50% of guests participate in the discussion around the question they asked.

Participation :: overview[edit]

The Teahouse is busier than it's ever been. Although participation by new editors dropped off somewhat in the ~2 months following the end of the pilot period on May 26th, participation rebounded in mid-late July and has since increased substantially. Since July 23rd, an average of 46 new editors have participated per week, an increase of 53% over the pilot period. Two likely factors contributing to this increase are:

New editors participating per week, 2/27 - 8/26/2012
Overall participation
total editors (on any WP:Teahouse/* page) 1250
total hosts 46
per week 19
joined & participated since pilot 12
total new editors (< 100 edits at first visit) 741
per week
during pilot 30
since 7/23 46
percent increase 53%
overall 32.6

Participation :: Q&A board[edit]

New editor participation in the Q&A board has risen sharply since the implementation of automatic invites. Since July 23rd, the average number of questions asked has increased 29% over the pilot period. The difference is even more stark in comparison to the three weeks immediately prior to automatic invitation (7.3 questions per day versus 4.8).

The Q&A board is highly interactive. 53% of questioners post a response (usually a follow up question, or a simple 'thank you') to the thread they started, and 22% of guests have asked more than one question. Despite the recent increase in questions per day, Teahouse host have maintained a high quality of service. Questions are still usually answered in about 30 minutes, and each question receives an average of 3 responses (not including any responses by the original questioner). The number of newcomers who respond to someone else's question remains low, at about 11%.

Questions per week, 2/27 - 8/26/2012
Teahouse/Questions
total questions 1081
average time to first response 1:06 hours
median time to first response 32 minutes
 % of guests who respond to their own question 53%
questions per guest 1.6
responses per question 3
responses by new editors 10
per day
during pilot 5.6
since 7/23 7.3
overall 6
per week
during pilot 39.5
since 7/23 51
percent increase 29%
overall 41.5

Participation :: Guests page[edit]

Guest profile creation has increased markedly since 7/23, up 53% per week from the pilot period. The implementation of automatic invites seems to have contributed to the number of new editors who introduce themselves at the Teahouse, even if they don't have an immediate question to ask.

Profiles by new editors per week, 2/27 - 8/26/2012
Teahouse/Guests
total new editor profiles 274
per week
during pilot 10.8
since 7/23 16.4
percent increase 53%
overall 10.1

Retention :: Visitors vs. non-visitors[edit]

In order to determine whether participating in the Teahouse has a positive effect on an editor's likelihood of continuing to edit Wikipedia and on their level of editing activity, we analyzed the editing behaviors of a sample of editors who were invited to the Teahouse during the pilot period, identified through either the Invitee report or the Articles for Creation process. Roughly half of these editors subsequently visited the Teahouse, the other half did not. To control for the possibility that new editors who did not visit had simply not seen the invitation, we excluded from our sample those who did not make at least one edit to Wikipedia after the date of invitation.*

Our main findings from this analysis are highlighted below.

  • Teahouse guests made more edits on average. (216.7 versus 74.4) than editors who were invited to the Teahouse, but did not visit.
  • Teahouse guests also subsequently edited more articles than invited non-visitors, even after controlling for whether the non-visitors had made at least 1 edit the day after receiving the invitation.
  • Teahouse guests made more edits to talk namespaces. A higher proportion of guest edits were to talk namespaces (user_talk, Wikipedia_talk, article_talk): 35% on average for Teahouse visitors, versus 16% for non-visitors.
Group sample size total edits articles edited % talk namespace edits
Visitors 153 216.7 66 35%
Invited non-visitors 164 74.4 25 16%

* This sample differs somewhat from the sample used in our previous analysis. Specifically, we have removed the 'Control A' group from this analysis, due to the difficulty of making meaningful comparison between editors who were intentionally selected by Teahouse hosts as good candidates for invitation (invitees) and a random sample of highly active new editors. Our previous findings showed that both invitees and invited visitors showed greater retention and higher productivity than this group, so we are not concerned that excluding this sample invalidates our results.


July Metrics[edit]

Basic activity metrics for the month of July 2012 are presented below.

Highlights[edit]

  • questions per week has increased slightly, up to 43 per week from 36 in June
  • 12 new hosts joined the Teahouse in July, up from 4 new hosts in June
  • both questions and profiles increase noticeably after 7/23. Possible result of automated invites
  • most other activity measures are roughly equivalent to the numbers for June

Q&A[edit]

Overall participation in the Q&A board is in line with previous months, resulting in an average of 5.8 questions per day. However, since the implementation of automatic invites on July 24th, the average number of questions per day has increased to 8.4, compared with an average of 4.8 for the period July 1 - 23.

WP:Teahouse/Questions # %
total participants 215
total hosts 23 10.70%
total guests 192 89.30%
guests - new editors (< 100 edits) 80 41.67%
guests - veteran editors (> 100 edits) 82 42.71%
guests - IP addresses 30 15.63%
hosts participating per week 13
guests participating per week 48
total questions 181
questions from new editors 77 42.54%
questions from veteran editors 62 34.25%
questions from IP addresses 30 16.57%
mean time to first response 1:02 hours
median response time 28 minutes
mean questions per guest 1.4
mean responses per question 2.6
responses by new editors 10
questions per week 43
questions per day 5.8

Guests page[edit]

Guest profile creation was slow for the first three weeks of July. However, the implementation of automatic invites seems to have contributed to a jump in the number of guests who create profiles. Since July 24th, when HostBot began sending out automated invitations to between 50 and 70 every day, 15 new editors have created profiles on the Guests page, a 300% rate increase from the weeks prior.

WP:Teahouse/Guests #
profiles per week 7
total new profiles created 30

April 24th metrics report[edit]

Activity Metrics[edit]

This section contains data related to activity on WP:Teahouse/Questions and WP:Teahouse/Guests, as well as activities related to inviting new editors to Teahouse. All data reported in this section was gathered between 2/27 and 4/24/2012.

Visitors[edit]

number of new editors visiting Teahouse/Questions for the first time per week, through 4/22/2012
number of new editors visiting Teahouse/Guests for the first time per week, through 4/22/2012

Teahouse receives most of its visitors through direct invitation. Visitors who have not received a direct invitation seem to be more likely to ask a question on Teahouse/Questions than to create an intro box on the Teahouse/Guests page. For this analysis, a "new editor visitor" is defined as a user who either created their account in 2012 or who had made fewer than 100 edits under that account at the time they edited Teahouse/Questions or Teahouse/Guests.

Questions[edit]

questions per week on Teahouse/Questions through 4/22/2012
the types of questions asked on WP:Teahouse/Questions and WP:Help_desk during February and March 2012. Coding scheme adapted from here

Question summary

Overall, the Teahouse has received an average of about 6 questions per day since February 27th, although this number varies significantly day-by-day and week-by-week, and has increased notably in recent weeks. It is common, but not the norm, for Teahouse visitors to ask more than one question. Questions are answered rapidly, with the average response time of about 1.5 hours and a median response time of half an hour.

* excludes questions asked by Teahouse hosts, WMF staff members and 1 disruptive editor

Question types

A content analysis of 100 questions each from Teahouse/Questions and Help Desk was conducted by a member of the research team. This analysis, which compared the types of questions asked in these two help forums over the same time period, suggests several intriguing differences between the help needs of new editors and the help needs of the general Wikipedia population:

  • Teahouse guests asked more questions about rules of editing (such as questions about reliability of sources, notability and conflict of interest)
  • Teahouse guests asked more (both technical and non-technical) questions about adding sources in general
  • Teahouse guests asked fewer technical questions (such as how to create a template, or move a page) in general
  • Teahouse guests asked more questions related to finding work to do, such as identifying articles in need of editing or WikiProjects looking for new members

Q&A page activity by host[edit]

This table contains the number of edits made to Teahouse/Questions by editors who have identified themselves as Teahouse hosts. Number of edits is a proxy for number of answers to questions. However, these numbers will be slightly skewed for editors who have also made changes to the layout of the Q&A page. For instance, the number of apparent "answers" by users Jtmorgan, Heatherawalls and Writ Keeper may be slightly inflated for this reason.

Guest introductions[edit]

Invitations to new editors[edit]

Invitees by source


Templated invites sent by host


Impact of email invites

We are interested in determining whether new editors who receive an email invite in addition to an invite template are more likely to ask a question or create an intro box at the Teahouse. In order to test this, we collected a sample of editors who were invited through the Teahouse invitee database reports, the new editor contrib filter or the feedback dashboard. We find that editors who received an email invite are roughly twice as likely to show up at Teahouse than editors who only received an invite template on their talk page.

Engagement and retention metrics[edit]

In order to begin to understand whether visiting the Teahouse has a positive impact on subsequent editing behaviors or editor retention, we analyzed the outcomes for three groups of editors in three different 'conditions'--two different control groups and an experimental group--drawn from editors who joined between mid-February and mid-April 2012.

  • Control A: 75 new editors who met the requirements to be included on the Teahouse invitee report but whose usernames were not included on that report, who were not invited through any other forum, and who did not find their way to the Teahouse on their own.
  • Control B: 74 new editors who were invited through the feedback dashboard, the Teahouse invitee reports and the new editor contribs filter, but did not ask a question on the Teahouse Q&A page or create an introduction box on the Teahouse/Guests page.
  • Experimental: 75 new editors who were invited through the feedback dashboard, the Teahouse invitee reports and the new editor contribs filter and who did edit the Teahouse/Questions and/or Teahouse/Guests pages.

Group outcomes[edit]

Amount of content in the article namespace content that survived in significant bytes (no spaces) as of 4/27/2012, for editors who visited WP:Teahouse vs non-visitors.
Amount of content in the article namespace that survived in significant bytes (no spaces) as of 4/27/2012, for editors who visited WP:Teahouse vs non-visitors. Average within experimental group.

Overview

The table below displays aggregate data related to subsequent editing activity by members of the three different experimental groups. These measures were generated using a custom-built analytics platform used by the Wikimedia Foundation's Global Development department in order to assess the impact of outreach initiatives on subsequent editing behavior and editor retention. We use these measures to analyze whether Teahouse visitors engage in certain positive editing activities, such as whether they edit the article namespace, how many articles they edit, and whether their edits persist over time or are subsequently reverted or deleted.

Teahouse visitors (Experimental group) are more likely to edit the article namespace after being invited, and show a higher average number of edits to the article namespace, a larger volume of article namespace contribution, and a higher degree of edit persistence than either of the control groups.

Editors who receive a Teahouse invite template but do not subsequently visit the Teahouse (Control B) exhibit higher performance than Control A, but still do not edit as actively or productively as the Experimental group. This is likely to be an artifact of an expected "invitation bias" on the part of Teahouse hosts, which is discussed further in the next section.

A table key with detailed descriptions of each measure is provided below.


Table Key

  • Editors with >0 NS0 edits: editors in this sample who made at least one edit to the article namespace after their invite date*.
  • Avg Global Edits: the average number of global edits, per editor
  • avg bytes added (by editor): the average number of bytes added per group, post-invite
  • avg bytes survived (by editor): the average number of bytes added which had not been reverted or deleted at the time the analysis was run
  • articles edited: total number of different article namespace pages edited (or created) by group members, post-invite

Level of recent activity[edit]

Overview

In order to determine whether Teahouse visitors were active at a higher rate than non-visitors, we measured how many members of each group had made at least one edit to any namespace between April 20th and April 27th. Each of our three sample groups contained editors who were invited (or, in the case of Control A, met the criteria for invitation) between February 27th and April 10th, the minimum time elapsed time between invitation and the time period in which we analyzed follow-up activity was 10 days. The invitation date distributions between the three groups was similar.

Our findings for this round of analysis are comparable to the proportions we found last time around. As expected, invitees who did not visit Teahouse (Control B) are still active at a higher rate than Control A (12% versus 5%). This is likely the result of an expected selection bias among Teahouse hosts: they (appropriately) tend to invite new editors who show promise, and avoid inviting new editors who show obvious tendencies towards vandalism or who they judge to be less engaged--in other words, Teahouse hosts may avoid inviting new editors who they believe are likely to be blocked, or whose editing behavior in their first few days otherwise indicates that they may be unlikely to stick around. This selection bias would likely result in the phenomenon seen below: the editors from Control B continue to be active at a higher rate than editors from control group A, which consists a random, unscreened sample of new editors from the same sources.

However, new editors who were invited and who subsequently visited the Teahouse (Experimental group) were much more likely to remain active weeks later than Control B (28% versus 12%), consistent with our previous analysis. Although our sample is still relatively small and more nuanced retention measures are still needed, this week's confirmatory findings (coupled with the other subsequent editing activity data presented directly above) provides additional evidence that Teahouse is having a positive impact on retention by:

  • encouraging sustained contribution from editors who would otherwise have stopped editing, and
  • providing key support infrastructure for 'high quality' newcomers who are already intent on sticking around, but who face many social and technical challenges as they integrate themselves into the community.

April 2nd metrics report[edit]

We report a variety of data related to the activity on WP:Teahouse, the experience of new editors who participated, and the impact of participation on editor retention. All data below refer to on-wiki activity that took place between 2/27/2012 and 4/2/2012. Additional analyses and updates to these existing analyses will be posted here in late April or early May, as the Teahouse pilot wraps up.


Activity Metrics[edit]

In this section we report metrics related to activity on WP:Teahouse and its sub-pages, as well as activities related to identifying and inviting new editors to Teahouse. All data reported in this section was gathered between 2/27 and 4/2/2012.

Overview

  • Overall, we show a steady level of activity on Teahouse/Questions (the q & a board) and Teahouse/Guests (the guests intro page).
  • More than half (61%) of new editors who ask a question on the q & a board return to thank the answering host, ask for followup or clarification, or ask an additional question (26%).
  • Most new editors are identified through a few major sources: the Teahouse invitee reports, the new editor contribs filter and the feedback dashboard. Of these, the invitee reports and the feedback dashboard demonstrate a slightly higher 'hit rate', but in general hit rate is fairly low, hovering between 3% and 6% for these three sources through the pilot period so far.
number of newbies visiting Teahouse/Guests for the first time, per week
total revisions to Teahouse/Questions per day
number of newbies visiting Teahouse/Questions for the first time, per week
questions per day on Teahouse/Questions
total revs per day to Teahouse/Guests


Visitor Summary[edit]

The total number of new editors who visited WP:Teahouse between 2/27 and 4/2. All invitees who visited are counted. Editors who were not invited and whose accounts were > 1 month old or who had > 100 edits at the time of their visit are not counted, which may lead to under-reporting.

Repeat Visitors[edit]

Many new editors visited Teahouse more than once. Hosts use a special Talkback template to let questioners know that their question has been answered, which increases the possibility that a questioner will return to Teahouse. Many visitors also returned to ask another question.


Invitees by source[edit]

Invitees and invited visitors to WP:Teahouse listed by where they were found. "unknown" means that an invite template was transcluded on the new editors talk page, but their name was not recorded in the invite tracking spreadsheet. "Other" indicates that the inviting host specified "Other" as the invite location in the invite tracking spreadsheet.

Invitees by host[edit]

Total invite templates transcluded since 2/27, by inviting host. "unknown" means that an invite template was transcluded on the new editors talk page, but the invite was not recorded in the invite tracking spreadsheet.

Engagement Metrics[edit]

Below we present a selection of questions and answers drawn from a survey of ~1500 Teahouse invitees and (new editor) visitors between February 27th and March 15th. This survey was active between 3/20 and 4/2/2012. We received a total of 160 responses (including partially complete surveys) and 124 completed responses.

Overview

  • Satisfaction with the Teahouse Q&A board and Guests' page were overwhelmingly positive (86%/78% 'satisfied' or 'very satisfied')
  • Overall satisfaction with Teahouse was roughly two-thirds 'neutral' and one-third 'positive', with very little 'negative' experience reported
  • respondents especially appreciated the speed with which questions were answered, and the welcoming and helpful attitude of the answering hosts
  • roughly 27% of respondents were female, noticeably higher than the average proportion for en.wiki


Q&A page satisfaction[edit]

Prompt: How satisfied were you with the quality of the answer you received at Teahouse/Questions?

Guests page satisfaction[edit]

Prompt: How satisfied were you with the experience of creating your guest page introduction?

Overall satisfaction[edit]

Prompt: How satisfied were you with your overall experience on Teahouse?

Negative experience with Teahouse[edit]

Prompt: Was there anything in particular you disliked about your Teahouse experience? Tell us about it.

Positive experience with Teahouse[edit]

Prompt: Was there anything in particular you liked about your Teahouse experience? Tell us about it.


On-wiki activities[edit]

Prompt: What are some of the things are you are excited about doing on Wikipedia?

New features[edit]

Prompt: Teahouse is still under development, and we're planning to add new features soon. Which of the features below would be most helpful to you? (select all that apply)

Participant gender[edit]

Prompt: Teahouse is intended to be a friendly, supportive environment for all new Wikipedians, but we make a special effort to welcome female editors, who are underrepresented within the Wikipedia community. If you're comfortable telling us, what is your gender?

Retention Metrics[edit]

Although Teahouse has only been live for a little over a month, we do have some preliminary data addressing the question of whether Teahouse visitors tend to continue editing Wikipedia at a higher rate than editors who did not visit Teahouse. The table below lists three groups of editors who share the following characteristics:

  • the created a user account between 2/26 and 3/17
  • they made at least 10 edits within their first 24 hours
  • they were not blocked from editing before their 10th edit

Some of these editors ("invited") were presented as 'potential invitees' on the Teahouse invitees database report, and were subsequently invited by Teahouse hosts via a talk page invite template, an email, or both. Of these, a small number ("visited" group) subsequently visited the Teahouse. A third group ("control") met the criteria for invitation, but were excluded from the invitees database report and thus Teahouse hosts were not given the opportunity to invite them.


Overview

In order to determine whether Teahouse visitors were active at a higher rate than non-visitors, we measured how many members of each group had made at least one edit between March 26th and April 2nd. As expected, "invited" visitors were active at a higher rate than the control group (12% versus 7%). This is likely the result of a selection bias among Teahouse hosts: they (appropriately) tend to invite new editors who show promise, and avoid inviting new editors who show obvious tendencies towards vandalism or who don't seem as engaged--in other words, they may avoid inviting new editors who are likely to be blocked, or who are more likely to leave, resulting in a higher proportion of subsequently active editors than the control group, which includes a wider variety of new editors.

However, new editors who were invited and who visited the Teahouse were much more likely to remain active weeks later than even their "invited" fellows (38% versus 12%). Although these numbers are small, and there may be additional selection biases at work, this is an early encouraging sign that Teahouse may be drawing editors in who would otherwise leave sooner and also providing key support infrastructure for 'high quality' newcomers who are already intent on sticking around, but who face many social and technical challenges as they integrate themselves into the community.


March 25th Report[edit]

the number of templated invitations sent out, per day, by Teahouse hosts during the first month of the WP:Teahouse pilot
the number of revisions, per day, to WP:Teahouse/Guests during the first month of the WP:Teahouse pilot
the number of revisions, per day, to WP:Teahouse/Questions during the first month of the WP:Teahouse pilot

Invitations by host[edit]

As usual, Rosiestep and SarahStierch lead the pack, invite-wise. Great work, you two!

The "unknown host" user here is intended to account for a large number of invite templates which were found through an API query, but which had not been included on the invite tracking spreadsheet.

Invitations by source[edit]

These are the places hosts are looking to find new users to invite. Since the database reports have been out of commission off and on for the last week or so(!), there have been relatively fewer newbies invited through that channel. However, database reports, new editor contribs and feedback dashboard remain the richest sources of newbies: new editors recruited from the feedback dashboard, the database reports and the new editor contribs page continue to show up at the Teahouse at the highest rates. Keep watching these sources!


As above, "unknown" in the table below accounts for a large number of invite templates which were found through an API query, but which had not been included on the invite tracking spreadsheet.

Overview of attendees[edit]

Total attendees, and another note on emailing

The following table summarizes the number of invited guests to WP:Teahouse.

As you can see from this table, emailing guests remains an effective strategy for getting them to show up. Guests who are emailed are significantly more likely to subsequently introduce themselves or ask a question than one who is not. Keep emailing!


Newbies who participated in WP:Teahouse

Because it's not easy to discern whether someone editing Teahouse pages is in fact a new user (as opposed to, say, a passing Wikipedian or an unofficial 'host'), editors were counted as 'newbies' if they they met one of the following two conditions:

  1. their account was less than 1 month old at the time they edited a Teahouse page, or
  2. they had fewer than 100 edits to their username at the time they edited a Teahouse page

Summary: A good number of Q&A participants are finding the Teahouse without being explicitly invited (or at least those invites aren't being recorded as such). However, new users appear to be much less comfortable introducing themselves on the 'guests' page when they have not been explicitly invited to Teahouse. This suggests to me that the 'welcoming' aspect of explicit invites puts guests more at ease, and may increase the perception that they are part of a community.

* This 'other' category is a catchall for a bunch of different kinds of editors, from those who are performing 'host' duties but have not included themselves on the Teahouse hosts' list,r to relative newbies who were not explicitly invited but found the Teahouse and asked ask a question, to many other Wikipedians of all levels of experience who dropped by to ask or to answer a question.

March 12th report[edit]

Basic stats - this week[edit]

40 new editors edited the Teahouse Q&A page this week. Invitees who received emails were about twice as likely to participate than those who were invited by talk page message alone.

*percent of guests who were emailed who actually showed up.


Guests invited per host

Rosiestep and SarahStierch invited the most guests!


Invitees per invite location

Most guests were invited through the New editor contrib page, followed by the Database report.

Basic Stats - since launch[edit]

Number of revisions to Teahouse/Questions per day since launch. This graph shows revisions by all users, not just new editors who asked questions.

Q&A Board guests, by invite location

Looking at the proportion of invitees who actually showed up, the Database report and Feedback Dashboard seem to be the best place to find new editors to invite. This suggests that hosts should focus on inviting editors from these locations, rather than focusing so heavily on the New editor contribs page.

Many guests who were not invited by hosts showed up at the Teahouse Q&A board. Many of these guests were students who are currently studying Wikipedia and were invited to participate by their Online Ambassadors or by J-Mo. We can expect the number of student participants to decline in coming weeks.

* percent of people invited through that location who asked a question on the Q&A board.

March 5th report[edit]

Hi everyone!

I wanted to share some number from our first week. These are not only interesting to look at, they can help us spot trends and identify strategies that are working. I'm posting it here for now because this is where the action is, but if someone wants to make a 'reporting' page for hosts and throw it up there, that would be totally fine with me.


Invite rockstars[edit]

First, invites! Here are the hosts who invited users to Teahouse (at least those who noted it on the "Invite Tracking" spreadsheet). Congrats to Rosie and Sarah for their contribution here!

Overall metrics[edit]

Next, I wanted to give some overall metrics on invitees. These are already out of date, as we've had several new questions and intros since I ran the numbers yesterday. The overall count isn't that impressive yet, but in general we're seeing a steady acceleration in the number of newbies per day.


Sampling strategies[edit]

As you can see, we sampled from a variety of different locations. Some locations seem offer more bang for your buck than others. In particular, the database reports and the new editor contrib filter seem like the most promising places to look for newbies to invite.



To email or not to email?[edit]

I wanted to mention one thing I saw in the data in particular: emailing users who have the email this user feature really seems to work. Consider this: we only emailed 102 users. However, of the 30 newbies who showed up, 7 had received an email. That's almost 25%! So it looks like using the "Email this user" feature in addition to using a talk page template is a good strategy for reaching out to newbies, if they have it enabled (the database report page shows which ones do). And remember that if you use the "email this user" feature, your email will not be shown.

That's it for now. Any questions for me? How are things going in general? Any 'feature requests' (I've already got the header for the invite templates on my to-do list)?. Keep up the good work! - J-Mo 20:12, 5 March 2012 (UTC)

I suggest a slightly different calculation:
102 people were emailed; 7 showed up (just under 7%)
592 people were contacted via user talk pages [694-102]; 23 showed up (just under 4%)
So emails appear to be roughly twice as effective as user talk pages. Unfortunately, there are two assumptions in that conclusion that may not be valid:
(1) New editors who enable email are similar to those who do not. (The only way to test this assumptions is to take a group of editors who have enabled email, and send half of them an email and half a user talk page invitation. It's not clear that was done. Alternatively, one could look at the 592 people contacted via user talk pages, and see what the response rate was for those with email enabled.)
(2) Those receiving an email invitation came from seven places where newbies were found, and in the same proportion, as those getting user talk page invitations. (Why is this critical? If emails were sent, for example, only to new editors found via the database report and the new editor contrib filter, then the expected response rate would be between 5 and 7%, and getting a 7% response would not indicate that email is better. Again, a systematic approach - start with a homogenous group, say only those with email enabled and found via the new editor contrib filter, and then split it into halves, randomly - is the only way to be comfortable with the conclusion that email is better.) -- John Broughton (♫♫) 00:17, 10 March 2012 (UTC)
Hi John, thanks for your input. I don't have a hypothesis either way about (1) or enough data to begin measuring it accurately if I did, and we definitely don't have a large enough sample yet to figure out whether or not (2) is true. There are other potential confounds too, of course. As there always are. :) I'd love to chat with you about them, if I see you at #wikipedia-teahouse. cheers, J-Mo Talk to Me Email Me 15:41, 13 March 2012 (UTC)