Research:Onboarding new Wikipedians/OB4.5

Overview

Screenshot of notifications

Cohorts

After we enabled the Getting Started landing page in OB4 as default for 100% of new signups, the Extension:Echo, a new notification system was deployed on English Wikipedia. Following launch, all new editors received both the Getting Started landing page with task suggestions, as well as the following notifications:

1. A generic welcome message
2. On email confirmation, a message via Web and email directing users to Special:GettingStarted, with slightly different language depending on whether the user had made a namespace zero edit or not.

The cohort of these users starts from launch of Echo at 8pm UTC on April 30, 2013, to the launch of the "red popup" gadget. Users after the gadget launch still receive notifications, but their experience of interacting with Web notifications is significantly more annoying and confusing, so we will discount these users from the initial cohort.)

In the present analysis we considered all new desktop users registered on the English Wikipedia between 2012-04-30T20:00:00Z and 2012-05-07T00:37:00Z.

Notes

Notifications generated by GettingStarted are special insofar as they are triggered (both on-site and by mail) only for users with an authenticated email address. This introduces a potential confound in the analysis, as we are not measuring the effect of notifications on the total population of new users registered in the above period, but on the subpopulation of users with an authenticated email address (who may have a higher level of engagement than occasional users who register without specifying an email address or without verifying it). We can correct for this confound by comparing this cohort only to the previous OB4 users who authenticated an email address.

Even without the above confound, minor differences are potentially due to seasonal fluctuations or other unknown differences between two week-long cohorts, but it may still be worthwhile to see if there are drastically better or worse activity levels in the cohort that got notifications. In the future we will run a proper A/B test measuring the impact of any notifications.

Research questions

• What is the overall activity level of the cohort who received notifications?
• How does this cohort's activity level compare to that of OB4, which received the same landing page but no notifications?

Metrics

• The rate at which this cohort hit 1+ NS0 edits in 24hrs
• The rate at which this cohort hit 5+ NS0 edits in 24hrs

Cohort analysis

OB4.5 cohorts

Cohort ID	Unique users	Description
e3_ob45	20,655	All new users registered in the 1st week since the launch of Echo notifications and landing on GettingStarted v.2.
e3_ob45_echo_gettingstarted-start-editing	4,703	Users with an authenticated email address registered in the 1st week since the launch of Echo notifications who received a gettingstarted "start editing" notification by mail within the same 1-week period.
e3_ob45_echo_gettingstarted-continue-editing	231	Users with an authenticated email address registered in the 1st week since the launch of Echo notifications who completed their first edit within the same 1-week period, received a gettingstarted "continue editing" notification and visited once more Special:GettingStarted within the same period.

Live account rate

We measured the proportion of "live accounts" or users in each cohort clicking at least once on the edit button on an article in the main namespace within 24 hours of registration (measurement taken 24h after the last valid registration).

OB4.5 ns0 24h live accounts

ID	Unique users	Live accounts	%
e3_ob45	20,655	7,699	37.3%
e3_ob45_echo_gettingstarted-start-editing	4,703	972	20.7%
e3_ob45_echo_gettingstarted-continue-editing	231	204	88.3%

1 edit in 24 hours

We measured the proportion of users in each cohort completing their first main-namespace edit within 24 hours of registration (measurement taken 24h after the last valid registration).

OB4.5 1+ ns0-edit 24h threshold

ID	Unique users	Editing users	%
e3_ob45	20,655	5,161	25.0%
e3_ob45_echo_gettingstarted-start-editing	4,703	354	7.5%
e3_ob45_echo_gettingstarted-continue-editing	231	223	96.5%

Note

• The near-100% threshold hitter rate for the e3_ob45_echo_gettingstarted-continue-editing cohort is expected by the very definition of this cohort.

5+ edits in 24 hours

We also measured the proportion of users in each cohort completing 5 or more main-namespace edit within 24 hours of registration (measurement taken 24h after the last valid registration).

OB4.5 1+ ns0-edit 24h threshold

ID	Unique users	Editing users	%
e3_ob45	20,655	864	4.2%
e3_ob45_echo_gettingstarted-start-editing	4,703	71	1.5%
e3_ob45_echo_gettingstarted-continue-editing	231	47	20.3%

Conclusions

In this cohort analysis (i.e. not a controlled A/B test, but an observational analysis of a group of editors who all joined during the same week), we found that the entire cohort exposed to the new GettingStarted notifications had a first time edit rate of 25% -- slightly higher than previous cohorts. We also found that users who did not yet edit by the time they confirmed their email address, but who received a notification inviting them to do so, came back and edited at a rate of 7.5%. While users who received a notification inviting them to continue editing if they are already had were very successful (20.3% of them reached 5 NS0 edits in 24hrs after registration), this is likely not due to the notifications alone. In the future, we may run a controlled test to further determine the precise impact of notifications on new editors.