Research:Onboarding new Wikipedians/OB4.5

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search
Shortcut:
R:OB4.5
Walnut.svg
This page in a nutshell: The first cohort we observed which received Getting Started notifications had a slightly higher rate of first article edits (25%) than other cohorts, and it is likely that we are slightly increasing the overall number of first time editors via notifications. Further testing is warranted, if we want to understand the precise impact of notifications on new editors.

Overview[edit]

Screenshot of notifications

Cohorts[edit]

After we enabled the Getting Started landing page in OB4 as default for 100% of new signups, the Extension:Echo, a new notification system was deployed on English Wikipedia. Following launch, all new editors received both the Getting Started landing page with task suggestions, as well as the following notifications:

  1. A generic welcome message
  2. On email confirmation, a message via Web and email directing users to Special:GettingStarted, with slightly different language depending on whether the user had made a namespace zero edit or not.

The cohort of these users starts from launch of Echo at 8pm UTC on April 30, 2013[1], to the launch of the "red popup" gadget[2]. Users after the gadget launch still receive notifications, but their experience of interacting with Web notifications is significantly more annoying and confusing, so we will discount these users from the initial cohort.)

In the present analysis we considered all new desktop users registered on the English Wikipedia between 2012-04-30T20:00:00Z and 2012-05-07T00:37:00Z.

Notes[edit]

Notifications generated by GettingStarted are special insofar as they are triggered (both on-site and by mail) only for users with an authenticated email address. This introduces a potential confound in the analysis, as we are not measuring the effect of notifications on the total population of new users registered in the above period, but on the subpopulation of users with an authenticated email address (who may have a higher level of engagement than occasional users who register without specifying an email address or without verifying it). We can correct for this confound by comparing this cohort only to the previous OB4 users who authenticated an email address.

Even without the above confound, minor differences are potentially due to seasonal fluctuations or other unknown differences between two week-long cohorts, but it may still be worthwhile to see if there are drastically better or worse activity levels in the cohort that got notifications. In the future we will run a proper A/B test measuring the impact of any notifications.

Research questions[edit]

  • What is the overall activity level of the cohort who received notifications?
  • How does this cohort's activity level compare to that of OB4, which received the same landing page but no notifications?

Metrics[edit]

  • The rate at which this cohort hit 1+ NS0 edits in 24hrs
  • The rate at which this cohort hit 5+ NS0 edits in 24hrs

Cohort analysis[edit]

OB4.5 cohorts
Cohort ID Unique users Description
e3_ob45 20,655 All new users registered in the 1st week since the launch of Echo notifications and landing on GettingStarted v.2.
e3_ob45_echo_gettingstarted-start-editing 4,703 Users with an authenticated email address registered in the 1st week since the launch of Echo notifications who received a gettingstarted "start editing" notification by mail within the same 1-week period.
e3_ob45_echo_gettingstarted-continue-editing 231 Users with an authenticated email address registered in the 1st week since the launch of Echo notifications who completed their first edit within the same 1-week period, received a gettingstarted "continue editing" notification and visited once more Special:GettingStarted within the same period.

Live account rate[edit]

We measured the proportion of "live accounts" or users in each cohort clicking at least once on the edit button on an article in the main namespace within 24 hours of registration (measurement taken 24h after the last valid registration).

OB4.5 ns0 24h live accounts
ID Unique users Live accounts %
e3_ob45 20,655 7,699 37.3%
e3_ob45_echo_gettingstarted-start-editing 4,703 972 20.7%
e3_ob45_echo_gettingstarted-continue-editing 231 204 88.3%

1 edit in 24 hours[edit]

We measured the proportion of users in each cohort completing their first main-namespace edit within 24 hours of registration (measurement taken 24h after the last valid registration).

OB4.5 1+ ns0-edit 24h threshold
ID Unique users Editing users %
e3_ob45 20,655 5,161 25.0%
e3_ob45_echo_gettingstarted-start-editing 4,703 354 7.5%
e3_ob45_echo_gettingstarted-continue-editing 231 223 96.5%

Note

5+ edits in 24 hours[edit]

We also measured the proportion of users in each cohort completing 5 or more main-namespace edit within 24 hours of registration (measurement taken 24h after the last valid registration).

OB4.5 1+ ns0-edit 24h threshold
ID Unique users Editing users %
e3_ob45 20,655 864 4.2%
e3_ob45_echo_gettingstarted-start-editing 4,703 71 1.5%
e3_ob45_echo_gettingstarted-continue-editing 231 47 20.3%

Conclusions[edit]

In this cohort analysis (i.e. not a controlled A/B test, but an observational analysis of a group of editors who all joined during the same week), we found that the entire cohort exposed to the new GettingStarted notifications had a first time edit rate of 25% -- slightly higher than previous cohorts. We also found that users who did not yet edit by the time they confirmed their email address, but who received a notification inviting them to do so, came back and edited at a rate of 7.5%. While users who received a notification inviting them to continue editing if they are already had were very successful (20.3% of them reached 5 NS0 edits in 24hrs after registration), this is likely not due to the notifications alone. In the future, we may run a controlled test to further determine the precise impact of notifications on new editors.

Notes[edit]

  1. The first gettingstarted-start-edit notification was delivered at 2013-04-30T19:58:59Z
  2. This community gadget was turned on by default at 00:37 UTC on May 7, 2013