Research:Onboarding new Wikipedians/OB2

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
Nutshell.png
This page in a nutshell: We ran a 1-week controlled test serving tasks to new users via GettingStarted. We found a significant increase in the proportion of new users clicking on the edit button (+4.3%) and successfully completing their first main-namespace edit (+1.8%) within 1 day of registration, compared to a control group.

Overview[edit]

Screenshot from just before the test period

This 1-week iteration started on 2013-01-25T00:00:00Z and ended on 2013-01-31T22:45:00Z. We randomly split new registered users on the English Wikipedia into a control group (OB2a) and a test group (OB2b). Users in the control group saw Wikipedia's default welcome screen. Users in the test group were served tasks via GettingStarted. We logged the editing activity of users entering one of the two following subfunnels from the GettingStarted landing page for the test group:

returnto
users clicking on the "Return To" button and attempting to edit the page they came from when they registered an account
gettingstarted
users clicking on one or more links in the task section of the GettingStarted page

Cohort analysis[edit]

We defined the following cohorts in order to measure their respective productivity:

OB2 cohorts
ID Unique users Description
e3_ob2a 14,595 Users in the control group
e3_ob2b 14,488 Users in the test group being served tasks via GettingStarted

Live account rate[edit]

Proportion of new registered users in the OB2 test clicking the edit button at least once in the main namespace within 24 hours of registration.

We measured the proportion of "live accounts" or users in each cohort clicking at least once on the edit button on an article in the main namespace within 24 hours of registration (measurement taken at 2013-02-01T00:00:00Z).

OB2 ns0 24h live account rate
ID Unique users Editing users  %
e3_ob2a 14,595 3,938 27.0%
e3_ob2b 14,488 4,533 31.3%

We compared the proportion of live accounts in the e3_ob2a and e3_ob2b cohorts and found a significant 4.30% increase in the test group compared to control (a chi-square test rejects the null hypothesis that the true proportion of live accounts is the same in both groups [X² = 65.1, N = 29,083, p < .001]).

1 edit in 24 hours[edit]

Proportion of 1+ 24h main-namespace editors among new registered users in the OB2 test.

We measured the proportion of users in each cohort completing their first main-namespace edit within 24 hours of registration (measurement taken at 2013-02-01T00:00:00Z).

OB2 1+ ns0-edit 24h threshold
ID Unique users Editing users  %
e3_ob2a 14,595 2921 20.0%
e3_ob2b 14,488 3,168 21.9%

We compared the proportion of threshold-hitting users in the e3_ob2a and e3_ob2b cohorts and found a significant 1.85% difference between the test and the control group (a chi-square test rejects the null hypothesis that the true proportion of threshold-hitting accounts is the same in both groups [X² = 14.96, N = 29,083, p < .001]).

5+ edits in 24 hours / 1 week[edit]

We also measured the proportion of users in each cohort completing their first 5 main-namespace edits within 24 hours of registration and found no significant difference between the groups (2.9% for control and 3.0% for test). The difference remains non significant when we compare the proportion of users editing 5+ times the main namespace within 1 week of registration (4.0% for control and 4.1% for test).

In the first 24 hours, roughly 14% of those who completed one NS0 edit from the control completed five. In the test group, this proportion was roughly the same, at 13%, though we have not performed a statistical test to compare these two.

OB2 5+ edits in NS0 threshold
control test
in 24 hours 424 438
in 1 week 582 599
total cohorts 14,595 14,488