Research:Onboarding new Wikipedians/OB3

From Meta, a Wikimedia project coordination wiki
Shortcut:
R:OB3

Overview[edit]

This iteration started on 2013-02-28T21:50:18Z[1] and ended on 2013-03-08T00:20:00.[2] We randomly split a sample of new users registered on the English Wikipedia into a control group (OB3a) and a test group (OB3b). Users in the control group were served a list of copyediting tasks generated by Extension:GettingStarted. Users in the test group saw the same tasks but in addition they were shown guiders when visiting any of the articles in the list. We showed the returnto button to these users but the button doesn't trigger a guided tour and it behaves exactly as in the control group.

We logged the editing activity of users entering one of the two following subfunnels from the GettingStarted landing page in both the control and the test group:

returnto
users clicking on the "Return To" button and attempting to edit the page they came from when they registered an account
gettingstarted
users clicking on one or more links in the task section of the GettingStarted page

Cohort analysis[edit]

We defined the following cohorts in order to measure their respective productivity:

OB3 cohorts
ID Unique users Description % of group
e3_ob3-split_a 15,821 Users in the control group 100%
e3_ob3-split_b 15,892 Users in the test group 100%
e3_ob3-split_a_gettingstarted_page-impression 1,734 Users in the control group who landed on a GettingStarted article upon successful account creation. 11.0%
e3_ob3-split_b_gettingstarted_page-impression 1,790 Users in the test group who landed on a GettingStarted article upon successful account creation. 11.3%
e3_ob3-split_a_returnto_page-impression 2,811 Users in the control group who returned to an editable page via the ReturnTo button upon successful account creation. 17.8%
e3_ob3-split_b_returnto_page-impression 2,842 Users in the test group who returned to an editable page via the ReturnTo button upon successful account creation. 17.9%
e3_ob3-split_a_other 11,289 Users in the control group neither returning to an editable page via the ReturnTo button nor landing on a GettingStarted article upon successful account creation. 71.3%
e3_ob3-split_b_other 11,279 Users in the test group neither returning to an editable page via the ReturnTo button nor landing on a GettingStarted article upon successful account creation. 71.0%

Live account rate[edit]

Fig.1 Proportion of new registered users in the OB3 test accepting a task and clicking the edit button at least once in the main namespace within 24 hours of registration.

We measured the proportion of "live accounts" or users in each cohort clicking at least once on the edit button on an article in the main namespace within 24 hours of registration (measurement taken 24h after the last valid registration).

OB3 ns0 24h live account rate (test vs control)
ID Unique users Live accounts %
e3_ob3-split_a 15,821 5,061 32.0%
e3_ob3-split_b 15,892 5,429 34.2%
OB3 ns0 24h live account rate (test vs control, users accepting a task only)
ID Unique users Live accounts %
e3_ob3-split_a_gettingstarted_page-impression 1,734 792 45.7%
e3_ob3-split_b_gettingstarted_page-impression 1,790 1,062 59.3%

We compared the proportion of live accounts in the two experimental groups and found a significant 2.2% increase in the test group compared to control [X² = 16.80, N = 31,713, p < .001]. The effect is much more pronounced if we only consider the subgroup of users accepting a task (i.e. landing on an article linked from the task) in the two conditions, where the difference between test and control is 13.6% [X² = 65.3, N = 3,524, p < .001] (see fig.1).

Baseline cohorts

OB3 ns0 24h live account rate (other users only)
ID Unique users Live accounts %
e3_ob3-split_a_other 11,289 3,419 30.3%
e3_ob3-split_b_other 11,279 3,487 30.9%
OB3 ns0 24h live account rate (returnTo users only)
ID Unique users Live accounts %
e3_ob3-split_a_returnto_page-impression 2,811 853 30.3%
e3_ob3-split_b_returnto_page-impression 2,842 885 31.1%

1 edit in 24 hours[edit]

Fig.2 Proportion of 1+ 24h main-namespace editors among new registered users in the OB3 test who accepted a task (test vs control).

We measured the proportion of users in each cohort completing their first main-namespace edit within 24 hours of registration (measurement taken 24h after the last valid registration).

OB3 1+ ns0-edit 24h threshold (test vs control)
ID Unique users Editing users %
e3_ob3-split_a 15,821 3,377 21.3%
e3_ob3-split_b 15,892 3,565 22.4%
OB3 1+ ns0-edit 24h threshold (test vs control, users accepting a task only)
ID Unique users Editing users %
e3_ob3-split_a_gettingstarted_page-impression 1,734 428 24.7%
e3_ob3-split_b_gettingstarted_page-impression 1,790 511 28.5%

We compared the proportion of threshold-hitting users in each group and found a significant 1.1% difference between test and control [X² = 5.42, N = 31,713, p < .05]. There was a larger, significant difference of 3.9% between test and control for users accepting a task [X² = 6.53, N = 3,524, p < .05] (see fig.2).


Baseline cohorts

OB3 1+ ns0-edit 24h threshold (other users only)
ID Unique users Editing users %
e3_ob3-split_a_other 11,289 2,462 21.8%
e3_ob3-split_b_other 11,279 2,532 22.4%
OB3 1+ ns0-edit 24h threshold (returnTo users only)
ID Unique users Editing users %
e3_ob3-split_a_returnto_page-impression 2,811 488 17.4%
e3_ob3-split_b_returnto_page-impression 2,842 523 18.4%

5+ edits in 24 hours[edit]

We also measured the proportion of users in each cohort completing 5 or more main-namespace edits within 24 hours of registration (measurement taken 24h after the last valid registration). The proportion of 5+ threshold hitters varies insignificantly between the two conditions (both when comparing the experimental groups with each other or the subgroup of users accepting a task).

OB3 5+ ns0-edit 24h threshold (test vs control)
ID Unique users Editing users %
e3_ob3-split_a 15,821 483 3.0%
e3_ob3-split_b 15,892 528 3.3%
OB3 5+ ns0-edit 24h threshold (test vs control, users accepting a task only)
ID Unique users Editing users %
e3_ob3-split_a_gettingstarted_page-impression 1,734 48 2.7%
e3_ob3-split_b_gettingstarted_page-impression 1,790 68 3.8%

Baseline cohorts

OB3 5+ ns0-edit 24h threshold (other users only)
ID Unique users Editing users %
e3_ob3-split_a_other 11,289 354 3.1%
e3_ob3-split_b_other 11,279 385 3.4%
OB3 5+ ns0-edit 24h threshold (returnTo users only)
ID Unique users Editing users %
e3_ob3-split_a_returnto_page-impression 2,811 81 2.9%
e3_ob3-split_b_returnto_page-impression 2,842 76 2.7%

QA[edit]

(early data collected as of 2013-02-11T21:30:00)

OB3 cohorts
ID Unique users (served GettingStarted) Description
e3_ob3a 7,651 Users in the control group
e3_ob3b 7,706 Users in the test group being served the 'gettingstarted' guided tour
Buckets
All eligible users in the test are accurately bucketed into one of the two groups (no eligible user missing a bucketId) and the sum of users with a valid bucketId is identical to the sample of eligible users (no spurious bucketIds corresponding to non-eligible users).
GettingStarted page-impression vs GuidedTour impression
697 eligible unique users generated a GuidedTour impression out of a total 726 users in the test cohort (the 29 user difference being due to users who tested guided tour without being part of the OB3 experiment). Out of these 697 users, 87 were in the control group but generated guided tour impressions erroneously. These 87 users in the control group who landed on an editable page in the gettingstarted funnel are a potential issue as they should not have been served a guided tour and should not generate any GuidedTour event.

We will be correcting for this error where 87 users in the control group were served the guided tour by ensuring that only first time visitors can be served the tour. When that fix was deployed 2/14 (48596), we ran the test for a full week through 2/21.

Also note that for the week-long test of 2/14-2/21, bug 45251 means that a number of users were logged as being in the test group, but may have not been delivered the test tour. This led us to re-enable the test from Friday 2/22-Thursday 3/07.

Notes[edit]

  1. First logged impression of a new user landing on the GettingStarted page uuid:fc3595d2e0cd5fb18fb23cfda7e97209.
  2. Last logged GettingStarted page impression from a new registered user uuid:27988c110a2a508783c7696ad558382a