Research:Onboarding new Wikipedians/OB1

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
Nutshell.png
This page in a nutshell: Shortly after launching the onboarding experiment, we collected data for 10 days, comparing the activity of users who chose 'getting started' tasks with those who completed edits after returning to their internal referrer (pre-registration). We found a significant 6.2% difference in the rate at which gettingstarted users completed a main namespace edit in 24 hours, compared to those who directly returned to an editable article. We did not see a statistically significant difference in the rate at which any cohort reached 5 edits in 24 hours.

Overview[edit]

Screenshot

This iteration started on 2013-01-15 and ended on 2013-01-25T00:00:00Z and logged the editing activity of users entering one of the two following subfunnels from the GettingStarted landing page:

returnto
users clicking on the "Return To" button and attempting to edit the page they came from when they registered an account
gettingstarted
users clicking on one or more links in the task section of the GettingStarted page

Funnel analysis[edit]

Cohort analysis[edit]

We defined the following cohorts in order to measure their respective productivity:

Cohort definition[edit]

OB1 cohorts
ID Unique users Description  % of group
e3_ob1 35,526 Users being served tasks via GettingStarted 100%
e3_ob1_noclick 29,624 GettingStarted users neither clicking on a gettingstarted link nor the returnto button 83.4%
e3_ob1_returnto_click 4,712 GettingStarted users clicking on the returnto link 13.3%
e3_ob1_gettingstarted_click 1,316 GettingStarted users clicking on one or more gettingstarted links 3.7%
e3_ob1_returnto_page-impression 5,904 GettingStarted users landing on an editable page in the returnTo funnel 16.6%
e3_ob1_gettingstarted_page-impression 3,757 GettingStarted users landing on a page in the gettingstarted funnel 10.6%

Notes:

  • The overlap between returnto and gettingstarted clickers is very small (N=126) and so is the overlap between users landing on an editable page via the returnto funnel and the gettingstarted funnel (N=40): the two groups are therefore clearly disjoint. The discrepancy between clicks and page-impressions is likely due to users using right-clicks to open multiple tabs, which we're currently not logging due to limitations of EventLogging.
  • Editable pages are those flagged with isEditable=true in Schema:GettingStarted.

1 edit in 24 hours threshold[edit]

We measured the proportion of users in each cohort completing their first main-namespace edit within 24 hours since registration.

OB1 1+ ns0-edit 24h threshold
ID Unique users Editing users  %
e3_ob1 35,526 8,124 23.1%
e3_ob1_noclick 29,624 6,848 23.1%
e3_ob1_returnto_click 4,712 1,041 22.1%
e3_ob1_gettingstarted_click 1,316 325 24.7%
e3_ob1_returnto_page-impression 5,904 1,173 19.9%
e3_ob1_gettingstarted_page-impression 3,757 980 26.1%

We compared the proportion of threshold-hitting users in the e3_ob1_returnto_page-impression and e3_ob1_gettingstarted_page-impression cohorts and found a significant 6.2% difference between these cohorts (a chi-square test rejects the null hypothesis that the true proportion of threshold-hitting accounts is the same in both groups [X² = 50.87, N = 9,661, p < .001]). The difference in the proportion of threshold-hitters between the e3_ob1_gettingstarted_page-impression and the e3_ob1_noclick cohorts (3.0%) is also significant [X² = 16.19, N = 33,381, p < .001]).

5 edits in 24 hours threshold[edit]

We measured the proportion of users in each cohort completing their first 5 main-namespace edit within 24 hours since registration.

OB1 5+ ns0-edit 24h threshold
ID Unique users Editing users  %  % from 1+ editing users
e3_ob1 35,526 1,157 3.2% 14.24%
e3_ob1_noclick 29,624 972 3.3% 14.19%
e3_ob1_returnto_click 4,712 145 3.1% 13.93%
e3_ob1_gettingstarted_click 1,316 40 3.0% 12.31%
e3_ob1_returnto_page-impression 5,904 180 3.0% 15.35%
e3_ob1_gettingstarted_page-impression 3,757 119 3.2% 12.14%

The proportion of threshold-hitters varies insignificantly between 3.0%-3.3% across the cohorts.