Research:Onboarding new Wikipedians/OB5

From Meta, a Wikimedia project coordination wiki
Shortcut:
R:OB5

In this iteration of onboarding experiments, we will compare a control group to one delivered a new landing GettingStarted page, as well as a new navigation bar delivered to those who accept GettingStarted tasks, which lets them skip to a new random task, go back to the GettingStarted landing page, or generate a short guided tour showing them how to complete the task.

Overview[edit]

Cohorts[edit]

Key events in the gettingstarted funnels, test cohort
  • Control - delivered no GettingStarted
  • Test - delivered GettingStarted, which breaks down in the following funnels:
    • gettingstarted-copyediting - editors who chose the copyediting task
    • gettingstarted-clarify - editors who chose the clarification task
    • gettingstarted-addlinks - editors who chose the add wikilinks task
    • returnto - editors who opted to click the "No thanks, return to the page I was reading" link to their internal referrer

Metrics[edit]

  • 1+ NS0 edits in 24 hrs
  • 5+ NS0 edits in 24 hrs
  • Revert rate

Research questions[edit]

  • How do the conversion rates of gettingstarted users exposed to this version of the landing page compared to the control group? Specifically...
    • Does the navigation bar's functions for returning to Special:GettingStarted or skipping to a new random article enhance conversion rates?
    • Does the guided tour generated by "Show help" enhance conversion rates and editor productivity?
    • How do these relative conversion rates compare to the results OB2 and OB4?
    • How do the conversion rates among task types (copyediting, clarification, add links) compare to each other?
  • What is the quality of edits made in each task funnel, as measured by revert rate and hand-coding?
  • Is there a difference (and what kind) in the activity of editors who authenticated an email address and those who did not, in both control and test? Note: Email is confound in this experiment, as it is likely that users who do authenticate an email address have naturally higher levels of engagement, in addition to receiving gettingstarted notifications.

Data[edit]

We ran the test for one week and logged activity for users in the two groups via the Schema:GettingStartedNavbar log. We introduced a fix on May 20 due to a bug that affected logging of users clicking on the ReturnTo button in the control condition.

OB5 timeline
Schema revid start end log version notes
5467309 2013-05-16T21:57:59Z 2013-05-20T23:37:37Z 1 Missing users in the returnto funnel
5496876 2013-05-20T23:38:26Z 2013-05-23T20:31:09Z 2 Fixed logging

Cohort analysis[edit]

We defined the following cohorts in order to measure their respective productivity. We included in the sample all new registered users on the English Wikipedia (excluding mobile registrations) during the two rounds of the split test (log version: 1, 2). Cohorts in the GettingStarted funnel were obtained by checking for the first click upon account creation. Cohorts in the ReturnTo funnel were obtained by checking for impressions on an editable article linked via the ReturnTo button.

OB5 cohorts
Cohort ID Unique users Description % of total bucketed in group
All users
e3_ob5a 12,201 Users in the control group 100%
e3_ob5b 12,375 Users in the test group 100%
GettingStarted funnel
e3_ob5b_gettingstarted-any_page-impression 4,047 Unique users in the test group who clicked on one of the three GettingStarted buttons upon successful account creation. 32.7%
e3_ob5b_gettingstarted-addlinks_page-impression 1,446 Users in the test group who clicked "Add links" upon successful account creation. 11.7%
e3_ob5b_gettingstarted-clarify_page-impression 981 Users in the test group who clicked "Clarify" upon successful account creation. 7.9%
e3_ob5b_gettingstarted-copyedit_page-impression 1,661 Users in the test group who clicked "Copyedit" upon successful account creation. 13.4%
ReturnTo funnel
e3_ob5a_returnto_page-impression 693** Users in the control group who returned to an editable page via the ReturnTo button upon successful account creation. 5.7%
e3_ob5b_returnto_page-impression 1,076 Users in the test group who returned to an editable page via the ReturnTo button upon successful account creation. 8.7%

Notes

  • Due to a bug that affected the logging of events in the control condition in v1, the e3_ob5a_returnto_page-impression cohort is incomplete and its size cannot be used for historical comparison.

Live account rate[edit]

We measured the proportion of "live accounts" or users in each cohort clicking at least once on the edit button on an article in the main namespace within 24 hours of registration (measurements taken 24 hours after the last valid registration).

OB5 ns0 24h live accounts (test vs control)
ID Unique users Live accounts %
e3_ob5a 12,201 4,142 33.9%
e3_ob5b 12,375 4,801 38.8%

We found a significant 4.9% difference in the proportion of live accounts between test and control [X² = 62.16, N = 24,576, p < .01]. The live account rates for the other cohorts are the following:

OB5 ns0 24h live accounts (subcohorts)
ID Unique users Live accounts %
GettingStarted funnel
e3_ob5b_gettingstarted-any_page-impression 4,047 1,731 42.8%
e3_ob5b_gettingstarted-addlinks_page-impression 1,446 521 36.0%
e3_ob5b_gettingstarted-clarify_page-impression 981 494 50.3%
e3_ob5b_gettingstarted-copyedit_page-impression 1,661 728 43.8%
ReturnTo funnel
e3_ob5a_returnto_page-impression 693 320 46.2%
e3_ob5b_returnto_page-impression 1076 450 41.8%

1 edit in 24 hours[edit]

We measured the proportion of users in each cohort completing their first main-namespace edit within 24 hours of registration (measurements taken 24 hours after the last valid registration).

OB5 1+ ns0-edit 24h threshold (test vs control)
ID Unique users Editing users %
e3_ob5a 12,201 2,831 23.2%
e3_ob5b 12,375 3,081 24.9%

We compared the proportion of threshold-hitting users in each group and found a significant 1.7% difference between test and control [X² = 9.55, N = 24,576, p < .01]. The threshold-hitter rates for the other cohorts are the following:

OB5 1+ ns0-edit 24h threshold (subcohorts)
ID Unique users Editing users %
GettingStarted funnel
e3_ob5b_gettingstarted-any_page-impression 4,047 928 22.9%
e3_ob5b_gettingstarted-addlinks_page-impression 1,446 230 15.9%
e3_ob5b_gettingstarted-clarify_page-impression 981 294 30.0%
e3_ob5b_gettingstarted-copyedit_page-impression 1,661 409 24.6%
ReturnTo funnel
e3_ob5a_returnto_page-impression 693 201 29.0%
e3_ob5b_returnto_page-impression 1,076 301 28.0%

5+ edits in 24 hours[edit]

We also measured the proportion of users in each cohort completing 5 or more main-namespace edits within 24 hours of registration (measurements taken 24 hours after the last valid registration). The proportion of 5+ threshold hitters varies insignificantly between the two conditions.

OB5 5+ ns0-edit 24h threshold (test vs control)
ID Unique users Editing users %
e3_ob5a 12,201 468 3.8%
e3_ob5b 12,375 489 3.9%

The 5+ threshold-hitter rates for the other cohorts are the following:

OB5 5+ ns0-edit 24h threshold (subcohorts)
ID Unique users Editing users %
GettingStarted funnel
e3_ob5b_gettingstarted-any_page-impression 4,047 142 3.5%
e3_ob5b_gettingstarted-addlinks_page-impression 1,446 39 2.7%
e3_ob5b_gettingstarted-clarify_page-impression 981 40 4.0%
e3_ob5b_gettingstarted-copyedit_page-impression 1661 65 3.9%
ReturnTo funnel
e3_ob5a_returnto_page-impression 693 29 4.2%
e3_ob5b_returnto_page-impression 1,076 45 4.2%

Revert rate[edit]

OB5 24-hour mean revert rate (test vs control)
ID Unique users Reverts %
e3_ob5a 12,201 998 8.1%
e3_ob5b 12,375 1,046 8.4%

Funnel analysis[edit]

Guided Tour[edit]

The following is the breakdown of users in the e3_ob5b_gettingstarted-any_page-impression group taking the Guided Tour embedded in the navigation bar within 24 hours of the end of the split test.

GuidedTour funnel
ID Total users Step 1 Step 2 Step 3 Step 4 Step 5
e3_ob5b_gettingstarted-any_page-impression 4,047 (100%) 366 (9.0% / 9.0%) 322 (88.0% / 7.9%) 274 (85.0% / 6.8%) 173 (63.1% / 4.3%) 54 (31.2% / 1.3%)

Conclusions[edit]

Though we have not yet made a gain in the rate of active editors hitting the 5+ edits threshold, we have so far observed the largest gain in new editors hitting the 1+ edits threshold of any of our controlled tests of Getting Started. Funnel analysis and further analysis of subcohorts (such as by task type and those who opted in to the guided tour) is merited, to understand more detail about which features may be driving this effect.