Research:Mobile editor engagement/Tutorial test
The tutorial A/B test is a controlled experiment designed to test if displaying a short tutorial about wiki markup to newly registered users editing on mobile devices affects their edit completion rate. The test was initially enabled for Beta users only. The actual controlled experiment was rolled out in November for users of the stable version of the mobile site.
Timeline[edit]
- 2013-11-01 00:00:00
- Test starts
- 2013-11-15 00:00:00
- Test ends
Experimental design[edit]
- We selected a sample of users newly registered via a mobile device on the English Wikipedia for a period of at least two weeks. Only users registering via the edit CTA were considered eligible for this test.
- We randomly assigned eligible users in the sample to one of the following two conditions, based on their user id
- Tutorial (test)
- when clicking for the first time on the edit button (whether on a section or on the entire page), the user sees a tutorial explaining how to edit wiki markup.
- No tutorial (control)
- users clicking on the edit button are directly sent to the edit screen. Note that this is not the default UX for new users registering via the edit CTA (as they are normally redirected to the article they came from, not the edit screen).
- Users remained bucketed in the experimental condition for the duration of the test (initially set to 2 weeks). At the end of the test, they reverted to default condition.
Research questions[edit]
- Does displaying a tutorial increase the edit completion rate of new mobile registered users (proportion of users registering via the edit CTA funnel and successfully editing the page they came from)?
- Does the tutorial affect the overall activation rate of these users?
- Are there differences in the quality of contributions made by users in each condition, as measured via unreverted edits?
Data sources[edit]
We used for this test a combination of data from the MediaWiki database and EventLogging logs (Schema:MobileWebEditing).
Cohort analysis[edit]
A total of 3,434 users mobile users registered on the English Wikipedia via the Edit CTA participated in this test.
| Cohort ID | Unique users | Description |
|---|---|---|
| mobile_tutorial_test_control | 1,780 | Users in the control group |
| mobile_tutorial_test_test | 1,654 | Users in the test group |
The following is the proportion of users in each group attempting and completing an edit on the referral page. We only considered editable ns0 pages and conservatively included edit events up to 1 hour after registration (i.e. not limited to the account registration referral page).
- We found a significant 8.5% difference in the proportion of users attempting to edit the referral page between control and test [X² = 27.95, N = 3,434, p < .001].
- Test was the winning condition for edit completion, with a significant, 8.4% difference in the proportion of users successfully completing an edit of the referral page compared to users in the control group [X² = 28.70, N = 3,434, p < .001].
| Cohort ID | Unique users | Description | % of total bucketed in group |
|---|---|---|---|
| Users attempting to edit the referral article | |||
| mobile_tutorial_test_control | 1,265 | Users in the control group | 71.1% |
| mobile_tutorial_test_test | 1,034 | Users in the test group | 62.5% |
| Users completing an edit on the referral article | |||
| mobile_tutorial_test_control | 465 | Users in the control group | 26.1% |
| mobile_tutorial_test_test | 572 | Users in the test group | 34.6% |
We also considered the overall main-namespace editing behavior of these users in their first 24 hours since registration.
- When looking at the overall number of users completing a main-namespace edit in their first day (whether on mobile or desktop), test significantly outperformed control with a 3.5% difference in proportion compared to control. [X² = 4.18, N = 3,434, p < .05].
- The difference between test and control was found only marginally significant when we considered mobile edits only and excluded edits these users completed on desktop [X² = 3.47, N = 3,434, p ~ .05].
| Cohort ID | Unique users | Description | % of total bucketed in group |
|---|---|---|---|
| Users with 1+ ns0 edits on mobile or desktop | |||
| mobile_tutorial_test_control | 687 | Users in the control group | 38.6% |
| mobile_tutorial_test_test | 696 | Users in the test group | 42.1% |
| Users with 1+ ns0 edits on mobile only | |||
| mobile_tutorial_test_control | 651 | Users in the control group | 36.6% |
| mobile_tutorial_test_test | 657 | Users in the test group | 39.7% |
Data quality[edit]
See a list of issues and proposed changes in the instrumentation.
Discussion[edit]
In response to the research questions we know: Does displaying a tutorial increase the edit completion rate of new mobile registered users (proportion of users registering via the edit CTA funnel and successfully editing the page they came from)? Yes we saw a statistically significant (8.5%) advantage to showing the edit guider/tutorial in getting users to complete an edit on that page.
Does the tutorial affect the overall activation rate of these users? Yes we saw a statistically significant (3.5%) advantage to showing the edit guider/tutorial in getting users to activate (1+ ns0 edits in 24hrs).
This became a marginally statistically significant (3.5%) advantage when we looked only at completed mobile edits. However, this was not the research question and is more a product of amount of data. It does not change the fact that activation rate for the test group was higher than the control group.
Are there differences in the quality of contributions made by users in each condition, as measured via unreverted edits? There are not statistical differences in the quality of edits. [data needs to be entered]
As a result of this we conclude that the edit guider is a good on boarding strategy for new users. This guider is similar to the strategy employed by the desktop Growth team's OB6 test. From this we can conclude that some combination of reminding the user what they were doing/the context they are in, warning the user about wikitext, and hand holding the user through the workflow; is a strategy for on boarding first time users that is useful on both desktop and mobile for new Wikipedia editors.
This edit guider will be the default experience for new editors going through the edit CTA and a modified version of the guider is being developed for users logging in through the left nav menu, this feature will also be subject to an A/B test.