Research:VisualEditor's effect on newly registered editors: Difference between revisions
+ notice of delay |
→Timeline: Updated for delay and captcha bug |
||
| Line 110: | Line 110: | ||
{{notice|The split test has been delayed for "a few days". This timeline will be updated when a new deployment date is set.}} |
{{notice|The split test has been delayed for "a few days". This timeline will be updated when a new deployment date is set.}} |
||
;<nowiki>2013-06-13</nowiki>: dry run of split test on enwiki: bucketing logic implemented and deployed to enwiki, all users bucketed in control; URL overrides to test the two conditions. |
;<nowiki>2013-06-13</nowiki>: dry run of split test on enwiki: bucketing logic implemented and deployed to enwiki, all users bucketed in control; URL overrides to test the two conditions. |
||
;<nowiki>2013-06-18 20:00:00</nowiki>: [[Research:Notifications/Experiment_1#Timeline|end Echo split test]]; start VE split test (bucketing weights changed to activate the split test). |
;<nowiki>2013-06-18 20:00:00</nowiki>: [[Research:Notifications/Experiment_1#Timeline|end Echo split test]]; VE test delayed |
||
;<nowiki>2013-06-24 </nowiki>: start VE split test (bucketing weights changed to activate the split test). |
|||
;<nowiki>2013-06-25 |
;<nowiki>2013-06-25 04:00:00</nowiki>: Captcha bug fixed. User activity after this point is OK for analysis. |
||
;<nowiki>2013-07-01</nowiki>: bucketing disabled; VE made default for ''all existing users'', including users previously bucketed in control; preliminary write-up of the results from the split test. |
;<nowiki>2013-07-01</nowiki>: bucketing disabled; VE made default for ''all existing users'', including users previously bucketed in control; preliminary write-up of the results from the split test. |
||
Revision as of 18:17, 2 July 2013
This page documents a planned research project.
Information may be incomplete and change before the project starts.
Key Personnel
- Aaron Halfaker (EpochFail)
- Dario Taraborelli (DarTar)
- Oliver Keys (Okeyes (WMF))
- James Forrester (Jdforrester (WMF))
Project Summary
- See mw:VisualEditor
One of the key questions about the release of VisualEditor is what effect it will have on newcomer activities. While the visual editor is designed to make editing easier and therefor boost the number of contributions in Wikipedia, it's possible that the new feature may also allow for more damage (e.g. vandalism and sub-par good-faith) to take place.
Our goal in this study is to find out how VisualEditor will affect Wikipedia's means of curating new revisions. We'll perform a small scale experimental release of VisualEditor to a randomly sampled subset of newly registered editors and compare their activity to newcomers who who did not have VisualEditor turned on by default. From the differences observed in this experiment, we'll extrapolate to the scale of Wikipedia.
Research Questions & Hypotheses
RQ1: Does VisualEditor (VE) make editing easier?
- H1.1: edit time will be reduced for VE user due to the increased ease of editing
- H1.2: edit completion rate will be increased for VE users
RQ2: How will the release of VE affect the workload of Wikipedians who perform quality control functions?
- H2.1: block rate will increase for VE users
- H2.2: revert rate will increase for VE users, but primarily due to good faith mistakes.
- H2.3: Wikipedia's current quality control mechanisms (ClueBot, Huggle, STiki, XLinkBot, etc.) will pick up most of the new work.
RQ3: How will the release of VE affect the early productivity of newly registered editors?
- H3.1: productive edits per user will increase for VE users
- H3.2: time to threshold will decrease for VE users
RQ4: How will the release of VE affect the retention of good newcomers?
- H3.1: short-term survival rate will increase for VE users
Methods
All new users in English Wikipedia for a period of one week (target: 17-24 June) will be split 50/50 into two conditions:
- control group: no change. The wikitext editor is enabled by default.
- test group: VisualEditor enabled by default (user preference switched on at account creation time)
We will use instrumentation of the edit funnel (see Schema:Edit), revert detection and dashboards tracking vandal fighting activity to monitor the impact of Visual Editor on new contributors.
Metrics
We'll capture a set of behavior metrics of newcomers activities in order to get a sense for how the presence of VisualEditor will effect Wikipedia.
The effectiveness of VE
- edit time: Time taken to complete edit (save time - page load time)
- edit completion rate: Proportion of edits completed
Cost to Wikipedia/Wikipedians
- block rate: # newcomers who edit / # newcomers who are blocked
- reverts per editor: # of reverted edits / # newcomers who edit
- This measures the cost in reverts per editor for each new editor. See this trello card for requirements.
Benefit to Wikipedia
- time to threshold: [n]th edit time - registration time (where [n] is drawn from {1, 5, 10, 25})
- productive edits per user: # of article edits that are not reverted / # newcomers who edit
- short-term survival rate: # of editors continuing at edit in their second week / # newcomers who edit
Timeline
 | The split test has been delayed for "a few days". This timeline will be updated when a new deployment date is set. |
- 2013-06-13
- dry run of split test on enwiki: bucketing logic implemented and deployed to enwiki, all users bucketed in control; URL overrides to test the two conditions.
- 2013-06-18 20:00:00
- end Echo split test; VE test delayed
- 2013-06-24
- start VE split test (bucketing weights changed to activate the split test).
- 2013-06-25 04:00:00
- Captcha bug fixed. User activity after this point is OK for analysis.
- 2013-07-01
- bucketing disabled; VE made default for all existing users, including users previously bucketed in control; preliminary write-up of the results from the split test.