Research:VisualEditor's effect on newly registered editors
One of the key questions about the release of VisualEditor is what effect it will have on newcomer activities. While the visual editor is designed to make editing easier and therefor boost the number of contributions in Wikipedia, it's possible that the new feature may also allow for more damage (e.g. vandalism and sub-par good-faith) to take place.
Our goal in this study is to find out how VisualEditor will affect Wikipedia's means of curating new revisions. We'll perform a small scale experimental release of VisualEditor to a randomly sampled subset of newly registered editors and compare their activity to newcomers who who did not have VisualEditor turned on by default. From the differences observed in this experiment, we'll extrapolate to the scale of Wikipedia.
- 1 Research Questions & Hypotheses
- 1.1 RQ1: Does VisualEditor (VE) make editing easier?
- 1.2 RQ2: How will the release of VE affect the workload of Wikipedians who perform quality control functions?
- 1.3 RQ3: How will the release of VE affect the early productivity of newly registered editors?
- 1.4 RQ4: How will the release of VE affect the retention of good newcomers?
- 2 Methods
- 3 Studies
- 4 Other resources
- 5 See also
Research Questions & Hypotheses
RQ1: Does VisualEditor (VE) make editing easier?
- H1.1: edit time will be reduced for VE user due to the increased ease of editing
- H1.2: edit completion rate will be increased for VE users
RQ2: How will the release of VE affect the workload of Wikipedians who perform quality control functions?
- H2.1: block rate will increase for VE users
- H2.2: revert rate will increase for VE users, but primarily due to good faith mistakes.
- H2.3: Wikipedia's current quality control mechanisms (ClueBot, Huggle, STiki, XLinkBot, etc.) will pick up most of the new work.
RQ3: How will the release of VE affect the early productivity of newly registered editors?
- H3.1: productive edits per user will increase for VE users
- H3.2: time to threshold will decrease for VE users
RQ4: How will the release of VE affect the retention of good newcomers?
- H4.1: short-term survival rate will increase for VE users
R:Newly registered users will be split 50/50 into two conditions:
- control group: no change. The wikitext editor is enabled by default.
- test group: VisualEditor enabled by default (user preference switched on at account creation time)
We'll capture a set of behavior metrics of newcomers to get a sense for how the presence of VisualEditor will effect Wikipedia.
The effectiveness of VE
- edit time: Time taken to complete edit (save time - page load time)
- edit completion rate: Proportion of edits completed
Burden on current Wikipedians
- block rate: # newcomers who edit / # newcomers who are blocked
- reverts per editor: # of reverted edits / # newcomers who edit
- This measures the cost in reverts per editor for each new editor. See this trello card for requirements.
Newcomer productivity and survival
- productive edits per user: productive new editors
- short-term survival rate: surviving new editors
- time spent editing: based on session duration estimation
In this early study of VE, we found that newcomers with VE enabled by default did not get reverted or blocked more often than those in the control. However, we also saw a significant loss in productivity for newcomers with VE enabled. We concluded that reason for the loss in productivity was likely due to the early state of the VE software at the time of the experiment.
Given substantial improvements to the VE software, we re-ran a similar experiment. We found that newcomers with VE enabled by default were reverted significantly less and blocked slightly less often than newcomers with VE disabled. Unlike the previous test, we found no significant difference in the productivity or survival of newcomers between conditions. However, we did find that editors spent more time making edits in VE than in wikitext and suspect that this was due to time spent exploring the new editor.
A follow-up to the May 2015 study which used a larger sample size and a longer study period (60 days after registration). It also found that the visual editor had no effect on newcomer survival or productivity. However, it did find that newcomers in the VE group were reverted somewhat less often than those in the control group.
- August 2015 WMF Metrics Meeting report
- At the WMF Metrics and activities meetings/2015-08 there was a presentation by Halfak about VisualEditor AB test and usability results (see also youtube video):