Research:VisualEditor's effect on newly registered editors
This page documents a planned research project.
Information may be incomplete and change before the project starts.
Key Personnel
- Aaron Halfaker (EpochFail)
- Dario Taraborelli (DarTar)
- Oliver Keys (Ironholds)
- James Forrester (Jdforrester_(WMF))
Project Summary
- See mw:VisualEditor
One of the key questions about the release of VisualEditor is what effect it will have on newcomer activities. While the visual editor is designed to make editing easier and therefor boost the number of contributions in Wikipedia, it's possible that the new feature may also allow for more damage (e.g. vandalism and sub-par good-faith) to take place.
Our goal in this study is to find out how VisualEditor will affect Wikipedia's means of curating new revisions. We'll perform a small scale experimental release of VisualEditor to a randomly sampled subset of newly registered editors and compare their activity to newcomers who who did not have VisualEditor turned on by default. From the differences observed in this experiment, we'll extrapolate to the scale of Wikipedia.
Research Questions & Hypotheses
RQ1: Does the Visual Editor make editing easier?
- H1.1: edit time will be reduce for VE user due to the increased ease of editing
- H1.2: edit completion rate will be increased for VE users
RQ2: How will the release of VE affect the workload of Wikipedians who perform quality control functions?
- H2.1: block rate will increase for VE users
- H2.2: revert rate will increase for VE users, but primarily due to good faith mistakes.
- H2.3: Wikipedia's current quality control mechanisms (ClueBot, Huggle, STiki, XLinkBot, etc.) will pick up most of the new work.
RQ3: How will the release of VE affect the early productivity of newly registered editors?
- H3.1: productive edits per user will increase for VE users
- H3.2: time to threshold will decrease for VE users
RQ4: How will the release of VE affect the retention of good newcomers?
- H3.1: short-term survival rate will increase for VE users
Methods
All new users in English Wikipedia for a period of one week (target: 17-24 June) will be split 50/50 into two conditions:
- control group: no change. The wikitext editor is enabled by default.
- test group: VisualEditor enabled by default (user preference switched on at account creation time)
Metrics
We'll capture a set of behavior metrics of newcomers activities in order to get a sense for how the presence of VisualEditor will effect Wikipedia.
The effectiveness of VE
- edit time: Time taken to complete edit (save time - page load time)
- edit completion rate: Proportion of edits completed
Cost to Wikipedia/Wikipedians
- block rate: # newcomers who edit / # newcomers who are blocked
- reverts per editor: # of reverted edits / # newcomers who edit
- This measures the cost in reverts per editor for each new editor.
 | The way in which the revert was performed is important to know. ClueBot is essentially free, Huggle is cheap, but manual reverts are expensive. There are many reasons for reverts and blocks. Will we determine whether the revert/block was due to actual bad-faith edits or good-faith mistakes? |
Benefit to Wikipedia
- time to threshold: [n]th edit time - registration time (where [n] is drawn from {1, 5, 10, 25})
- productive edits per user: # of article edits that are not reverted / # newcomers who edit
- short-term survival rate: # of editors continuing at edit in their second week / # newcomers who edit
Timeline
- June 6-17th: Development of experimental framework:
- June 17-24th: Experiment deployment
- June 17-31st: Observation peroid
- July 2nd: Priminary writeup