Research:VisualEditor's effect on newly registered editors

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
VisualEditor analytics Impact on new users Anonymous edits Real-time dashboards

Project home
Open data

A/B test docs
A/B test results draft

Descriptive stats draft

Hourly edits by user type, as % of all edits: endeesfrheitnlplrusv
Hourly edits by user type, as % of all edits by user type: endeesfrheitnlplrusv
Hourly edits as % of all edits by user type: endeesfrheitnlplrusv
Hourly edits, absolute number: endeesfrheitnlplrusv
Hourly edits, absolute number by user type: endeesfrheitnlplrusv
Daily unique editors by user type: endeesfrheitnlplrusv
Hourly reverted revisions by revert type: endeesfrheitnlplrusv
All graphs: endeesfrheitnlplrusv

A draft of the results is available
Aaron Halfaker
Dario Taraborelli
Oliver Keyes
James Forrester
Duration:  2013-06 — 2013-07
Contact: n/a
Open data project  Open data
no url provided
Open access project  Open access
no url provided
VisualEditor - Icon - Check.svg
This page documents a completed research project.

Key Personnel[edit]

Project Summary[edit]

See mw:VisualEditor

One of the key questions about the release of VisualEditor is what effect it will have on newcomer activities. While the visual editor is designed to make editing easier and therefor boost the number of contributions in Wikipedia, it's possible that the new feature may also allow for more damage (e.g. vandalism and sub-par good-faith) to take place.

Our goal in this study is to find out how VisualEditor will affect Wikipedia's means of curating new revisions. We'll perform a small scale experimental release of VisualEditor to a randomly sampled subset of newly registered editors and compare their activity to newcomers who who did not have VisualEditor turned on by default. From the differences observed in this experiment, we'll extrapolate to the scale of Wikipedia.

Research Questions & Hypotheses[edit]

RQ1: Does VisualEditor (VE) make editing easier?[edit]

  • H1.1: edit time will be reduced for VE user due to the increased ease of editing
  • H1.2: edit completion rate will be increased for VE users

RQ2: How will the release of VE affect the workload of Wikipedians who perform quality control functions?[edit]

  • H2.1: block rate will increase for VE users
  • H2.2: revert rate will increase for VE users, but primarily due to good faith mistakes.
  • H2.3: Wikipedia's current quality control mechanisms (ClueBot, Huggle, STiki, XLinkBot, etc.) will pick up most of the new work.

RQ3: How will the release of VE affect the early productivity of newly registered editors?[edit]

  • H3.1: productive edits per user will increase for VE users
  • H3.2: time to threshold will decrease for VE users

RQ4: How will the release of VE affect the retention of good newcomers?[edit]

  • H4.1: short-term survival rate will increase for VE users


All new users in English Wikipedia for a period of one week (target: 17-24 June) will be split 50/50 into two conditions:

  • control group: no change. The wikitext editor is enabled by default.
  • test group: VisualEditor enabled by default (user preference switched on at account creation time)

We will use instrumentation of the edit funnel (see Schema:Edit), revert detection and dashboards tracking vandal fighting activity to monitor the impact of Visual Editor on new contributors.


We'll capture a set of behavior metrics of newcomers activities in order to get a sense for how the presence of VisualEditor will effect Wikipedia.

The effectiveness of VE[edit]

  • edit time: Time taken to complete edit (save time - page load time)
  • edit completion rate: Proportion of edits completed

Cost to Wikipedia/Wikipedians[edit]

  • block rate: # newcomers who edit / # newcomers who are blocked
  • reverts per editor: # of reverted edits / # newcomers who edit
    • This measures the cost in reverts per editor for each new editor. See this trello card for requirements.

Benefit to Wikipedia[edit]

  • time to threshold: [n]th edit time - registration time (where [n] is drawn from {1, 5, 10, 25})
  • productive edits per user: # of article edits that are not reverted / # newcomers who edit
  • short-term survival rate: # of editors continuing at edit in their second week / # newcomers who edit


dry run of split test on enwiki: bucketing logic implemented and deployed to enwiki, all users bucketed in control; URL overrides to test the two conditions.
2013-06-18 20:00:00
end Echo split test; VE test delayed
start VE split test (bucketing weights changed to activate the split test).
2013-06-25 04:00:00
CAPTCHA bug fixed (users triggering a CAPTCHA, e.g. by adding an external link, were just stranded with no-where to save). User activity after this point is OK for analysis.
2013-07-01 21:00:00
bucketing disabled; VE made default for all existing users, including users previously bucketed in control. Analysis of use behavior begins.

Draft Results[edit]

A draft of the results is available.

External links[edit]