Research:VisualEditor's effect on newly registered editors

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
VisualEditor analytics Impact on new users Anonymous edits Real-time dashboards
VisualEditor-logo.svg

Project home
Blog
Open data

A/B test docs
A/B test results draft

Descriptive stats draft

Hourly edits by user type, as % of all edits: endeesfrheitnlplrusv
Hourly edits by user type, as % of all edits by user type: endeesfrheitnlplrusv
Hourly edits as % of all edits by user type: endeesfrheitnlplrusv
Hourly edits, absolute number: endeesfrheitnlplrusv
Hourly edits, absolute number by user type: endeesfrheitnlplrusv
Daily unique editors by user type: endeesfrheitnlplrusv
Hourly reverted revisions by revert type: endeesfrheitnlplrusv
All graphs: endeesfrheitnlplrusv

One of the key questions about the release of VisualEditor is what effect it will have on newcomer activities. While the visual editor is designed to make editing easier and therefor boost the number of contributions in Wikipedia, it's possible that the new feature may also allow for more damage (e.g. vandalism and sub-par good-faith) to take place.

Our goal in this study is to find out how VisualEditor will affect Wikipedia's means of curating new revisions. We'll perform a small scale experimental release of VisualEditor to a randomly sampled subset of newly registered editors and compare their activity to newcomers who who did not have VisualEditor turned on by default. From the differences observed in this experiment, we'll extrapolate to the scale of Wikipedia.

Research Questions & Hypotheses[edit]

RQ1: Does VisualEditor (VE) make editing easier?[edit]

  • H1.1: edit time will be reduced for VE user due to the increased ease of editing
  • H1.2: edit completion rate will be increased for VE users

RQ2: How will the release of VE affect the workload of Wikipedians who perform quality control functions?[edit]

  • H2.1: block rate will increase for VE users
  • H2.2: revert rate will increase for VE users, but primarily due to good faith mistakes.
  • H2.3: Wikipedia's current quality control mechanisms (ClueBot, Huggle, STiki, XLinkBot, etc.) will pick up most of the new work.

RQ3: How will the release of VE affect the early productivity of newly registered editors?[edit]

  • H3.1: productive edits per user will increase for VE users
  • H3.2: time to threshold will decrease for VE users

RQ4: How will the release of VE affect the retention of good newcomers?[edit]

  • H4.1: short-term survival rate will increase for VE users

Methods[edit]

R:Newly registered users will be split 50/50 into two conditions:

  • control group: no change. The wikitext editor is enabled by default.
  • test group: VisualEditor enabled by default (user preference switched on at account creation time)

We'll capture a set of behavior metrics of newcomers to get a sense for how the presence of VisualEditor will effect Wikipedia.

The effectiveness of VE[edit]

  • edit time: Time taken to complete edit (save time - page load time)
  • edit completion rate: Proportion of edits completed

Burden on current Wikipedians[edit]

  • block rate: # newcomers who edit / # newcomers who are blocked
  • reverts per editor: # of reverted edits / # newcomers who edit
    • This measures the cost in reverts per editor for each new editor. See this trello card for requirements.

Newcomer productivity and survival[edit]

Studies[edit]

June 2013 study
In this early study of VE, we found that newcomers with VE enabled by default did not get reverted or blocked more often than those in the control. However, we also saw a significant loss in productivity for newcomers with VE enabled. We concluded that reason for the loss in productivity was likely due to the early state of the VE software at the time of the experiment.
May 2015 study
Given substantial improvements to the VE software, we're re-running a similar experiment. We found that newcomers with VE enabled by default were reverted significantly less and blocked slightly less often than newcomers with VE enabled. Unlike the previous test, we found no significant difference in the productivity or survival of newcomers between conditions. However, we did find that editors spent more time making edits in VE than in Wikitext and suspect that this was due to time spent exploring the new editor.
August 2015 WMF Metrics Meeting report
At the WMF Metrics and activities meetings/2015-08 there was a presentation by Halfak about VisualEditor AB test and usability results (see also youtube video):

See also[edit]