Research:VisualEditor's effect on newly registered editors

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
VisualEditor analytics Impact on new users Anonymous edits Real-time dashboards
VisualEditor-logo.svg

Project home
Blog
Open data

A/B test docs
A/B test results draft

Descriptive stats draft

Hourly edits by user type, as % of all edits: endeesfrheitnlplrusv
Hourly edits by user type, as % of all edits by user type: endeesfrheitnlplrusv
Hourly edits as % of all edits by user type: endeesfrheitnlplrusv
Hourly edits, absolute number: endeesfrheitnlplrusv
Hourly edits, absolute number by user type: endeesfrheitnlplrusv
Daily unique editors by user type: endeesfrheitnlplrusv
Hourly reverted revisions by revert type: endeesfrheitnlplrusv
All graphs: endeesfrheitnlplrusv

A draft of the results is available
Gnome globe current event.svg This project page documents a research project currently in progress.
Information may be incomplete and can change rapidly as science advances.
Research project
Visual Editor
Main contact
Aaron Halfaker
Co-investigators
Dario Taraborelli
Oliver Keyes
James Forrester
WMF contact n/a
Start 2013-06
End 2013-07
Status in progress Icon 66 percent.png
Open data This project has published open-licensed data
Open access This project has open access publications
WMF support
Wikimedia research projects Wikimedia research projects

Key Personnel[edit]

Project Summary[edit]

See mw:VisualEditor

One of the key questions about the release of VisualEditor is what effect it will have on newcomer activities. While the visual editor is designed to make editing easier and therefor boost the number of contributions in Wikipedia, it's possible that the new feature may also allow for more damage (e.g. vandalism and sub-par good-faith) to take place.

Our goal in this study is to find out how VisualEditor will affect Wikipedia's means of curating new revisions. We'll perform a small scale experimental release of VisualEditor to a randomly sampled subset of newly registered editors and compare their activity to newcomers who who did not have VisualEditor turned on by default. From the differences observed in this experiment, we'll extrapolate to the scale of Wikipedia.

Research Questions & Hypotheses[edit]

RQ1: Does VisualEditor (VE) make editing easier?[edit]

  • H1.1: edit time will be reduced for VE user due to the increased ease of editing
  • H1.2: edit completion rate will be increased for VE users

RQ2: How will the release of VE affect the workload of Wikipedians who perform quality control functions?[edit]

  • H2.1: block rate will increase for VE users
  • H2.2: revert rate will increase for VE users, but primarily due to good faith mistakes.
  • H2.3: Wikipedia's current quality control mechanisms (ClueBot, Huggle, STiki, XLinkBot, etc.) will pick up most of the new work.

RQ3: How will the release of VE affect the early productivity of newly registered editors?[edit]

  • H3.1: productive edits per user will increase for VE users
  • H3.2: time to threshold will decrease for VE users

RQ4: How will the release of VE affect the retention of good newcomers?[edit]

  • H4.1: short-term survival rate will increase for VE users

Methods[edit]

All new users in English Wikipedia for a period of one week (target: 17-24 June) will be split 50/50 into two conditions:

  • control group: no change. The wikitext editor is enabled by default.
  • test group: VisualEditor enabled by default (user preference switched on at account creation time)

We will use instrumentation of the edit funnel (see Schema:Edit), revert detection and dashboards tracking vandal fighting activity to monitor the impact of Visual Editor on new contributors.

Metrics[edit]

We'll capture a set of behavior metrics of newcomers activities in order to get a sense for how the presence of VisualEditor will effect Wikipedia.

The effectiveness of VE[edit]

  • edit time: Time taken to complete edit (save time - page load time)
  • edit completion rate: Proportion of edits completed

Cost to Wikipedia/Wikipedians[edit]

  • block rate: # newcomers who edit / # newcomers who are blocked
  • reverts per editor: # of reverted edits / # newcomers who edit
    • This measures the cost in reverts per editor for each new editor. See this trello card for requirements.

Benefit to Wikipedia[edit]

  • time to threshold: [n]th edit time - registration time (where [n] is drawn from {1, 5, 10, 25})
  • productive edits per user: # of article edits that are not reverted / # newcomers who edit
  • short-term survival rate: # of editors continuing at edit in their second week / # newcomers who edit

Timeline[edit]

2013-06-13
dry run of split test on enwiki: bucketing logic implemented and deployed to enwiki, all users bucketed in control; URL overrides to test the two conditions.
2013-06-18 20:00:00
end Echo split test; VE test delayed
2013-06-24
start VE split test (bucketing weights changed to activate the split test).
2013-06-25 04:00:00
CAPTCHA bug fixed (users triggering a CAPTCHA, e.g. by adding an external link, were just stranded with no-where to save). User activity after this point is OK for analysis.
2013-07-01 21:00:00
bucketing disabled; VE made default for all existing users, including users previously bucketed in control. Analysis of use behavior begins.

Draft Results[edit]

A draft of the results is available.

External links[edit]