Research:Account creation UX

From Meta, a Wikimedia project coordination wiki
(Redirected from Research:ACUX)
Jump to: navigation, search
Research project
Account Creation User Experience
Main contact Growth team
WMF contact Steven Walling
Start 2012-10
End 2013-01
Status completed Icon 100 percent.png
Open access This project has open access publications
WMF support
Wikimedia research projects Wikimedia research projects

This is a project of the Editor Engagement Experiments team at the Wikimedia Foundation.

Goals[edit]

Comparison of new account registrations per hour on the English Wikipedia (2007 vs 2012).

Our goal for account creation UX experiments is to make small, measurable, iterative changes to the account creation process to find what, if anything:

  1. increases account creation completion rates and
  2. increase the number of new accounts that edit

Methodology[edit]

Pre-launch
Permanently disable current A/B test that has been live for months or more with no analysis, bug 37787[1] (links from mw m w) filed. ACIP is partially documented here. Done mid-August 2012
ACUX-1
Test the current signup page against a fully redesigned version. October 4, 2012
ACUX-2
Test of current signup page against redesigned version, with no list of benefits and added error logging. October 18, 2012
ACUX-3
Test of client-side form validation vs control. November 6, 2012
CAPTCHA
Test of removal of the CAPTCHA on overall registration rates. See /CAPTCHA for documentation. January 10, 2013

Metrics[edit]

Account creation click-through rates
Number of users clicking on the submit button per landing page impression
Account creation conversions
Number of accounts successfully created for a given treatment per landing page impression
Active editors
Proportion of users who complete at least one edit
Productive editors
Proportion of users who complete an NS0 edit in 24 hours that is not reverted
User blocks
Proportion of new accounts that go on to be blocked for username-related reasons (as measured by all blocks, filtered by block log comment)
Email authentication rate
Proportion of user accounts per treatment with a verified email address

Experiments[edit]

Methods[edit]

Our first experiment will randomly deliver an account creation landing page to potential new users: 50% will receive the current default as control, with the other 50% receiving the new signup page (full description of differences is here). Users who do not have JavaScript enabled or who are visiting from an unsupported browser will not be bucketed in either group, and will be served the default registration page.

The experiment involves the measurement of conversions, blocked accounts, and user contributions as a function of the treatment received. As of October 2012, there are roughly 4,000 new account registrations per day, therefore in a one week duration we should expect approximately 30,000 data points (15,000 per treatment). The vast majority of these users will be genuine new registrations (a small number of "attached" users, identifiable sockpuppets and bots will be filtered prior to the analysis).

Since client-side form validation is a new addition to MediaWiki account registration, the version deployed in the first iteration will not include it, but rather update the user interface significantly while still relying on the old server-side validation method.

The activation time of this experiment is as follows (all times UTC):

Iteration Start End Bucket IDs Log version
ACUX-1 (new design, server-side validation vs control) 2012-10-04 23:30:00 2012-10-18 23:30:00 acux_1, control_1 v.3
ACUX-2 (new design without list of benefits, server-side validation vs control, high-level error logging) 2012-10-18 23:30:00 2012-11-02 06:28:04 acux_2, control_2 v.4
ACUX-2* (same as ACUX-2, patch for IE introduced, log version bumped) 2012-11-02 06:28:04 2012-11-06 11:30:00 acux_2, control_2 v.5
ACUX-3 (new design, with client-side validation vs control) 2012-11-06 11:30:00 2012-11-23 11:30:00 acux_3, control_3 v.6
ACUX-3* (same as ACUX-3, removed bucketing) 2012-11-29 23:16 deployed on 1.21wmf4 2012-12-03 19:05 (lost when wmf5 deployed to enwiki) acux_3 v.7
ACUX-3* (same as ACUX-3, removed bucketing) 2012-12-04 01:31 re-deployed on wmf5 2013-28-03 acux_3 v.7
CAPTCHA TBD TBD TBD NA

Notes

  • ACUX-1 A temporary interruption of the experiment occurred between 2012-10-08 11:36:00 and 2012-1-09 23:34:43 which effectively disabled the ACUX test for new users clicking on the Create Account link. This resulted in 36 hours of missing client-side events and server-side events with no bucket.
  • ACUX-2 A good faith change in Setup.php that went live on all wikis with 1.21wmf2 [1] produced as an unintended effect the display of the "real name" field on the account creation page of all Wikipedias. This resulted in an abnormal growth of new users (58% of all registrations) with a real name set. We should consider filtering out all these users (N=5,366) registered between 2012-10-22 18:17:02 and 2012-10-24 05:58:56 if we have reasons to believe this change will affect their behavior compared to regular users.
  • ACUX-2. A test of a Signup Call to Action for Article Feedback users was deployed on 2012-10-25 22:10:33. All these users are directed to the ACUX_2 interface and the events they generate include a aftv5_cta4 flag in the additional data field of the clicktracking log as well as in the userbuckets field tracked by server events.
  • ACUX-2 On 2012-11-01 a patch was applied to ACUX that addresses an issue with IE7 and IE8 users not correctly being assigned to ACUX buckets. The bucket identifiers remain the same but the version number in the log has been bumped to 5.
  • ACUX-3 The benefits did not appear as consistently in this edition of the test, with older versions of IE not supporting the method used to dynamically resize them. This caveat should not directly impact the ability of clientside validation to improve the ux, since that feature worked as expected in IE6 and better.
  • ACUX-3 Client-side validation is not perfect, and may produce errors on submit, even though it previously validated as correct. This is due to extensions such as AntiSpoof which don't yet get exposed to the client very well. See bugs 41849 and 40648.
  • ACUX-3 Starting on November 16 we observed a large seasonal spike in registrations, at 5,642 total for the day. This was likely due to an extremely successful fundraising test, which resulted in more than 100,000 individual donations. These donors received an invitation to join Wikipedia and edit in a thank you letter, and this was the largest site-wide change on the day of the spike.

Data collection requirements[edit]

Metrics and data requirements for ACUX first iteration. Events to be logged and corresponding metadata are marked in yellow. Key funnel metrics for this experiment are: CTR (clickthrough rate), CMP (completion rate) and CNV (conversion rate).

We will be logging client and server generated events in order to measure the clickthrough and conversion rates described above. Details on the log format can be found on this page.

Results[edit]

Notes[edit]

In the following cases, registered, logged in editors may unintentionally trigger an event on the account creation page:

  • Users bucketed as part of the Account Creation Improvement Project (ACIP), who are taken through multiple signup screens
  • Users who are browsing Wikipedia in multiple tabs, log in through one, and then attempt to log in through another (may be clicking "Create an account" to log in, possibly because the login/create account links used to be unified and were only recently split)
  • Users entering an invalid password in the signup form

User experience and technical documentation[edit]

See MediaWiki.org

Future work and iteration ideas[edit]

Footnotes[edit]