Research:MoodBar/Email confirmation

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search

MoodBar

Pilot: Early data
(July 2011 - September 2011)
Stage 1: Usage and UX
(September 2011 - May 2012)
Stage 2: Impact on editor engagement
(May 2012 - September 2012)
Created
2012/07
Contact
Giovanni Luca Ciampaglia
Collaborators
Dario Taraborelli
Duration:  2012-07 — 2012-08
Contact:
Giovanni Luca Ciampaglia
Open data project  Open data
no url provided
VisualEditor - Icon - Check.svg
This page documents a completed research project.


Key Personnel[edit]

Project Summary[edit]

This project explores few key statistics about usage of email as a means of notification on the English Wikipedia, with a specific focus on users going through the MoodBar email confirmation funnel.

  • How many registered editors have a confirmed email?
  • How many MoodBar editors/users have a confirmed email?
  • How long does it take editors to confirm their email address?

Methods[edit]

Data have been collected from a replicated instance of the English Wikipedia DB. We collected data from the mw:user_table. Here we focus the analysis only on users registered on the English wiki but our scripts could be reused for other wikis where MoodBar has been deployed.[1] We compare the timestamp of local user registration with the one stored in the mw:CentralAuth database, and excluded users who registered elsewhere by checking if the registration timestamp recorded locally on enwiki is posterior to the timestamp recorded in the CentralAuth database.

When users post an item of feedback using MoodBar they are asked to register an email address, in case they did not have already a confirmed address. Since we want to analyze the effect of the MoodBar funnel on the volume of email confirmations, we focus only on users registered after the deployment of the phase 3 of MoodBar (see MoodBar's status document).

Definitions[edit]

We look at three categories:

  • all users that registered an account within the eligibility window.
  • active users that, additionally to register an account, clicked at least once on the “Edit” button, and hence have the MoodBar extension active.
  • MoodBar users that, additionally to activating the extension, also used it to send at least an item of feedback. Of this group we further distinguish based on the MoodBar-related treatment.
    1. Feedback: users who sent at least a feedback item but never received any answer
    2. Feedback+Response: sent at least a feedback and received at least a response, but did not mark any response as “Helpful”
    3. Feedback+Helpful: sent at least a feedback, received at least a response, and marked at least one response as “Helpful”

The eligibility window for this analysis runs from December 14, 2011, to June 14th, 2012 and it is further subdivided into three periods, as shown in Table 1. Users are assigned to a period based on their registration timestamp.

Table 1: Eligibility window periods
Period Starting date Ending date Description
historical December 14, 2011 00:00:00 May 22, 2012, 23:59:59 MoodBar phase 3
treatment May 23, 2012, 00:00:00 June 13, 2012, 23:59:59 Enhanced tooltip deployed.[2]
control June 14, 2012 00:00:00[3] June 29, 2012 23:59:59 MoodBar blackout

For more information on the meaning of the three periods, see the experimental design document and the results of the controlled experiment.

Metrics[edit]

We report:

  • Proportion of accounts with confirmed email address across: all users, active users (as tracked by the EditPageTracking extension), and MoodBar users, with breakdown by treatment.
  • Average lag between account registration and email confirmation.
  • Percentage of MoodBar users who confirmed the email after posting their first MoodBar feedback item.

Results[edit]

Figure 1
Percentage of confirmed emails among different users categories: active users, all users, and MoodBar users.

Figure 1 shows that the percentage of accounts with confirmed emails is higher among MoodBar users (40%) than in the population of active users (30%) and of all registered accounts (31%). The lag between account registration and email confirmation, varies across the three groups: 30 days for all users, 2.38 days for active users, and 10.5 days for MoodBar users. The higher confirmation lag found among MoodBar users might be a consequence of the possibility to register an email at the end of MoodBar funnel. To test this hypothesis we focus on the sample of MoodBar users. There are 7,789 belonging to this category. Of them, 3,319 confirmed their email after posting their first MoodBar feedback (42%). These results suggest that the MoodBar funnel provides an effective context for asking users to register an email address.

We now focus only on active users (the reference) and active MoodBar users. We looked at the breakdown by treatment type in the three periods. Table 2 reports the results. Confirmed emails are more common among MoodBar users who marked as “Helpful” at least one response to one of their feedback items (Feedback+Helpful, 63% and 58% in the historical and treatment groups. This suggest, albeit with some caution, that email notifications might be an effective tool for learning about responses to feedback items posted with MoodBar.[4]

Table 2: Email confirmation statistics for active and MoodBar users. Gray rows indicate group totals.
period treatment type group size no. confirm. emails % confirm. emails confirmation lag (days) confirm. lag std
historical Active users (Reference) 479437 142775 29.78 1.70 14.77
historical Feedback 7952 2966 37.30 6.73 25.53
historical Feedback+Response 4548 1864 40.99 6.69 24.23
historical Feedback+Helpful 840 533 63.45 11.07 31.10
historical 492777 148138 30.06 1.90 15.34
treatment Active users (Reference) 56512 16540 29.27 0.87 14.55
treatment Feedback 882 283 32.09 1.97 8.70
treatment Feedback+Response 1479 487 32.93 2.18 8.68
treatment Feedback+Helpful 131 76 58.02 2.60 7.26
treatment 59004 17386 29.47 0.93 14.32
control Active users (Reference) 36913 11347 30.74 0.74 3.55
control 36913 11347 30.74 0.74 3.55
Total 588694 176871 30.04 1.73 14.77

References[edit]

  1. The dutch wiki, the french wikisource, and others all have MoodBar deployed, see here and here.
  2. This deployment involved an enhancement of MoodBar's UI, as proposed here, and it was required in order to increase the sample size for our experiment on the effect of MoodBar on user retention.
  3. Users registered in this day are further excluded from the analysis because deployment of the blackout code takes several hours and during this time it is not possible to ascertain whether a user had MoodBar enabled or not.
  4. We lack data on how many users actually saw a response but did not mark it as helpful, which means that in Feedback+Response there are both users who did see the responses and users who didn't.