- Aaron Halfaker
- Dario Taraborelli
- Fabrice Florin
- Oliver Keyes
The aim of this project is to evaluate the usage and impact of the new notification infrastructure (also known as Echo) designed by the Wikimedia Foundation and first deployed on the English Wikipedia on May 1, 2013.
Our primary focus is on two sets of questions:
- Who is sending and receiving notifications?
- What is the volume of notifications delivered by type?
- By comparison, the total number of mostly-MediaWiki email notifications sent is about 3.3 billions per year.
- How many notifications are delivered on-site vs other delivery methods?
- What is the relative volume of notifications triggered by bots?
- What is the conversion rate for different types of notification?
- How does the use of private notifications compare to public ones (e.g. talk pages)?
- How do notifications change user behavior?
- Do notifications drive new editor retention and productivity?
- Are there notifications that increase the impact of negative warnings?
- At what rate of emailing do user's react (turning notifications to web only or opting out entirely)?
Data for research on Echo is mostly obtained by instrumenting MediaWiki via EventLogging. We are using three sources of data, described by the following models:
- logs notifications generated by the system and their corresponding delivery method. Notifications logged with the 'email' delivery method indicate that they are staged for delivery to users with validated email addresses and not opted out, not that they have effectively been delivered.
- logs how many email notifications are effectively being sent by MediaWiki, categorized by type (individual, bundled, digest)
- logs user interactions with individual notifications.
We will use this data to measure conversions and other funnel metrics. We will also use cohort analysis (via the UserMetrics API) to run controlled tests and compare how the productivity and retention of different groups of users is affected by notifications.
On top of EventLogging data, we will monitor changes in preferences (user_properties) to determine the timing and volume of opt-outs or changes in default delivery methods.
We also rely on usertesting sessions and short surveys to study usabilty and UX design of this feature.
Privacy and licensing terms