Research:MoodBar/UI design
| Pilot: Early data (July 2011 - September 2011) |
Stage 1: Usage and UX (September 2011 - May 2012) |
Stage 2: Impact on editor engagement (May 2012 - September 2012) |
|
 | This page in a nutshell: This report discusses how design decisions regarding the UI of MoodBar affect its rate of adoption by Wikipedia editors, and what we could do in order to improve it. |
This report discusses how design decisions regarding the UI of MoodBar affect the rate of adoptions of the extension by Wikipedia editors. We discuss findings from other reports and propose actions aimed at increasing the number of newly-registered editors reporting feedback about editing (and the Wikipedia UX in general).
We first review the finding from previous reports and then discuss a certain number of proposals.
Summary of Findings
[edit]- The most effective invitation link text is “Editing Wikipedia made me…”
- The saliency of MoodBar is temporary
We discuss each in a separate section.
The most effective invitation link text is “Editing Wikipedia made me…”
[edit]
Weighted number of feedback by bucket category
The most effective invitation link text was tested by means of A/B testing. Three buckets with the following messages were set up, with the following weights:
- Feedback about editing (weight: 80%, bucket: feedback)
- Editing Wikipedia made me...(weight 10%, bucket: editing)
- Share your experience (weight 10%, bucket: share)
The percentages within parentheses refer to the weight of each bucket. The A/B test with those weights ran between July 25, 2011, and November 30, 2011. It collected a total of 8960 feedbacks divided as:
| bucket | messages |
|---|---|
| editing | 1480 |
| feedback | 6643 |
| share | 846 |
Figure 1 show the total number of messages weighted according to the importance of each bucket. The editing invitation text elicited the highest number of feedbacks per user.
We can actually estimate from the above table the odds of sending feedback under two particular choices; these are just the proportions between the heights of the bars in Figure 1.[1] The results are:
| editing vs feedback | share vs feedback | editing vs share |
|---|---|---|
| 1.785553 | 1.020661 | 1.749409 |
The above table should be read as follows: the first column means that the odds of sending a message given that the user saw the editing invitation text are 1.79 : 1 against those of sending a message given feedback. Thus, we expect that the number of feedbacks should increase by roughly 79% if we switched from feedback to editing.[2]
The saliency of MoodBar is temporary
[edit]
A mock-up of the MoodBar notification tooltip. From the MoodBar design document.
Estimated hazard rate, without tooltip notification.
Estimated hazard rate, with tooltip notification
What could be the cause for this? Upon activation MoodBar notifies the user with a tooltip (see Figure 2). Is it possible that the behavior of users [3] could be attributable to this feature? Luckily there is a way to test this question.
The tooltip feature was in fact introduced a few months after the deployment of MoodBar, in the effort of increasing the saliency of MoodBar within the general Mediawiki UI.[4] Because the tooltip appears only once, all users who activated MoodBar before that date have never seen it, while the majority of the other users saw the tooltip only once.[5] So, if the increased likelihood of reporting feedback is due to the increased saliency of the MoodBar link, for this earlier group we should see a flatter hazard.
We thus divided our dataset of users in two groups, depending on when they had MoodBar activated. The first group (without the tooltip) consists of users who had MoodBar activated before the tooltip notification feature was introduced (9,961 users), while the latter consists of users who had MoodBar activated after this date of 79,744.[6] We estimated again the hazard rate using the same technique, and show them in Figs 2 and 3. A comparison of the two figures confirms our hypothesis: before the introduction of the tooltip the hazard peaks at a lower value, and takes longer to reach the baseline. Moreover, for the pre-tooltip group the baseline hazard after 50 days is higher than in the post-tooltip group.
A note about the pre-tooltip period
[edit]A flatter hazard rate means that moods are sent more uniformly in time, and this is certainly a good thing because we would like to hear from editors also at later stages of their participation into Wikipedia. So, does this mean that before the tooltip we were better off? Luckily for us this is not the case, and here's why.
As can be seen from the MoodBar data dashboard (in particular the second plot), the number of users who sent a mood before the introduction of both the icon and the tooltip is relatively much lower than after. As we can expect, the introduction of the tooltip notification – and to some extent of the icon before it—increase the saliency of the MoodBar link within the MediaWiki UI. However, this has two distinct effects on user adoption: the first is that because more people are aware of the possibility to report their mood, we receive (unsurprisingly!) more feedbacks. The second is that because a one-off notification eventually goes forgotten, the vast majority of the feedbacks we get are sent only in the first few days after the tooltip is shown.
Design Enhancement Proposals
[edit]Invitation text
[edit]The current bucket for all users is feedback. However, the present analysis shows that the odds of sending a message with the invitation text associated to the editing bucket are higher (1.79:1). We thus propose to switch the invitation text accordingly.
Overall MoodBar saliency
[edit]Users tend to report their mood only immediately after the activation of MoodBar; after the first four days since its activation, the chances of sending feedback become in fact tenuous (14x lower hazard on day 10 compared to day 1). We impute this behavior to the effect of the tooltip notification that MoodBar shows upon its activation. A comparison of the hazard rate, before and after the introduction of these features, seems to support this conclusion.
Of course this result has to be read in the general context of the usage of MoodBar. The tooltip notification increases the prominence of the MoodBar UI on the screen and, in fact, since its introduction the number of unique daily posts has increased by roughly 50%.[7] Thus, what our analysis says is that the beneficial effects of the tooltip reminder on user adoption of MoodBar endure only for a few days. Right now only 3% of all users who have had MoodBar activated ever sent feedback.[8] The present analysis suggests that the overall adoption of MoodBar could be improved if, instead of only once, users were reminded of the possibility to send feedbacks multiple times—for example by showing the tooltip after the initial 4 days period.
Notes
[edit]- ↑ More in detail, this amounts to applying Bayes rule of probability using the proportion of the number of feedbacks of a given bucket on the total number of feedbacks as an estimate of the prior probability of that bucket given that a feedback was sent: . The bucket weights are instead .
- ↑ This analysis assumes that the decision to click on the MoodBar link is independent from the decision to actually send a feedback. Because the rest of the MoodBar UI was the same for all buckets, this assumption seems fair.
- ↑ See the report on the Time to first feedback, in particular Figure 5.
- ↑ The actual date of introduction of the tooltip notification is December 14, 2011.
- ↑ MoodBar stores a cookie to prevent from showing the tooltip every time it is loaded. This means that users who do not save cookies across sessions will see it more than once. However, it is safe to assume that this group of users is only a minority of the total population.
- ↑ The first group additionally excludes observations taken after the introduction of the icon next to the MoodBar link, which happened on 2011-11-01. This was needed because we cannot tell whether those users saw the tooltip or not. This means that the increased saliency of the MoodBar UI is also due to this other element.
- ↑ See our MoodBar data dashboard, in particular the second graph.
- ↑ See the graph of the MoodBar data dashboard about this metric.
