Research:Portal experiments
This page documents a planned research project.
Information may be incomplete and change before the project starts.
As part of improving the user experience for our readers, the Wikimedia Foundation is looking at UX and code changes to the Wikimedia portals - the site elements at www.[project].org, such as the wikipedia.org portal. This page describes the background to the experiments and our planned data collection, and will expand to include the analysis of the experiments themselves once they have launched.
Background[edit]
Portals[edit]
The portals, such as wikipedia.org, have been around in a form fairly similar to their current one for a literal decade. They are currently stored not as MediaWiki UI elements or in versioned code environments, but as templates on Meta.
In UI terms, the portals have not been updated substantially in years, and this shows. Running the desktop and mobile versions through Google's PageSpeed system shows relatively low scores, particularly on mobile, where the massive number of links, combined with their small text size, makes it unappealing and hard to use. Despite this, the portal is heavily utilised. Every day it gets around 10 million pageviews - 1.5-2% of all pageviews to our sites. That's a lot of people to be giving an unoptimised site.
Experimental goals[edit]
The Discovery team at the Wikimedia Foundation wants to improve the portal's UI design. This is not with the goal of manipulating where users go - aiming them towards search, say, or towards the sitelinks - but instead about making sure that they do go somewhere. Our three concerns/areas of improvement are:
- Minimising the amount of time people have to spend on the portal. The portal is a crossroads, not a destination; having to spend a lot of time there identifying where to go does not translate to any benefit for the user.
- Maximising the proportion of portal visits that end with the user clicking through to somewhere else. The portal's goal is to point people to the resource they need: if they're not going anywhere, it isn't doing its job.
- Minimizing a perceived clunkiness to the page by not forcing inline JavaScript elements.
Initial data collection[edit]
Identifying precise targets for these improvements (say, making the search box bigger because people are using search a lot but finding it clunky) is hard to do because we don't have any baseline. We are not currently collecting any data about portal usage, beyond the very basic traffic data mentioned briefly above (the 10m figure).
To change that, we are implementing an EventLogging schema that is aimed at telling us four things:
- What the click rate is for Portal users;
- What people choose to click on when they do, and;
- How long people spend on the Portal before taking an action.
- The user's preferred languages ("accept-language" header).
With this, we can identify how well we're doing with the current interface, and where improvements might best be targeted. We can also design the schema in such a way that it can be trivially used for A/B testing.
Experiments and Analyses[edit]
On 25 March 2016 we published a report of an analysis of how the clickthrough rates and section usage varies by Portal visitors' language preferences. We found that approximately 70% of Wikipedia Portal visitors only have English as their preferred language, and approx. 18% set a language other than English. Around 12% of our users are multilingual (according to their browser preferences), but many of those included English. Users whose primary language is English or who included English as a preferred language clicked through and searched at a remarkably higher rate on a daily basis than users whose primary language is another language (60% vs 50% and 50% vs 30%), but the latter group used the language links at a much higher rate than the former (10% vs 20%). In fact, English-speaking visitors' overall clickthrough rate was 12.5%-14.4% higher than non-English-speaking visitors', and were 1.3 times more likely to click through, indicating, perhaps, that increased localization efforts could better engage our non-English-speaking visitors. Based on the data and patterns observed, we strongly support and encourage the language detection and localization efforts the Portal team has begun pursuing.