List of experiments in Product and Technology/Archive

Experiments in Product & Technology

• Overview
• FAQ
• Archive

Archive

This is an archive of experiments from the Product and Technology department that have already reached a decision.

Experiment	Team	Experiment Timeline	Experiment Type	Audience	Problem to solve / why we are doing this	Outcome
Mobile page previews	Reader Growth	April 20 - May 18	A/B test	10% of logged-out readers on Arabic, French, Italian, Polish, and Vietnamese Wikipedias and 0.1% of logged-out readers on English Wikipedia	We want to test Mobile Page Previews — a pop-up bottom sheet that appears when a reader taps a blue link, showing the thumbnail, lead paragraph, and an option to open the article — on mobile web article pages. Page Previews already exist on the Wikipedia desktop experience and both mobile apps.	Stopped: Based on flat retention and negative indicator metrics, our hypothesis was not supported, and readers on mobile web did not prefer Page Previews over navigating directly to blue-linked articles.
Redesign account creation form on mobile web Redesign the mobile web account creation form to follow Codex guidelines and reduce visual and cognitive complexity. The updated design shortens explanatory text, improves layout, and allows users to view all required fields on a typical mobile screen without scrolling.	Contributor Growth	Start Date: 2026-04-13 End Date: 2026-06-27	A/B test	10% of traffic to auth.wikimedia.org	As a mobile web user, I want a cleaner and less verbose account creation form so that I can view all required fields at a glance without scrolling, and complete registration with reduced information overload.	Ship it! The results were clear and positive. Users who saw the redesigned form created permanent accounts at a rate of 20.4%, compared to 16.4% in the control group. That works out to a 24.4% relative increase in the account creation rate, far exceeding our hypothesis goal of a 2% increase, and the experiment reached a 100% chance of winning. The 95% confidence interval for the increase is entirely positive, ranging from 13.7% to 35.1%. In other words, while our best estimate of the lift is 24.4%, even the most conservative end of the plausible range is a 13.7% increase, still well above our 2% goal. This gives us strong confidence that the improvement is real rather than a product of chance.
Add user account button to header on mobile web Improve the discoverability of account creation on mobile web.	Contributor Growth	Start Date: 2026-03-31 End Date: 2026-06-28	A/B test	10% of users on Hindi, Indonesian, Bengali, Thai, and Hebrew Wikipedias	Account creation is not easily discoverable on mobile web. Users must open the hamburger menu to find "Log in," then close the auto-focused keyboard on the login page, and scroll to the bottom to discover the "Create account" option. A analysis of the top 20 most visited websites shows that nearly all platforms display a way to create a account directly from the homepage (T410558). We think this will significantly improves discoverability and reduces friction in getting readers to create accounts.	Ship it! We observed that adding a user account button to the mobile web header increased permanent account creation by approximately 20.2% relative to the control group, substantially exceeding the hypothesis target of a 2% increase. As a result of this work, we will roll out the feature to all wikis on mobile web through T428220: Scale Mobile Account Menu to All WMF Wikis
Improve the logged-out warning message on mobile web Reduce the sense of friction or alarm when users encounter the logged-out warning upon editing, and highlight the benefits of creating an account (e.g., edit attribution, watchlist, communication tools, and reputation building).	Contributor Growth	March – June 2026	A/B test	10% of users on Arabic, French, Spanish, German, Russian, Chinese, Italian, Portuguese, Persian, and Polish Wikipedias 0.1 % of users on English Wikipedia	When users attempt to edit while logged out, they encounter a jarring warning message that emphasizes editing as an IP or temporary account. This interface can feel abrupt and discouraging, and it frames logged-out editing as the primary path rather than encouraging users to create an account. There is an opportunity to soften the logged-out warning message and guide users toward account creation in a more supportive, motivating way. By improving the tone and visual hierarchy, we can better communicate the benefits of creating an account and make the overall experience less intimidating.	Ship it! We observed a 30.6% relative increase in account creation among exposed users. This intervention significantly exceeded the original goal of increasing account creation by a relative 2% among exposed users. We will scale the improved logged-out edit warning message to all Wikimedia wikis (T424595)
Revise Tone structured task Support newcomers by introducing a Suggested Edit that focuses on identifying and improving non-neutral language in articles. [includes a machine learning model]	Contributor Growth	Start Date: 2026-03-31 End Date: 2026-04-16	A/B Test	50% of users on English, French, Arabic, and Portuguese Wikipedias	Writing in a neutral tone is a pillar of Wikipedia, but is a practice many new volunteers find to be unintuitive. An October 2024 analysis of English Wikipedia found that 56% of new content edits from newer volunteers contained peacock words, with 22% of these edits being reverted. Revise tone structured task is a way that we can help identify and improve non-neutral language in articles, alongside other implementations of the tone check model.	Ship it! We observed that task completion rates for Revise Tone were more than 30% higher than for Copyedit, demonstrating that a more guided and structured workflow can improve newcomer participation and task completion. We will scale the Revise Tone task to additional wikis where the Tone Check model is already supported (T426364 ) and explore expanding the Tone Check model to additional languages.
A/A test on retention rate for logged-in readers baseline Establish a baseline retention rate for logged-in readers.	Reader Experience	March 24 to May 30, 2026	A/A test	Randomized users across top 50 trafficked wikis	We do not currently have an existing measurement of user retention rate for logged-in Wikipedia readers. This test is intended to etablish a baseline that will be useful in helping us set key result goals for improving logged-in and active user retention on web next fiscal year.	Logged in Reader Retention baseline established. See Logged-in Reader Retention Baseline (Users with 0 edits on top 50 Wikipedias): 2nd-Day Retention: 21.34% 2nd-Week Retention: 46.35% 21-Day Cumulative Retention: 61.93% We will Use the newly established definition and baseline to evaluate future logged-in reader experiments and product interventions
Mobile Table of Contents Offer a table of contents on mobile web.	Reader Growth	March - April 2026	A/B/C test	Mobile web users on Arabic, French, English, Indonesian, and Vietnamese Wikipedias	Readers struggle to navigate our mobile interface without wayfinding help. We want to explore whether a table of contents supports readability.	Dont Ship: Readers get overwhelmed and have difficulty navigating long text blocks. The ToC buttons in our treatments did not improve wayfinding enough to mitigate the overwhelm observed in our prior Mobile Expanded Sections experiment, so we saw the same outcome of longer sessions due to more friction, resulting in lower retention
Reader → Contributor Attribution Baseline Experiment Measure how many readers create contributor accounts and see whether the rate differs depending on how people came to the site (by referrer)	Various teams involved in Key Result WE5.3	March 22nd – April 2nd	Invisible (no discernable UI change)	Sampled anonymous readers across Wikipedia projects, sampling the top 50 largest Wikipedias	WMF would like to understand how different types of readers become contributors. This experiment puts additional code on existing interfaces that tracks how many readers become contributors across a variety of characteristics (like which device a person uses, how they found the site, etc.)The goal is to test the code to see if such experiments can be carried out going forward and gain insight into how readers become contributors. The data gathered from this experiment will be aggregated or discarded once the experiment is complete, in accordance with the data collection policies.	We need to learn more The results of the 7-day experiment are promising and directionally helpful for understanding downstream outcomes by referrer, however, we need to launch a larger sampled experiment to get conclusive results. We're working towards launching a larger experiment.
Hybrid search Show richer results to readers when they search for something in natural language	Apps	February – April 2026	A/B test	Android app users on Greek, English, French, and Portuguese Wikipedias	Readers struggle to find specific information on Wikipedia using internal search, and hybrid search (keyword search plus semantic search) shows potential for improving navigation and wayfinding.	Proceed to Phase 2: Based on Phase 1 results, the experiment showed strong potential for improving search and navigation. The team is moving forward to Phase 2 (build for release).
Activity Tab Transforming the History tab into a richer Activity experience that highlights reading, editing, and contribution insights.	Apps	2025 – early 2026	A/B test and scaling	Wikipedia app users (iOS, logged-in and logged-out)	The team aimed to increase engagement, retention, and account creation by helping users better understand and interact with their activity across reading, editing, and contributions.	Ship it! — The Activity Tab showed strong impact: Account creation increased by 91% after scaling (vs. 5% goal) 25% of all new accounts were created through the Activity Tab New users engaged meaningfully: 18.5% saved an article and 2.9% made an edit Users who accessed the Activity Tab returned at higher rates (93.8% logged-in, 91.8% logged-out vs. lower baseline rates) Only 2.4% of users turned the feature off The feature has been scaled and continues to be improved, with focus on onboarding clarity and navigation.
Improve verification email Simplify the email that asks users to confirm their email	Growth	2025 Dec - 2026 Jan	Release to pilot wikis	en.wiki, Wikidata, Commons	The current verification (confirmation) email can be perceived as alarming, especially with language about the user's IP address. This may deter users from confirming their email or engaging further. We aim to revise this email to make it more welcoming and informative, improving the overall onboarding experience and increasing email confirmation rates.	Ship it! Improved verification email has been released to all wikis (T416748). Email verification rates increased across all three pilot wikis (English Wikipedia, Commons, Wikidata) in both month-over-month and year-over-year comparisons. Same-week verification rate rose from 41.6% to 45.9%, a 4.3% absolute increase (~10% relative improvement). Observed largest gains on Commons; improvements are consistent across both app and non-app registrations.
Section editing dead-end on mobile web Allow editors to navigate between sections in edit mode on VisualEditor in mobile	Editing	2025 Dec - 2026 Jan	A/B test	10% of mobile web edits on the 100 largest Wikipedias, 0.1% for en.wiki	This feature is intended to make it easier for people on mobile web to edit any article section, regardless of which section-edit icon they tapped to begin.	Ship it! Full-page editing affordance on mobile web has been released to all Wikipedias. 2.6% increase in constructive edits for logged-out users.
Expanded mobile sections Testing autoexpanded mobile web sections.	Reader Growth	2025 Dec	A/B Test	10% of mobile web readers on Arabic, Chinese, French, Indonesian, Vietnamese	Sections on mobile web are currently collapsed by default, which makes browsing difficult.	Don't ship. Instead, run another experiment including a mobile Table of Contents with expanded sections.
Reference Check	Editing	2025 Nov-Dec	A/B test	38% of new content edits on English Wikipedia	If you add a large amount of text without any citations, the reference check will remind you to add a citation. This only works for edits made using the visual editor. This check is shown when you click on the Publish button. Introduced in late 2023.	Ship it! Reference Check has been enabled for all newcomers on English Wikipedia (T367343), and is now available to all Wikipedias. When Reference Check was shown, edits were far more likely to add a reference or acknowledge/explain why they did not: Mobile web: ~17.5× more likely (2.8% → 48.9%) Desktop: ~2.2× more likely (30.7% → 68.2%) We observed the following impact on constructive edits (not reverted within 48 hours) Mobile web: 56.4% → 66.7% (+18.2% relative) Desktop: 75.5% → 77.9% (+3.2% relative)
Image browsing A feature that will make discovery of images and videos easier on Wikipedia.	Reader Growth	2025 Nov–Dec	A/B Test	10% of mobile web readers on Arabic, Chinese, French, Indonesian, Vietnamese; 0.1% on English Wikipedia	In WMF's surveys of global internet users, “more images/photos” is the most asked-for improvement to Wikipedia - and by creating richer, more visual experiences, we aim to attract new readers and encourage them to return.	We need to learn more: Do another experiment focused on image carousel and default full-screen display, with aim to gather more data before rollout.
Reading lists Allow readers to save articles to a list for reading later.	Reader Experience	2025 Nov–Dec	A/B Test	Logged-in users with zero edits on Arabic, Chinese, English, French, Indonesian, Vietnamese	We aim to learn whether offering curation options helps readers engage more deeply with Wikipedia - and ultimately stay longer and return more often.	Ship it: Start with a beta rollout release, with the aim of eventually shipping globally.
hCaptcha A/B Test Test a bot detection service (hCaptcha) as a possible replacement for the current CAPTCHA system.	Product Safety and Integrity	2025 Oct – 2026 Jan	A/B Test	French Wikipedia: 10%	The current CAPTCHA system is not equipped to defend against modern malicious automated activity. If we prove the usability of the new service, we’ll be able to better protect Wikimedia projects from malicious bots while potentially improving usability and accessibility.	Ship it: We plan to expand hCaptcha to more editing interfaces and more wikis. In the A/B test, the rate of completing the account creation process went up by ~10% and we observed a 31% reduction in block rates for user accounts where hCaptcha was involved, indicating that automated bad actors were more likely to fail in registering accounts when they encountered hCaptcha compared to FancyCaptcha.
Wikipedia Year in Review	Apps	October 11 – December 31, 2025 (with release in early December)	A/B testing (with different variations on Android and iOS)	Wikipedia app users (readers, editors and donors)	The apps team wants to help users feel more connected to Wikipedia by showing them their reading habits and activity. This can make them more engaged, come back more often, and even encourage donations.	Ship it! — The feature showed strong engagement: Many users viewed multiple slides Users shared their results on social media It increased donations compared to the previous year
Paste Check	Editing	2025 Oct-Dec	A/B test	50% of users editing a desktop or mobile main namespace page using Visual Editor on 22 Wikipedias	This check is an effort to increase the likelihood that the new content people are adding to Wikipedia is aligned with the Movement’s commitment to offering information under a free content license.	Ship it! Paste Check is now available to all Wikipedias (T405127). We observed a -18% relative decrease [10.5% → 8.6%] in the revert rate for edits shown Paste Check compared to edits eligible but not shown Paste Check. However, the data is insufficient to statistically confirm that this effect was not due to random noise.
Tone Check Show new editors a warning when their writing is considered to be NPOV by a transformer model [includes a machine learning model]	Editing	2025 Sept-Jan	A/B test	0.1% of edit attempts eligible for Tone Check on fr.wiki, ja.wiki, pt.wiki (limited to edits by unregistered users and users with 100 or fewer edits)	Tone Check is meant to address two core issues: Newcomers publishing edits to Wikipedia that contain promotional, derogatory, or otherwise subjective language because they lack the awareness that this kind of editing is not aligned with Wikipedia policies. Experienced volunteers being burdened by the effort and attention they need to allocate towards patrolling and reviewing preventable damage made in good faith. This can come at the expense of identifying and addressing more subtle and complex forms of vandalism.	Ship it! Tone Check will be released to all Wikipedias (T416406) and included in Suggestion Mode. Overall: Tone Check improved the rate of constructive edits by +6.2% [4.4] percentage points. We observed improvements in overall edit quality at each of the three partner Wikipedias. Platform: on desktop, constructive edit rate increased by +6.4% while we observed no statistically significant change in mobile web constructive edits. Experience level: Tone Check appears especially effective at increasing the constructive edit rate of a registered Junior Contributors, where we observed a +14.8% increase [10.2 pp] in constructive edit rates.
Donate Button The donate button will be changed to a more eye-catching design.	Readers	2025 Aug–Sep	A/B Test	English Wikipedia: 0.1% — Test Wikipedia: 10%	Through this experiment, we want to see whether changing the “Donate” button entry point in English Wikipedia would encourage more clickthroughs from logged-out users and ultimately increase the number of donations.	Ship it! — Proposal to scale to all users. More information
Watchlist: Group By Toggle The "group results by page" feature will be shown more prominently for a subset of contributors.	Moderator Tools	2025 Aug–Sep	A/B Test	Greek, Bengali, Urdu, Czech, Portuguese, Chinese, French Wikipedias: 100%	We want to improve feature discoverability and make it easier and faster for contributors to locate the edits they are interested in, and as a side effect, motivate their return.	Discontinued due to inconclusive results.
"Which came first?" Game	Apps	May 2025 (A/B test for 20 days)	A/B Test	Wikipedia Android app users (mainly logged-out readers)	The team wants to increase engagement and retention by offering a fun, daily game that encourages users to come back to the app and explore more content.	Ship it! Users who played the game returned 7.3% more 71.4% of users who started the game completed it 10% engaged further (shared score, read, or saved articles) 169 app installs came from shared game links 95% of users were satisfied or neutral