Research:Offline Evaluation of Wikipedia Search

Tracked in Phabricator:
Task T414795

Created

08:37, 8 May 2026 (UTC)

Contact

Martin Gerlach

Wikimedia Foundation

Collaborators

Mykola Trokhymovych

University Pompeu Fabra

Duration:  2025-October – 2026-May

---

Research:Projects

This page documents a completed research project.

The Readers Team is working on a project to improve Wikipedia's search in order to make it easier for readers to find answers on Wikipedia. Specifically, the existing lexical search will be complemented with a semantic search model with the idea to better support question-style or exploratory queries even if the words don't perfectly match the title or the exact words in an article.

The goal of this work is to develop a framework for the offline evaluation of the different search models. This will help us provide quantitative evidence for how well these models are working under different conditions and help us improve them iteratively.

The work consists of two main parts:

• Generating a benchmark dataset of queries with annotations of relevant results from Wikipedia articles
• Evaluating different search models using evaluation metrics from information retrieval

Generating a benchmark dataset (WikiQRel)

WikiQRel consists of three core components:

• a curated set of 600 search queries sampled from Wikimedia search logs
• a fixed English Wikipedia paragraph corpus used as the retrieval source,
• human relevance judgments for query-paragraph pairs.

Query collection

We collect queries from the full-text search request logs over the article namespace of English Wikipedia. In particular, we use the full available 90-day log dataset, starting from September 23rd, 2025. The data collection procedure has undergone a privacy review by the Wikimedia Foundation.

First, we apply a sequence of automatic filters to the raw logs in order to remove navigational queries~\cite{broder_taxonomy_2002}, noise (e.g. from automated requests), or personally identifying information (PII). Specifically, we discard queries that are shorter than two words or that exactly match the title of any article (incl. redirects). We then remove queries using advanced search functionality (e.g., logical operators or prefixes), and exclude queries containing emails, links, adult content, or gibberish. Moreover, to remove automated requests, we drop queries from identities that issue more than 1,000 requests per day on average and retain only queries issued by at least 25 distinct identities to lower the privacy risk according to Wikimedia's data publication guidelines.

Second, we define a heuristic to group queries into four different buckets (query types): short (2-3 words), medium (4-7 words), long (8+ words), and natural questions (queries starting with a question word) in order to capture informational queries with different user goals. Within each bucket, we rank queries by a priority score designed to favor queries that are not only frequent but also issued by many different users and apply additional postfiltering to remove near-duplicates or queries reflecting automated log artifacts . After postfiltering, we keep the top 350 queries per bucket and proceed to manual review.

Third, we apply manual filtering to ensure the final query set is safe, high-quality, and analytically valid. Two annotators independently reviewed the candidate queries, following a protocol that excludes queries related to personal information, biographies of living persons, adult or offensive content, navigational or non-encyclopedic intent, and structurally invalid queries . After manual review, we keep the top 150 queries from each bucket, resulting in the final set of 600 queries.

Search corpus

To ensure that relevance judgments remain valid and that future systems can be compared under identical conditions, we fix the search corpus as part of the dataset. We use the Wikimedia Enterprise Dump of Structured Contents dump dated January 25th, 2026. We choose this source because it provides page content parsed from HTML, which is generally preferred over raw wikitext when the goal is to extract the content as seen by the reader. From each article, we extract content at paragraph granularity, drawing paragraphs from body text, tables, and lists, and discarding fragments shorter than 40 characters.

Candidate paragraph retrieval

For each query we identify a set of candidate results using a pooling approach. We first retrieve the top 10 articles from each of two independent search engines, Wikipedia's full text search and Brave Search, restricting results to the main namespace of English Wikipedia and merging them. All searches are performed simultaneously on January 28th, 2026, to ensure maximum consistency with the fixed corpus. Second, we extract all paragraphs from the retrieved articles and score each paragraph's relevance to the query using a pretrained cross-encoder reranker. We select the top 10 paragraphs per query, limiting each page to a maximum of two paragraphs to ensure page diversity, and apply a minimum score threshold of 0.1 to filter out obviously non-relevant items. As a result, we obtain 5,991 query-paragraph pairs that we pass to human relevance annotation.

Human relevance annotation

We collect human relevance judgments using the Prolific platform. Each query-paragraph pair is independently annotated by at least three annotators at collection time, each annotator labels 10 pairs, resulting in 1,841 unique participants. Annotators are shown the query together with the page title, section title, and paragraph text, and are asked whether the paragraph is relevant to answer the query, choosing one of four labels:

• Yes (the information is relevant)
• Somewhat (partially relevant)
• No (not relevant)
• Maybe (I don't know, or I am unsure),

In order to improve the quality of relevance judgments, we filter annotations from the top 10\% of unreliable raters, identified by the average deviation of their scores from the per-sample consensus mean. This improves inter-annotator agreement, measured using Krippendorff's alpha for ordinal values, from 0.402 to 0.524, indicating moderate agreement, which is expected for search relevance, where judgments are inherently subjective.

Finally, we aggregate the individual judgments into a unified relevance score by calculating the mean over the mapped ordinal values (Yes = 2, Somewhat} = 1, No = 0, ignoring Maybe} judgments).

Dataset analysis

Corpus composition

The resulting corpus contains 7,121,961 articles and 54,972,688 paragraphs. The corpus draws paragraphs from different structural elements of each page. Most of the corpus consists of article body text (82.7%); however, a substantial fraction is embedded in other article elements such as tables and lists. The average length of paragraphs is 394.3 characterers with some variation across different types. On average, each article contains 7.7 paragraphs, though half of all articles have four or fewer.

Query characteristics

The dataset includes queries from four different types: short, medium, and long queries as well as natural language questions. The distribution of queries is heavily skewed in the original logs, with short queries far more frequent than longer ones or natural questions . In order to be able to evaluate search systems across the full range of queries (and not just the most common ones), we select a biased sample which contains the same number of queries for each query type.

Frequency and relevance judgements for each query type: S = short, M = medium, L = long, NQ = natural questions. † Queries shorter than 2 words were filtered.

	XS†	S	M	L	NQ
Share in original logs (%)	15	55	25	3	2
# Sampled queries	-	150	150	150	150
# Judged pairs	-	1487	1485	1469	1499
Mean unified score	-	1.03	0.97	0.88	1.19
Unified score ≥ 1 (%)	-	62.1	59.2	53.4	73.1

The selected queries also show patterns specific to information needs of Wikipedia users. For example,  8.5% of queries start with the word ``list'', aimed at Wikipedia's dedicated list pages. We also observe that 6.7% of queries contain the word ``film'' or ``movie'', reflecting the high level of interest in this topic also reported previously. Similarly, 14.8% of queries contain a year, of which 48.3% contain ``2025'', suggesting that users are often searching for up-to-date information.

Relevance Judgments and Coverage

Our dataset contains 15,564 judgments for 5,940 query-paragraph pairs. The distribution of relevance labels is relatively evenly split across the different values (Yes: $32.8%, Somewhat: $37.9%, No: 29.3%). We observe that the mean relevance score differs across query types (long queries: 0.88, natural questions 1.19). Of the 600 queries, only 15 had no paragraph judged relevant (unified score >=1), where the pool returned no relevant candidate at all. Overall, this suggests that most queries are paired with candidate results that are judged relevant or somewhat relevant.

We also demonstrate the value of generating candidates from two independent search engines by confirming that they yield substantially different candidates. We measure engine overlap per query: for each query, we compute the share of its pooled articles, then average it across the queries. On average, 19% of pooled articles come from both engines, 41.6% from Brave only, and 39.4% from Wikipedia only.

Evaluation Results

Experimental setup

We evaluate full-corpus retrieval, i.e. for a given a query, each system retrieves top-10 documents from the fixed corpus, and we score the resulting ranking against the human judgments on two granularities:

• Paragraph-level retrieval tests fine-grained passage ranking: Our benchmark is natively built for paragraph search, so we evaluate rankings directly against the unified per-paragraph relevance score;
• Page-level retrieval reflects the unit that most production search systems, including Wikipedia search, operate on: We define the score of a page as the maximum score over its paragraphs where the page-level relevance label is defined the same way, as the maximum unified score over the page's judged paragraphs.

Search systems

For lexical search we use BM25, a strong sparse baseline that ranks paragraphs by weighted term overlap.

For dense search, we evaluate four compact embedding models:

• Qwen3-Embedding-0.6B
• EmbeddingGemma-300M
• jina-embeddings-v5-text-nano-retrieval
• pplx-embed-v1-0.6B

We retrieve corpus paragraphs by cosine similarity to the query embeddings.

As a production system, we use Wikipedia search currently deployed in English Wikipedia. It is a lexical retrieval system highly customized for Wikipedia purposes. Since it indexes articles rather than paragraphs, we evaluate it at the page level only.

Metrics

The primary metric we use for comparison is NDCG@10, the main metric for the retrieval task in the MTEB benchmark. We also report Bpref as an alternative metric which is more robust to incomplete judgments. Both metrics are calculated using pytrec_eval.

We report each metric per query type and two granularities (paragraph-level and page-level). We report the total score (Tot) as the macro-average over all 600 queries, giving each query equal weight. We also report Tot_w, which reweights the four query types by their empirical shares in the search logs , so the aggregate reflects the real traffic a deployed system would see.

Results

The full results are presented in the following table.

Retrieval results on WikiQRel, per query type, with the macro-average (Tot) and traffic-weighted average (Tot_w) with its bootstrap 95% confidence interval. Paragraph scores are aggregated to pages by max-pooling; Wikipedia search uses its native article ranking and has no paragraph-level result.

Metric	System	Paragraph-level						Page-level
		S	M	L	NQ	Tot	Totw	S	M	L	NQ	Tot	Totw
NDCG@10	BM25	.189 ± .034	.219 ± .038	.247 ± .046	.033 ± .016	.172 ± .019	.196 ± .024	.313 ± .048	.396 ± .048	.404 ± .051	.130 ± .029	.311 ± .023	.336 ± .034
	Qwen3-Emb-0.6B	.283 ± .036	.306 ± .037	.285 ± .043	.386 ± .034	.315 ± .018	.292 ± .026	.483 ± .045	.528 ± .039	.478 ± .048	.583 ± .031	.518 ± .021	.498 ± .032
	EmbeddingGemma	.328 ± .044	.317 ± .041	.317 ± .041	.456 ± .034	.355 ± .020	.327 ± .030	.548 ± .051	.536 ± .049	.517 ± .047	.665 ± .029	.566 ± .022	.546 ± .036
	jina-v5-nano	.326 ± .042	.345 ± .038	.327 ± .040	.458 ± .034	.364 ± .019	.334 ± .029	.548 ± .047	.575 ± .041	.533 ± .045	.649 ± .026	.576 ± .021	.557 ± .033
	pplx-embed-v1-0.6B	.279 ± .036	.299 ± .039	.296 ± .043	.447 ± .037	.331 ± .019	.290 ± .026	.485 ± .048	.561 ± .044	.510 ± .048	.620 ± .029	.544 ± .021	.511 ± .034
	Wikipedia search	--	--	--	--	--	--	.602 ± .044	.580 ± .044	.484 ± .053	.448 ± .033	.528 ± .022	.587 ± .031
BPREF	BM25	.200 ± .042	.230 ± .039	.279 ± .052	.031 ± .016	.185 ± .021	.208 ± .029	.322 ± .055	.397 ± .054	.389 ± .055	.131 ± .030	.310 ± .025	.342 ± .037
	Qwen3-Emb-0.6B	.272 ± .042	.294 ± .039	.281 ± .047	.385 ± .036	.308 ± .021	.282 ± .030	.438 ± .052	.469 ± .044	.431 ± .053	.534 ± .037	.468 ± .025	.449 ± .039
	EmbeddingGemma	.316 ± .047	.319 ± .042	.338 ± .049	.421 ± .035	.348 ± .022	.320 ± .033	.502 ± .057	.477 ± .051	.485 ± .050	.598 ± .039	.516 ± .026	.496 ± .042
	jina-v5-nano	.305 ± .046	.323 ± .040	.350 ± .049	.435 ± .036	.353 ± .022	.315 ± .032	.499 ± .054	.505 ± .047	.501 ± .053	.601 ± .035	.527 ± .025	.503 ± .040
	pplx-embed-v1-0.6B	.308 ± .047	.307 ± .042	.325 ± .052	.421 ± .036	.340 ± .022	.311 ± .033	.482 ± .053	.495 ± .050	.479 ± .054	.555 ± .038	.503 ± .026	.487 ± .040
	Wikipedia search	--	--	--	--	--	--	.561 ± .053	.502 ± .052	.428 ± .057	.347 ± .033	.459 ± .026	.534 ± .038

Dense models clearly outperform plain BM25 with the gap decreasing as queries become longer. Surprisingly, BM25 performs substantially worse than dense retrieval for natural questions. This observation is in contrast to previous experiments which found that dense retrieval did not outperform BM25 for Q&A datasets such as MIRACL. This could be due to the fact that the natural questions in our dataset are much shorter (~3 words on average for our dataset vs. ~8 for MIRACL). Among the dense models, jina-v5-nano show the best results overall, but with EmbeddingGemma-300M showing a slight advantage for short queries.

As expected, page-level scores are higher than paragraph-level ones, since locating a relevant page is easier than ranking the exact paragraph. At the same time, we conclude that relative ordering of models is largely the same for both granularities (except for Wikipedia search which only provides results on the article level).

Notably, for page-level retrieval, Wikipedia search performs better or on-par with dense models for short and medium queries, but falls behind dense models on long queries and, more strongly, on natural questions. It is interesting to note that the performance of the two lexical approaches, Wikipedia Search and BM25, differs qualitatively: the former yields more relevant results for short queries and vice versa. Given the fact that the majority of queries are short, Wikipedia search yields the best result on overall search traffic (Tot_w) remaining competitive with the best dense model.

The results are largely robust with respect to the considered metric. The strength of Wikipedia search at the page level is therefore not an artifact of incomplete judgments or of its contributing to the annotation pool. At the paragraph level, both metrics agree that BM25 significantly underperforms all dense models, but the ranking among the dense models themselves is close enough that the differences are not significant given their overlapping bootstrap confidence intervals.

For comparison with an existing benchmark, we show retrieval performance on the MIRACL dataset.

Retrieval results on MIRACL-en dev (NDCG@10). All 95% bootstrap CI half-widths are between 0.020 and 0.024.

System	Paragraph	Page
BM25	TBD	TBD
Qwen3-Emb-0.6B	.475	.707
EmbeddingGemma	.536	.737
jina-v5-nano	.510	.717
pplx-embed-v1-0.6B	.535	.736

Overall, the absolute numbers in MIRACL are substantially higher, suggesting that WikiQRel constitutes a more challenging benchmark, especially for queries that are not natural questions. Specifically, for page level retrieval, MIRACL yields NDCG@10>0.7 and its scores do not statistically significantly discriminate models' performance. Further, the benchmarks lead to a substantially different ranking of the dense models, which indicates that WikiQRel captures different aspects of how a model performs in retrieval.

Resources

• Dataset: tba
• Code: tba
• Paper: tba