Research:A meta-method for analyzing NPOV on Wikipedia

Created

03:04, 23 September 2025 (UTC)

Contact

Jacob Rogers

Wikimedia Foundation

Leila Zia

Wikimedia Foundation

Collaborators

Aaron Shaw

Northwestern University

Kaylea Champion

University of Washington Bothell

Pablo Aragon

Wikimedia Foundation

Isaac Johnson

Wikimedia Foundation

Duration:  2025-03 – ??

---

Research:Projects

This page documents a research project in progress.
Information may be incomplete and change as the project progresses.
Please contact the project lead before formally citing or reusing results from this page.

Introduction

Wikipedia is edited an average of 4 times a second by hundreds of thousands of volunteer editors who come to the website every month to create encyclopedic knowledge in more than 300 languages. Their work is guided by principles of iterative improvement, transparency, and consensus building anchored in content policies—including neutral point of view (NPOV)—that they have developed and interpreted, also in an iterative way, over the past quarter of a century. For editors to be able to make iterative decisions and improve the content and the interpretation of policies that govern their work, they need access to timely data and insights. Contentious topics and the implementation of the NPOV policies around contentious topics have posed unique challenges that editors have requested further support around.

On the other hand, Wikipedia is accessed on an even vaster scale: an average of 15 billion times a month. Readers around the world rely on Wikipedia to provide up-to-date, reliable, in-depth, and neutral information about any imaginable topic in a variety of languages. Most readers have never edited Wikipedia in any language on any topic, and may not have the time or inclination to investigate the systems, principles or rules that editors have created to guide and manage their work. When an article on a contentious topic appears to lack balanced perspective, to represent the world in an incorrect way, or simply to overlook some important facts, concerned readers may lose trust in Wikipedia.

This research project aims to support the needs of both editors and readers of Wikipedia in the domain of contentious topics. The primary goal of the project is to develop a rigorous methodology for assessing NPOV on Wikipedia. We will then apply the methodology to one or more contentious topics and demonstrate what one can learn about how Wikipedia handles NPOV by utilizing this method.

Prior research shows that with sufficient time and diversity of editors to contribute to contentious topics Wikipedia's content quality converges to the highest quality. As a result, we see significant opportunity in investing to support editors with rigorous and reliable research. We hope that this research helps them find actionable recommendations or insights that support them in better understanding the implementation of the NPOV policy on Wikipedia and to find ways to improve their work. We further note that the long term sustainability of Wikipedia relies on more readers to gain a more nuanced understanding of the implementation of NPOV on Wikipedia.

Methods

Below we propose a draft meta-method for analyzing the state of NPOV on Wikipedia.

For a given topic t, a set of languages L={L_1, L_2, …, L_m}, and a time period P:

Step 1. Build a superset of entities E = (e_1, e_2,...e_i) relevant to t over all m languages.

1A. For every language L_i, use Wikipedia categories, subcategories, and templates related to t in L_i.

1B. Expand E from 1A using reliable external datasets for each language L_i.

1C. Use appropriate techniques (e.g., Named Entity Recognition, mapping of entities e in E to Wikidata QIDs, etc.) to create a mapping for E across L_m.

1D. Deduplicate, refine, and reduce E as needed to ensure that all e in E are relevant to t, mapped accurately over L_m, and consistent with other constraints.

Step 2. For each language L_m, build corpora of content (e.g., articles, sentences, phrases, sections, sources, …) that are relevant to t in P and identify subsets of the corpora that encompass coverage of any e_i. To do this, we may use the entities from Step 1 to search and retrieve relevant content for a Wikipedia dataset and an external comparator dataset directly. Alternatively, it may be preferable to define a t-relevant corpus of content (on Wikipedia and/or the external comparator) by some other means and then use the entity set to index into this corpus and identify content that is jointly in t and P for L_i. Whatever strategy is applied, we anticipate that some additional selection criteria will need to be developed to ensure that resulting subsets are commensurable, well-scoped for comparison, and amenable to the questions and analysis techniques planned for Step 3.

Step 3. Compare Wikipedia corpora to the comparator corpora to answer focal research questions. Some preliminary and generic examples (neither final versions of research questions nor meant to imply any specific analysis strategy) include :

• For a given Wikipedia language edition L_i and topic t, what is the distribution of the language of sources used in the L_i Wikipedia?
• What proportion of E related to t are covered in Wikipedia L_i versus the corresponding comparator corpus in L_i over the period P?
• What qualities are characteristic of the content covering the subset of E in Wikipedia L_i versus the subset of E in comparator corpus L_i.

Additional and specific comparisons and analyses will be developed. As part of the project development, we also plan to iteratively adapt, pilot, and refine the meta-method for application to example case(s) and selected language editions.

Policy, Ethics and Human Subjects Research

This is not a human subject research. We are developing a meta-method and we will not analyze individual edits or editor activities as part of this research.

Resources

As we share about ongoing research in public venues, we provide the link to these shareouts when available below.