Research:Assessing Wikimedia Research Reproducibility: A Study on Collaborative Practices
This page documents a proposed research project.
Information may be incomplete and may change before the project starts.
Reproducibility underpins scientific credibility. The Wikimedia movement advocates open knowledge, yet the reproducibility of its research outputs has not been evaluated. This project applies a two-phase mixed-methods design. Phase I surveys peer-reviewed publications and documentation (including preprints and white papers) related to Wikimedia, published between 2010 and 2025. A structured reproducibility checklist is applied to a corpus derived from bibliographic databases, generating standardized z-scores for each publication. Phase II uses these scores to draw a stratified random sample, from which corresponding authors are selected for semi-structured interviews on research practices and the influence of their work on the Wikimedia community. Outcomes include a reproducibility package with protocols, data, and code; one peer-reviewed article; and a white paper with recommendations and dissemination strategies. The project engages the Wikimedia research community and connects with funded initiatives such as bibliometric mapping by Buttliere et al. (2025). It aims to reinforce the epistemological basis of research about the Wikimedia movement.
Methods
[edit]This project assesses reproducibility indicators: the presence, completeness, and accessibility of artifacts (data, code, environment specifications, protocols, and licensing) that would make replication possible in principle. It does not attempt to replicate or re-execute any of the studies in the corpus. Replication of a single computational study typically requires multiple weeks and falls beyond the resource constraints of this grant. The checklist applied in Phase I operationalizes reproducibility as an ordinal, artifact-based construct, consistent with the Transparency and Openness Promotion (TOP) Guidelines (Nosek et al., 2015) and the Replication Recipe framework (Brandt et al., 2014).
Phase I. Bibliographic Reproducibility Analysis
[edit]The corpus comes from four sources: (1) indexed databases (e.g., Web of Science, Scopus, PubMed, IEEE Xplore) searched using Wikimedia terms; (2) open repositories such as arXiv, Zenodo, and OSF Preprints; (3) the bibliometric corpus compiled by Buttliere et al. (2025), comprising approximately 9,000 OpenAlex-indexed publications, which serves as the starting point for our search and will be verified, updated, and supplemented with records from Wikidata via SPARQL; and (4) reference lists from systematic reviews on Wikimedia research published after 2015. The team will resolve disagreements by adjudication and will remove duplicate records. Inclusion criteria encompass all empirical or data-oriented studies, both quantitative and qualitative. Theoretical or strictly conceptual studies are excluded. Two independent raters will screen titles and abstracts for relevance. Inter-rater agreement is assessed using Cohen's kappa.
Full-text retrieval follows a tiered access strategy. Bibliographic metadata is harvested primarily through OpenAlex and other structured databases. For full-text download, Scrape and Unpaywall retrieve open-access articles. Subscription-based articles are accessed through team members' institutional affiliations and, where available, through the Wikipedia Library. If institutional access is unavailable, corresponding authors are contacted with a reprint request. Only openly accessible articles are included in the reproducibility checklist analysis; articles that remain inaccessible after all retrieval attempts are recorded as such and excluded from scoring.
Given an estimated corpus of up to 10,000 records after full search and deduplication, the working sample is defined on capacity grounds. With an estimated coding time of 25 to 40 minutes per article, approximately 150 to 400 articles can be reliably assessed. The lower bound of 150 articles corresponds to roughly 1.5% of the estimated corpus, sufficient for descriptive and inferential analysis given the homogeneity expected within disciplinary strata. Sample selection is stratified by publication year to ensure temporal representativeness across the 2010 to 2025 period. All research projects funded by the Wikimedia Research Fund are included as a priority stratum, regardless of z-score, given their direct relevance to the funder and the availability of associated Meta pages.
Reproducibility Checklist
[edit]Each publication is assessed with a reproducibility checklist that adapts the TOP Guidelines (Nosek et al., 2015) and the Replication Recipe framework (Brandt et al., 2014). The checklist covers nine ordinal dimensions: (1) data availability (none/partial/full open); (2) code availability (none/partial/full open); (3) environment specification (none/partial/full); (4) documentation quality (none/minimal/comprehensive); (5) availability of protocols and other procedural instruments; (6) licensing (none/restrictive/open); (7) pre-registration (none/post-hoc/prospective); (8) independent verification record (none/informal/formal replication); and (9) presence and completeness of an associated Wikimedia Meta page (none/stub/complete), capturing community-facing transparency specific to the Wikimedia research ecosystem. Checklist dimensions are further informed by the principles for "good data" outlined in Daly et al. (2019), particularly the criteria concerning consent, agency, and contextual integrity.
Not all dimensions apply equally to every study design. Purely qualitative or survey-based studies, for instance, may not require environment specification or code sharing. Where a dimension is inapplicable, it is coded as "not applicable" (NA) and excluded from the aggregated score, computed only over applicable dimensions. Each publication then receives a final score standardized as a z-score, which expresses how a value relates to the mean of the distribution. These scores enable temporal comparison and stratified sampling for Phase II. Sensitivity analyses assess how NA rates and the variable number of applicable dimensions affect z-score distributions and corpus rankings.
Annotation Pipeline
[edit]To manage annotation at scale, a semi-automated pipeline is implemented as an executable Python script (not a conversational chatbot interface). In alignment with the project's commitment to open science, the analysis utilizes non-proprietary, open-source models to enhance transparency and auditability of the classification process. The pipeline operates in three steps. First, each article's full text is segmented into logical sections (abstract, methods, data availability statement, etc.). Second, a targeted term search is applied per reproducibility indicator (e.g., repository URLs, licence identifiers, pre-registration platform names). Third, the surrounding text of each detected term is extracted and passed to a small language model (SLM, Qwen2.5-7B) running locally for classification. This constrained input design, which provides only short relevant excerpts rather than full texts, suits the capabilities of SLMs and avoids reliance on general-purpose reasoning. If the SLM does not reach adequate performance, a large language model (LLM, Allen Institute's OLMo-3.1-32B) serves as a fallback for the same pipeline. Should an LLM be required, inference will utilize the University of São Paulo's high-performance computing infrastructure. These clusters provide the necessary VRAM capacity at no cost to the project.
Validation and Human-Machine Coding
[edit]In this design, the SLM functions as the second annotator under the supervision of the MSc-level researcher, who acts as the first annotator. Before deployment, the pipeline is validated against a manually coded set of approximately 30 articles. The model is used in production only if it reaches an agreement rate of ≥ 0.70 with the human rater, consistent with the Cohen's kappa threshold applied throughout the project. If this threshold is not met after reasonable tuning, checklist coding proceeds entirely by hand. Known biases of the chosen SLM are documented before deployment, with particular attention to the tendency, reported in related applications, to over-classify partial reproducibility artifacts as full (or the reverse) and to misread negation. Bias estimates from the validation set are reported alongside the kappa value and inform manual review. Results from the automated first pass are imported into Zotero for review, correction, and final validation by the human rater, preserving full human oversight. Discrepancies between machine and human codes are adjudicated by the research team.
Descriptive statistics (absolute and relative frequencies; means and standard deviations by year, domain, and journal) characterize the distribution of artifact types. Temporal trends are assessed using non-parametric trend tests. To identify predictors of full data and code availability, logistic regression models are fit, controlling for publication year, funding source, and disciplinary category. The results of this analysis ground the Phase II sampling.
Phase II. Semi-Structured Interviews
[edit]Participants are drawn from the Phase I corpus by stratified random sampling. Stratification uses z-scores computed only over applicable dimensions (see Phase I), so that variability in the number of applicable criteria does not distort classification across reproducibility strata. The design contrasts two groups of interest: publications with the highest z-scores and those with the lowest.
Sampling combines z-score terciles (low, medium, high) with five-year publication intervals (2010 to 2014, 2015 to 2019, 2020 to 2025), and deliberately oversamples the high-reproducibility stratum. The rationale is that high-reproducibility cases reveal more about the practices, incentives, and infrastructures that enable Open Science, whereas low-reproducibility cases mostly reflect disciplinary defaults. The design ensures temporal representation while preserving analytical sensitivity. Email invitations are sent to corresponding authors of eligible publications. Each stratum aims for 12 to 15 participants, based on theoretical saturation (Guest, Bunce & Johnson, 2006). Sampling continues until no new themes emerge across three consecutive interviews in a stratum.
The interview protocol covers four themes: (A) Knowledge of Open Science and reproducibility standards, such as TOP, FAIR, the Open Science Monitor (PLOS ONE), and pre-registration platforms; (B) Perceptions of Wikimedia research as a scientific practice and views on how the movement's open infrastructure relates to rigour, including a specific question on whether researchers consider the open licensing of Wikimedia datasets, by itself, sufficient to ensure the reproducibility of studies that draw on them; (C) Transfer effects, including a specific inquiry into how conducting research within the Wikimedia ecosystem may have fostered a more open and reproducible approach in their broader scientific practices; (D) Barriers to Open Science in Wikimedia, such as technical, institutional, and cultural obstacles. A pilot test with three outside researchers validates the protocol.
All interviews are conducted via video conference, audio-recorded with informed consent, and transcribed verbatim. Transcripts are processed through thematic Content Analysis following Bardin's (2011) framework, which employs systematic coding and classification of text to identify themes. The process uses a priori categories derived from the protocol, supplemented by emergent categories identified inductively. Coding follows a human-and-machine design where researchers serve as primary coders, and the 'Context' qualitative-computational tool performs an automated thematic structuring of the transcripts. Inter-coder reliability between human and tool is assessed using Cohen's kappa. The research team adjudicates discrepancies. Qualitative data are then quantified as category frequencies and analyzed for patterns across reproducibility strata and publication cohorts.
Timeline
[edit]- Months 1–2 (July–August 2026) — Corpus building and eligibility screening (5 weeks)
- Months 2–3 (August–September 2026) — Reproducibility checklist coding (5 weeks)
- Months 3–4 (September–October 2026) — Statistical analysis (3 weeks)
- Months 4–5 (October–November 2026) — Interview protocol development and ethics application (2 weeks)
- Months 5–7 (November 2026–January 2027) — Participant recruitment and interview conduction (8 weeks)
- Months 7–9 (January–March 2027) — Qualitative content analysis (7 weeks)
- Months 9–12 (March–June 2027) — Manuscript and white paper preparation (13 weeks)
All phases share a 6-month contingency buffer extending to December 31, 2027.
Policy, Ethics and Human Subjects Research
[edit]A study protocol is preregistered on the Open Science Framework before data collection. A data management plan is maintained on the DMP Tool platform. Ethical approval is obtained from the Research Ethics Committee of the University of São Paulo, Bauru School of Dentistry, before participant recruitment. All interview data are anonymized; the anonymized dataset and codebook are deposited in a public repository upon publication. Datasets released publicly are fully anonymised: no researcher names, institutional affiliations, or other identifying information are included. Results are presented in aggregate form (by year, disciplinary domain, or reproducibility stratum) to prevent identification of individual authors or publications. These procedures follow the applicable Wikimedia Research Best Practices Around Privacy, consulted before data collection and again before public deposition.
Results
[edit]Results will be posted here as they become available. This section will present the descriptive statistics of reproducibility indicators from Phase I, including absolute and relative frequencies, means and standard deviations by year, domain, and journal, as well as temporal trends and predictors of full data and code availability. Phase II findings from the semi-structured interviews will be integrated, with qualitative themes quantified as category frequencies and analyzed across reproducibility strata and publication cohorts.
Resources
[edit]Expected Outputs
[edit]- Peer-reviewed Article (Phase I and II combined)
- A single mixed-methods article integrates the quantitative reproducibility scores and temporal trends from Phase I with the qualitative interview findings from Phase II. Combining the two phases in one publication produces richer and more interpretable findings than separate reports, since quantitative patterns can be read through researchers' own accounts. Intended venues include PLOS ONE, Scientometrics, or the Journal of the Association for Information Science and Technology (JASIST). Primary audience: scholars working on Open Science, scientometrics, and information science. Benefit: establishes the first empirical baseline for methodological transparency in Wikimedia research and enables future longitudinal tracking.
- Open Datasets
- An anonymized, documented dataset contains the reproducibility checklist scores, z-scores, metadata, and coding scheme for all corpus publications. Deposited on Zenodo, with a FAIR-compliant data management plan. Where applicable, structured metadata (publication identifiers and reproducibility scores) is contributed as statements to existing Wikidata items, enhancing discoverability within the linked data ecosystem. The full corpus bibliography is also maintained as a public, read-only Zotero group library. Primary audience: researchers, funders, and movement strategists. Benefit: enables reuse, extension, and longitudinal comparison.
- White Paper
- A practitioner-oriented document translates findings into recommendations for three audiences: researchers, journal editors, and funders. Disseminated via Meta-Wiki and submitted to the Wikimedia Foundation. Benefit: provides actionable guidance aligned with the Wikimedia 2030 Strategic Direction.
- Open Research Software
- Data acquisition scripts, plus statistical and qualitative analysis code (R, Quarto, and Python), are deposited in a public repository (e.g., GitHub/Zenodo) under an open licence. Documentation is sufficient to reproduce the tables and figures used in the published article. Benefit: auditability, collaboration, and reuse in future research.
Community Dissemination
[edit]Presentations at Wiki Workshop, WikiCite, Wikimania, or the Wikimedia Research Showcase, with announcements on relevant mailing lists (wiki-research, Wikidata mailing list) and Wikimedia project pages dedicated to research about the movement. Wikimedia Brasil is approached as a dissemination partner to share outputs through its communication channels and community events, with a focus on reaching researchers and contributors. National Wikimedia affiliates, particularly Wikimedia Brasil, are engaged as dissemination partners. Community feedback on the white paper is solicited through Meta-Wiki talk pages before final publication.
References
[edit]
- Abdo, A. H., & Chagas, A. M. (2024). An empirical assessment of Open Hardware practices in the academic community and their policy implications. Paper presented at the Eu-SPRI Annual Conference, Brighton, United Kingdom. https://hal.science/hal-04488702
- Bardin, L. (2011). Content analysis. Éditions 70.
- Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 452–454. https://doi.org/10.1038/533452a
- Begley, C. G., & Ellis, L. M. (2012). Raise standards for preclinical cancer research. Nature, 483(7391), 531–533. https://doi.org/10.1038/483531a
- Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & van 't Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224. https://doi.org/10.1016/j.jesp.2013.10.005
- Buttliere, B., Vetter, M., Rasberry, L., Pensa, I., Mietchen, D., & Mkrtchyan, S. (2025). The state of science and Wikimedia: Who is doing what, and who is funding it? Wikimedia Research Fund Proposal.
- Daly, A., Devitt, S. K., & Mann, M. (Eds.). (2019). Good Data. Institute of Network Cultures. https://networkcultures.org/wp-content/uploads/2019/01/Good_Data.pdf
- Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822X05279903
- Ioannidis, J. P. A. (2005). Why most published research findings are false. PLOS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
- Jacob, L. C. B., Cardoso, M. J., & Peschanski, J. A. (2023). Collaborative production in digital environments: Using Wiki to promote digital literacy and communication. Revista Ibero-Americana de Estudos em Educação, 18, e023085. https://doi.org/10.21723/riaee.v18iesp.1.18510
- Jankowski, S., Ford, H., Iliadis, A., & Sidoti, F. (2025). Uniting and reigniting critical Wikimedia research. Big Data & Society, 12(3). https://doi.org/10.1177/20539517251357292
- Kharazian, Z., Starbird, K., & Hill, B. M. (2024). Governance capture in a self-governing community: A qualitative comparison of the Croatian, Serbian, Bosnian, and Serbo-Croatian Wikipedias. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1), Article 61. https://doi.org/10.1145/3637338
- Matos, H. G. C., Morata, T. C., Alvarenga, K. F., & Jacob, L. C. B. (2026). Indicadores de reprodutibilidade na audiologia brasileira: um estudo de metaciência. SciELO Preprints. https://doi.org/10.1590/SciELOPreprints.15352
- Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., … Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. https://doi.org/10.1126/science.aab2374
- Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
- Stodden, V., Seiler, J., & Ma, Z. (2018). An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11), 2584–2589. https://doi.org/10.1073/pnas.1708290115
- Turki, H., Taieb, M. A. H., Aouicha, M. B., Rasberry, L., & Mietchen, D. (2024). Ten years of Wikidata: A bibliometric study. Wikidata Workshop 2023. CEUR Workshop Proceedings, 3640. https://ceur-ws.org/Vol-3640/
- Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., … Mons, B. (2016). The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 3, 160018. https://doi.org/10.1038/sdata.2016.18