Community Insights/Community Insights 2021 Report/Survey Methodology

2021 Community Insights Report

• Executive Summary
• Brand Awareness
• Platform Evolution
• Thriving Movement
• Survey Methodology
• Endnotes
• Supplement: Collaboration, Diversity & Inclusion
• Community Insights Home

How was this data collected?

The Community Insights survey was conducted in September and October of 2020 through Qualtrics. Most survey participants were selected to take the survey using a stratified random sampling strategy, with the goal of collecting a sufficient number of responses among different Wikimedia projects and levels of editing activity to identify experiential differences between groups in a statistically valid way. Active online contributors who had "email this user" enabled were sampled by activity level and project, and those sampled received an email asking them to opt-in to take the survey. If they agreed, they then received a link to take the survey via their email. In addition, movement organizers were sampled randomly from Foundation grantee contacts and affiliate contacts, and were also contacted to take the survey via email.

How many people responded, and how many completed the survey?

28,181 contributors were randomly selected from across Wikimedia projects. 3332 of them (12%) agreed to be sent a link to the survey, 2638 (9%) began the survey, and 1808 (6%) finished at least half of the survey.

How was this data analyzed?

Unless otherwise specified, all frequencies and statistical tests are weighted by editing activity level using an inverse propensity score method. The propensity score used was the likelihood that each activity-strata of editors would complete at least 50 percent of the Community Insights survey. Effectively, weighting in this way means that those who were less likely to take the survey (this tended to be less active editors) carry slightly heavier weight in our analyses, so that we can report results that do not overrepresent the experiences of highly active editors, who are more likely to take the survey. In the interest of equitably representing Wikimedians--especially less-active editors who are less likely to frequent spaces where we tend to collect data--we have not excluded those completing less than 50 percent from the analyses in this report, and have applied the same weights to their responses as those established  for users who completed at least half the survey. Where including this group may have influenced analytic results, details are noted in relevant endnotes in Methodological Appendix B.

For the 2020 data set, we began weighting non-intra-year-comparative population estimates by both editing activity and the project or project family respondents were sampled by, again using the inverse propensity score weighting method described above. This means that some year-over-year comparisons, for example the estimated proportion of contributors who are women, will appear slightly different for 2019 data than the proportion reported in the previous year's report. Analyses involving movement organizers are also weighted differently this year, applying a weight for movement organizers for whom we know editing activity and/or home project and including unweighted organizers for whom this information is not known. Therefore, year-over-year analyses involving movement organizers may also differ slightly from what was reported in 2019 due to our change in weighting procedures. When we present year-over-year comparisons here, we are weighting each year's data in the same way.

In this report, any differences between groups reported are only those that reached a level of statistical significance—meaning that the difference is big enough, and/or seen among enough respondents, to lead us to conclude that a real population difference has likely been detected, rather than statistical “noise.” There are groups for whom we detected no differences, but this does not always mean that differences do not exist. At times, we did not have a large enough subsample to detect those differences. This is why geographic differences are often reported at the regional/continental level—we acknowledge that continents are diverse and complicated spaces, but we often lack the statistical power—usually due to sample size—to detect country- or even sub-continent-level differences.

Countries and territories are categorized into continents according to the United Nations Statistics Division, with the exception of the Americas which are separated into the UN’s two sub-regional classifications, Northern America and Latin America/Caribbean. For more about which countries make up the regional and subregional classifications used in this report, see the UN's Geographic Regions. We also look at geographic differences by whether a country or territory is classified as "emerging" in this proposed classification from the former Community Engagement department at the Wikimedia Foundation.

Newcomers are defined as people who began contributing to Wikimedia projects in 2019 or 2020.

Youth are defined as people 18-24 years old. Due to the various legal protections of minors in research participation, we did not attempt to survey people under the age of 18.

What are the limitations of the Community Insights survey?

Each year, we work towards making the data from Community Insights more representative of Wikimedia communities worldwide. This year, the survey was written in English and translated into 19 other languages: Simplified Chinese, Latin American Spanish, Brazilian Portuguese, Russian, Japanese, French, German, Italian, Arabic, Polish, Ukrainian, Dutch, Bahasa Indonesia, Bengali, Hindi, Korean, Thai, Turkish, and Vietnamese.

In addition to expanding the languages available, we are also actively working to shorten the survey further to encourage greater participation in the future. We are also exploring alternative means of survey distribution to better reach Wikimedia communities.

Finally, this data was collected during a unique time in world history-- the Covid-19 pandemic. This period has seen drastic social and economic changes, and in the case of the Wikimedia movement, some record highs in traffic on our project sites and editing activity. It is difficult to say whether some of the changes found since 2019 in this report are reflective of long-term trends in demographic and experiential shifts among Wikimedia contributors, or reflective of the particular circumstances in which we currently find ourselves.