Global Data and Insights/Movement Data/Equity Landscape/Pilot & Consultation/Directed Review Questions

From Meta, a Wikimedia project coordination wiki

General Reflection Questions[edit]

First, we present five general questions related to the words and data we use which we hope most folks might have perspective on.

Question 1. Considering the output metrics developed to understand Wikimedia presence & growth; are there major domains we have failed to identify? [edit]

Click here to review the metrics at hand

The metrics of the pilot and initial public dashboard will be limited to our first round of data exploration and mapping and include:

The domain-level metrics

Review this deck to learn more about the project

The project focuses on triangulating our best available signal metrics across key engagement and enablement domains. All metrics are scaled 0-100, averaged across triangulation points, and then scaled 0-100 again so that the global percentile rank is 50 and those scores nearer 100 represent geographies signaling the most engagement in any named domain and those nearer 0 signaling the least. All scores are comparative indices in which no further modeling has been implemented beyond averaging and scaling triangulated signal metrics. The initial domains we have been mapping are:

  • Readership
  • Editorship
  • Program Leadership
  • Grants Leadership
  • Affiliate Leadership
  • Access
  • Freedom
  • WP Editor Population Penetration
  • Population Presence & Growth

The facets In addition to the main domain-level metrics, for those domains in which we have robust enough data (Readers, Editors, Grants, and Affiliates) we will also share sub-metrics along two facets:

  • Presence
  • Growth

The high-level metrics In addition to each of the domain metrics we will share three high-level roll-up metrics:

  • Overall Engagement
  • Overall Enablement
  • Rank in under-representation

Additional metric domains under consideration which we hope to progress toward

In progress Gender Equity

  • Estimated Contributor Gender Gap (Estimated Contributor Gender Ratios (% women), World Bank Gender Ratios (% of total population women))
  • Gender Inequality (Gender Inequality Index, Gender Development Index)

General Inclusion & Equity

  • Human Development (Human Development Index (HDI), Labour share of GDP, comprising wages and social protection transfers, Income-adjusted HDI (IHDI))
  • Palma Ratio (Income held by the richest 10%, Income held by the poorest 40%)
  • Income Inequality (Income Inequality GINI Index, Income Index (GNI per capita)
  • Human Inequality (Overall loss in HDI due to inequality, Coefficient of Human Inequality)
  • Inclusion (Looking at Peoples Under Threat & Inclusiveness Index)

In exploration

  • On-wiki Leadership (Looking at rights and use of administrative rights on the projects)
  • Content (Total wiki content and net new content added)
  • New Editors (New Editors, New Active Editors, and New Retained Active Editors)
  • Content Interactions (Integrating content previews along with pageviews)
  • Disinformation and cyber security threats (Oxford data)
  • Partnerships (Count and Type across grantees and affiliates)
  • Grants Leadership (Improved to integrate outcome metrics as well as grants counts and dollars)

Question 2. Are there input measures that could be improved from the landscape of existing data or measures which should not be part of the measurement framework for Wikimedia presence & growth?[edit]

Click here to review the input data to the metrics
Click here to review the underlying metrics at hand

Domain Measure
Readers Average monthly unique devices

Average monthly pageviews

Editors Average monthly editors

Average monthly active editors

Programs Count Education Events

Count GLAM Events Non-affiliate project grants Count organizing hubs engaged

Grants Annual grants FY

Historic grants

Growth in Grants

Affiliates Highest Governance Type

Affiliate grants

Affiliate count, size, and tenure

Count new recognitions

Population World Population

Population Growth

Access Population Accessing Internet

GSMA Mobile Connectivity Score

Access to Basic Knowledge

Access to Information & Communications

Freedom Freedom Index

World Press Freedom Index

Control of Corruption Score

Question 3. Are you aware of any key data sources we have missed for understanding Wikimedia presence & growth?[edit]

Question 4. What are possible use-cases of these metrics for you?[edit]

Question 5. Do you have any other thoughts you would like to share with us about the Wikimedia presence & growth measurement framework or the planned dashboard elements?[edit]

Specific Design Considerations[edit]

Here we go a little deeper with design considerations on naming and analysis that are intended to be responded to as they are relevant to you as a potential data partner or user.

Consideration 1. The Nomenclature (or what do we call this project and its data)[edit]

Click in this space to reveal the question prompt and background information
Do any of the proposed project, metric domain, or facet labels present potential challenges to you or your group’s planned projects and partnerships?

  • We have active voting on the title of the equity data landscape in All our ideas
  • You can review the metrics schema table to review metrics names and comment with any challenges

Answer on the talk page

Consideration 2. Metadata frame[edit]

Click in this space to reveal the question prompt and background information
To what extent do the metadata classification systems enable or inhibit potential cross-use with datasets you or your team use to understand Wikimedia communities?
We plan to catalog the following metadata for cross-referencing across datasets:
  • UN Country labels and codes
  • UN Continent & Subcontinent classifications
  • IBAN 2- and 3-Digit Country Codes
  • Maxmind country label
  • Wikimedia organizing hubs alignments (i.e., CEE, ESEAP, Iberocoop, WikiIndaba, North America (& US Coalition), Northern Europe, WikiArabia, WikiFranca, South Asia, Chairperson's Group, Chapter EDs
  • Official language listings and ISO 639-1, 639-2, and 639-3 Language codes as available

Answer on the talk page

Consideration 3. Referring to (In)Equity[edit]

Click in this space to reveal the question prompt and background information
Does referring to equity or inequity resonate more with recent patterns of use you have seen in research and evaluation of inequity in your operating space?
Referring to equity vs inequity. Some comments have been shared that we should focus on improving equity rather than reducing inequity and labeling our aims and all coefficients positively in this way.

Answer on the talk page

Consideration 4. Measuring Diversity & Inequity[edit]

Click in this space to reveal the question prompt and background information
Kolm categorized absolute and relative measurements of inequality as “rightist,” “centrist,” and “leftist,” depending on their treatment of inequality as an absolute or relative concept. Leftist measures are sensitive to absolute changes; they do not change when all incomes go up by the same absolute amount. An example of this type of measure would be the absolute gini, which is a standard gini coefficient multiplied by the mean of the distribution. Kolm defines centrist measures as measures which show increased inequality when average incomes rise and the relative distribution stays the same, and decreased inequality when all incomes rise by the same absolute amount. Rightist measures are purely relative; when all incomes go up in the same proportion, they are unchanged.

Using percentile ranking relies on a relative view of each group as opposed to an absolute view of capacity. When it comes to measuring change over time, we will rely on measuring the distributions of the input measures. For this we have several options:

For (In)Equity:

For Diversity & Dissimilarity

If you are experienced with any of the above, please share your thoughts on these options via direct comment.

Answer on the talk page

Consideration 5. Scaling and Weighting[edit]

Click in this space to reveal the question prompt and background information
For scaling and weighting we have identified some key options and are leaning to the options noted below. Please share what, if any, concerns or alternative suggestions you may have.

Scaling: For relative comparisons and roll-up ranking we need to triangulate across more than one input data point to estimate a more generalized global ranking. This means scaling for relative comparison and combining for relative presence and growth.

We considered raw ordered ranks across the distribution, percentile ranks across the distribution, Z-scores, and ratios. With the exception of some background calculations to determine limits to room for growth which apply ratios, we are currently applying percentile ranks 0 to 100 across all inputs for triangulation and calculation of output metrics.

Weights: Grants dollars must be weighted by the local economy and all input measures must be weighted by some population factor in calculating coefficients of inequity.
  • For grants, we considered: median income, per capita GDP, median equivalised income, and Per capita GDP (PPP). We currently plan to use per capita GDP, PPP
  • For GDP (PPP) weighting, there are two available weights for annual per capita GDP, PPP, and current international $ and constant international $. We currently plan to apply constant international $ for the weighting of historical grants and current international $ for the corresponding calendar year of grants.
  • For all other input measures, we are considering weighting relevant resourcing data either by general population, readers, or editors.

We currently propose to apply as follows:

  • Once we can derive unique editor counts by geography, resources could be weighted by editor and/or active editors population for consideration of underserved editor populations in the world or by general population to consider underrepresented populations in our ecosystem
  • Editor data could also be weighted by general population to understand a concept such as “activation rate” for growth potential or by readers to understand a concept like “under-activated” consumers.
  • Similarly, Readers stats could be weighted by editor, and/or active editors population, and/or general population in consideration of slightly different reach and inclusion questions and arrive at different answers regarding participation gaps.
  • Lastly, regional roll-ups could be weighted by population.

Considering what engagement and resourcing gaps are most important to the strategy work you or your group engage in, please share what excitements, concerns, or curiosities you have about the above weighting options.

Answer on the talk page