User:Shouston (WMF)/Sandbox/Global Metrics/Re-design

Idealab

Redesigning Global Metrics

Update Global Metrics, as well as the tools and resources that support the collection of that data.

contact emailshoustonwikimedia.org, abittakerwikimedia.org

idea creator• Shouston (WMF)

project manager• Shouston (WMF)

advisor• JAnstee (WMF)

volunteer• Abittaker (WMF)

this project needs...

volunteer

developer

designer

community organizer

researcher

Background

The Community Resources team, in conjunction with Learning & Evaluation, conducted a two-month long retrospective on Global Metrics, a requirement of all grantees requesting grants from the Wikimedia Foundation. The results of this retrospective revealed benefits to having Global Metrics, but also revealed many issues associated with Global Metrics and it's training, resources and supporting tools. Moreover, while Global Metrics was originally intended to provide a high-level overview of certain grant-related outcomes - in the areas of Participation and Content - the results of the retrospective revealed that Global Metrics is being used in many more ways than intended, in ways that it was not designed to support well.

Idea

What problem are you are trying to solve?

The update to Global Metrics aims to address (at least partially) three primary problems within the Wikimedia grantmaking ecosystem. While Global Metrics has been used in many other ways, these will be the primary situations and needs designed for.

1. Help those within the Wikimedia movement (specifically grantees, Committees, WMF Staff) need to understand the outputs & outcomes of funded activities. This was the original problem the first iteration of Global Metrics was intended to solve, and has with mixed success.
2. Help WMF Grant staff and Grant Committees need to understand historical and future achievements, to make informed decisions about grant funding.
3. Help grantees with adaptive planning, by providing useful data that will help improve the planning and designing of their activities / programs

What is your solution?

Update the requirement of Global Metrics - the structure, the measures used, and the way it is included in grant proposals and reports - as well as the supporting tools and resources that aid in the collection of this information (e.g. Wikimetrics, associated learning patterns).

For updating the requirement of Global Metrics: We have created two proposals based on the suggestions we heard. However, we encourage any other ideas to be submitted! There were two suggested proposals we did not go forward with: removing the entire requirement of Global Metrics, and have requirements that are specific to each grant program. See the appendix for more details on why.

For updating the tools and resources supporting Global Metrics:

Timeline

Month	Acitvities
6 June 2016	Consultation and voting starts
30 June 2016	Consultation and voting ends
1 - 15 July 2016	Review feedback and any additional proposals
18 July 2015	Final selection of replacement for Global Metrics. New requirement will be integrated into Project and Conference/Travel grants. Begin discussion on transition plan for APG and Simple APG programs. Begin updating resources and training materials.
August 2016 onward	Continue updating resources and training materials.
July 2016 to July 2018 or 2019	This next iteration of Global Metrics will last 2-3 years, contingent on the final transition plan for APG & Simple APG. The next evaluation of this grant requirement will be conducted at the end of this timeframe.

Proposed updates to the requirement

Based on the feedback we heard from community and WMF staff, we are proposing to update Global Metrics - the structure, the metrics required, the way the requirement is integrated into grant proposal and report forms - as well as the tools and resources that support the collection of Global Metrics (e.g. Wikimetrics and it's supporting documentation in the Grants:Evaluation space on Meta).

How will we evaluate the strengths & weakness of each proposal?

There were many issues raised by community and staff about Global Metrics, more than can be addressed through just changing the metrics or the requirements. However, it's important to recognize where this replacement for Global Metrics will solve issues, create new issues, or lead to the same issues as today. To this end, we identified a set of criteria, based on the project's design principles, that we have used to assess the strengths and weaknesses of each proposed solution (which can been see under the "Assessment of this option against design criteria" section of each proposal).

Proposal 1: 3 shared measures

Structure and measures ▼ Structure Three shared metrics, two quantitative and one qualitative, that grantees will be required to report only when relevant to their project goals. These three shared metrics will be shared across all grant programs. Measures Participation related measures: Quantitative: Could be either: Number of individuals involved. This includes number of people who participated in your program that you reached out to individually and could follow up with after the event. (Even if you did not follow up with them.) For a list of the sorts of participation that should and should not be included, see here. Number of active editors retained after 1/3/6 months. This is the number of editors who have made at least 5 edits on any combination of wiki projects in the 30 days after the day the program ends. Qualitative: Question on community building Content related metric: Quantitative: The number of content pages added, by project. Assessment of this option against design criteria ▼ This proposal resolves many of the issues identified in the Global Metrics retrospective and passes most of the design criteria with a medium or higher and scores especially high on relevancy, existence of tools, and ease of collection. It scores lower on flexibility and accountability. Area Criteria Assessment Strike a balance between flexibility and consistency Flexibility Low flexibility: report only when relevant, but only three possible metrics Strike a balance between flexibility and consistency Consistency Med consistency: three metrics would be reported across all programs Be relevant Expressed interest High relevancy: metric definitions updated to reflect requested specificity Support accountability WMF Grants to WMF leadership and donors Support accountability Grantee to WMF Grants Medium-low accountability: gives a broad idea of the outputs of a program, but lacks context Ease community burden Existence of tools High ease: tools exist to collect all three measures. Process of collecting could be made easier Ease community burden Effort to collect High ease: reduced to only three measures, all of which have existing tools to collect them. Ease community burden Availability of guidance Medium availability: tools exist to collect these measures, but an initial investment is necessary to learn how to use them, and new guidance would have to be written to accommodate new definitions of measures. Support Comment on the talk page Comment privately via the survey

Proposal 2: 3 shared measures + 2 grantee-selected measures

Structure and measures ▼ Structure Three shared metrics that grantees will be required to report, only when relevant to their project goals. These three shared metrics will be shared across all grant programs. In addition each grantee reports on two outcomes that are relevant across their grant activities. These outcomes do not have to measured quantitatively (that is, they don't have to be another metric). For example: APG grantees that have identified gender diversity as a strategic focus across all of their programs might report on how they have improved the content gap on notable women. Project grantees that Measures 3 shared measures are the same as Proposal 1: Participation related measures: Quantitative: Total individuals involved Qualitative: Question on community building Content related metric: Quantitative: The number of content pages added, by project. Assessment of this option against design criteria ▼ This proposal resolves many of the issues identified in the Global Metrics retrospective and passes most of the design criteria with a medium or higher, and scores especially high on flexibility, relevancy, and accountability. It scores lower on existence of tools and guidance. Area Criteria Assessment Strike a balance between flexibility and consistency Flexibility High flexibility: only three standard measures to report when relevant, two additional measures choose objectives important to their program. Strike a balance between flexibility and consistency Consistency Medium consistency: three measures shared across all programs, but there is a risk that the extra measures submitted will contribute only to the understanding of the program and not cross-program learning. Be relevant Expressed interest High relevancy: metric definitions updated to reflect requested specificity and requested space created for grantees to submit their own measures. Support accountability WMF Grants to WMF leadership and donors Support accountability Grantee to WMF Grants Medium-high accountability: two additional measures allow grantees to showcase more facets of their program's objectives and outcomes. Ease community burden Existence of tools Medium-low ease: tools exist to collect all three shared measures, but risk exists that grantees will want to collect metrics that tools do not exist for, and the WMF will be unable to support this. Ease community burden Effort to collect Medium ease: the three shared measures can be collected with existing tools, but grantees may select two additional measures that are more difficult to collect. Ease community burden Availability of guidance Low availability: tools exist to collect the three shared measures, but for the two additional measures, grantees may want to collect the same measures as other grantees but not have the capacity to do so or the guidance to learn how. Support Comment on the talk page Comment privately via the survey

Propose a new solution Make your suggestion

Get Involved

Community notification

(Links to where we have announced this consultation)

Appendix

Design principles

The three problems stated above represent (at times) conflicting needs: how can one metric be both nuanced enough to provide grantees with relevant data to improve their plans, as well as consistent enough such that WMF can aggregate the data in a meaningful way? How can one set of metrics or learning questions be relevant across all grant programs, given the diversity of grant-funded activities?

The answer we heard from grantees, grant committees, and WMF staff is that one set of metrics or learning questions will not be able to meet all these needs perfectly. While there are benefits to having some kind of standard requirement, Global Metrics (or its reincarnation) will always need to complement the other metrics and qualitative data that grantees collect, report, and find useful. To this end, the goals in re-designing Global Metrics is that the updated requirement should:

1. Strike a balance between offering flexibility and retaining consistency to have meaningful, movement-wide information
2. Be relevant to grantee goals, objectives, or measures of success
3. Support accountability, e.g.
   • the information collected should allow WMF to show donors the collective outcomes of all donor dollars invested in grants
   • the information collected should allow grantees to show to WMF the outcomes of the resources (monetary and nonmonetary) invested in grant
4. Ease community burden of identifying and collecting data and reporting
5. Have sufficient community engagement in the development
6. Training, resources, and support that is integrated into the grant processes & resources

Assessment criteria

The following criteria will be used to assess the strengths and weaknesses of each proposed replacement for Global Metrics:

1. Principle: Strike a balance between offering flexibility and retaining enough consistency to have meaningful, movement-wide information
   • Criteria:
     • Flexibility: Does the structure and metrics provide enough flexibility to meet the various needs of grantees, grants committees, and WMF Grant staff?
     • Consistency: Are the metrics/question definition specific enough such that everyone is collecting the same information?
2. Principle: Be relevant. As much as possible, metrics should be relevant to the grant’s goals, objectives/measures of success
   • Criteria: Have grantees demonstrated interest in understanding the outcomes captured by these metrics / questions?
3. Principle: Support accountability, e.g.
   • the information collected should allow Wikimedia Foundation to show donors the collective outcomes of all donor dollars invested in grants
     • Criteria: Will the WMF Grant teams be using the information collected to communicate to WMF leadership, donors, or broader audiences?
   • the information collected should allow grantees to show to the Wikimedia Foundation the outcomes of the resources (monetary and nonmonetary) invested in grant activities
     • Criteria: Will the information reported by grantees demonstrate the outputs/outcomes of their work? Is there clear association of metrics to project goals?
4. Principle: Ease community burden of identifying and collecting data and reporting.
   • Criteria:
     • Do the chosen metrics have working, available tools for collecting the information?
     • For each grant awarded, do we anticipate the training & collection of information to take more than 20% of the volunteer or grantee staff time?
     • Is there guidance available to collect this information?

The following criteria will be used to assess the overall re-design:

1. Principle: Community engagement in development of new solutions or resources
   • Criteria: Have the same number (or more) of grantees & committee members given feedback on these proposed solutions, as in Phase 1 of the project?
2. Principle: Training, resources, and support for quantitative and qualitative data collection needs to be integrated into the grant processes & resources.
   • Criteria:
     • Mutual Participation: Will multiple people be able to contribute their own ideas / experiences around metrics?
     • Shared Understanding: Do the existing or new resources establish a common definition of each metrics / question)

Other ideas that were not pursued

There were two other ideas proposed that were not pursued:

• Removing the requirement of Global Metrics: (RATIONALE NEEDS TO BE ADDED)
• Have a requirement that is specific to each grant program: (RATIONALE NEEDS TO BE ADDED)