User:Shouston (WMF)/Sandbox/Global Metrics

Timeline

2009 & 2010

• c. 2009 - Grants at the Wikimedia Foundation begins organically, with one-off requests to Erik and later Asaf for minimal financial support. This eventually evolves into the PEG program, then called the Wikimedia Grants Program.
• July 2010 - Bridgespan completes is year-long strategy process, culminating in five strategic priorities for the Wikimedia Foundation and a set of associated targets. These strategic priorities are integrated into the grant proposal & report forms under the "Fit to Strategy" sections, as a first attempt to align grantee "impact" to the Foundation's strategic direction.

2011 & 2012

• Aug 2011 - Wikimedia Board of Trustees meets at Wikimania in Haifa, Isreal, to discuss fundraising within the Wikimedia movement, specifically funds raised through the Wikimedia sites
• March 2012 - Wikimedia Board of Trustees resolves:
  • Funds raised via the Wikimedia project sites should be considered to be movement money, not the entitlement of a particular organization or stakeholder:

    We affirm that all funds given to the Wikimedia movement are given in support of our global projects, in response to the value created by the global Wikimedia movement. Therefore, funds raised via the Wikimedia project sites should be considered to be movement money, not the entitlement of a particular organization or stakeholder. Decision-making about funds dissemination should be broad and inclusive, consistent with our mission, vision and values.

• • Creation of the Funds Dissemination Committee (FDC) to distribute those funds:

    To support a broader and more inclusive decision-making process for funds distribution, the Wikimedia Foundation will create a volunteer-driven body (working title: the Funds Dissemination Committee, or FDC) whose sole purpose will be to make recommendations to the Wikimedia Foundation for funding activities and initiatives in support of the mission goals of the Wikimedia movement. All funds raised via the Wikimedia project sites will be distributed via the recommendations of the FDC, with the exception of Wikimedia Foundation core operating costs and the operating reserve as described above. The work of the FDC should be guided by our principles for funds distribution (as laid out in our January 2012 resolution): to protect core operations, assess impact, promote transparency and stability, support decentralized programs, promote responsibility and accountability, and make decisions openly and collaboratively.

• July 2012 - Design of the Funds Dissemination Committee begins, with an initial framework created for feedback and iteration.
• Nov 2012- Wikimedia Board of Trustees approves the WMF's updated strategic direction and FY2012-13 annual plan, better known as Narrowing Focus, identifying "Grantmaking" as a core competency of WMF, one that should be improved to be "strategic, impactful, and accountable".

2013

"Impact" continues to grow as conversation, not just the question of "What is the impact of the ~8M USD awarded via WMF grants?", but also the question of "What is the impact of the Wikimedia Foundation?".

• March - Given the WMF's nascent understanding of grantee outcomes, the Foundation begins to look at the "efficiency and effectiveness" of volunteer led activities (called "programs") - e.g. editathons, on-wiki writing contests, photo competitions, GLAM. (See blog post by Frank) - as a first attempt to understand "Impact".
• (Date?) - The Program Evaluation and Design team is created begins to collect data on the metrics used by volunteers program leaders for popular programs. This data is eventually published into the pilot set of program reports.
• June - This first set of program reports is used in a series of sessions at the first [Program Evaluation and Design Workshop] in Budapest, to understand which of metrics were the most commonly reported and feasible to capture.
  • This conversation at Budapest is the first discussion of standardizing metrics (see session documentation).
• (Date?) -  "What is the impact of your project?" is added to PEG proposals, with applicants asked to describe how their project was aligned with (the now defunct) 2010 Wikimedia strategic priorities of Reach, Participation, Quality, Infrastructure, and Innovation.

2014

• June 2013 - May 2014: ED transition from Sue to Lila. Concerns about declining page views and future fundraising levels leads to greater pressure (by who?) for accountability on donor dollars spent.
  • Lila's tenure as ED included an continuous move toward quantitative solutions for capturing progress and impact (e.g. internal KPIs).
• May - L&E beings the Metrics Dialogue through a session at the Wikimedia Conference (need more info from Jaime on whether a standardized set of metrics was discussed here).
• June - September: The Learning & Evaluation team, in conjunction with other WMF staff (?), met to create a proposed set of standardized metrics. The proposal of the current global metrics is created based on those internal conversations, and adjusted somewhat based on additional comments and concerns received in the limited time for community review before they were launched in September 2014.

Miscellaneous information

It would be helpful to have a regular framework of evaluation and impact assessment, to support the FDC's work as well as that of the rest of the movement. The FDC will play a key role here, and should fit into existing processes and help align them with one another. This should be part of how we listen to our communities, our reusers, and our coordinators; an essential part of how we make the projects better each year.

— SJ, Comment on FDC proposal talk page

Assortment of sources of information

FDC draft proposal

• Comment on talk page from Sue: "So. In my view, the purpose of the FDC process is to make allocations to FDC-eligible entities, based on an assessment of the extent to which requested funding will enable those entities to move the needle towards realizing the movement's shared objectives, through a process in which assessment is a function of i) how much impact their proposed activities are likely to have and ii) how well the entities are situated, to execute them responsibly and well. The work then breaks down into three buckets of activity: assessing the likely impact of plans against the movement's global objectives, assessing the ability of entities to execute those plans effectively, and appropriately aligning the funding allocations with those assessments."
• Comment on talk page from SJ: "It would be helpful to have a regular framework of evaluation and impact assessment, to support the FDC's work as well as that of the rest of the movement. The FDC will play a key role here, and should fit into existing processes and help align them with one another. This should be part of how we listen to our communities, our reusers, and our coordinators; an essential part of how we make the projects better each year."

Interview with Anasuya

• "Impact" as a key topic entered the conversation in the creation of the FDC

March 2013 - blog post by Frank S.

Narrative / major points:

• “Measuring impact” in this case refers to quantifying the long-term effects those programmatic activities have on Wikipedia and its sister projects.
• Why program evaluation is important:
  • "If you’re supporting programmatic work as a volunteer, you most likely want to know whether your activities are worth the effort."
  • "If you’re helping to improve existing programs, you’re most likely eager to find out which changes will make your program more effective and efficient."
  • "If you’re thinking about starting a new program, you will want to have some evidence that your new program is working. How else would you convince others to participate in your program? And how else would you convince a funder to provide you with a grant so you can eventually execute your program and grow it over time? Program evaluation will help you to make a strong case for your idea. And it will also prevent you from embarking on activities that have no impact."
  • "If you’re serving on the board of a Wikimedia chapter or a thematic organization, you might want to know which kind of programmatic activities produce the “biggest bang for the buck”. "
  • Finally, with the Funds Dissemination Committee (FDC) being in place since last year, there’s also another reason why program evaluation will be more important than ever: after the first round of funding for 2012/2013, the FDC requested more information about program impact, so it has a better foundation for making recommendations on what to fund in the future. This means that from now on, funding decisions will rely heavily on the ability of grantees to demonstrate what impact their programmatic activities have.

Sept 2014 - blog post by WMF

Narrative / major points:

• Three major insights from first impact reports on grants:
  • "People are still finding it difficult to measure their work in clear ways"
  • "the larger the grants, the less proportionate the impact seems to be (and one challenge may be reporting)"
  • "we (WMF?) are finding it difficult to assess the collective impact of the considerable work supported by these grants in any systematic fashion. In particular, as a movement, we are not yet skillful in offering both the stories and the numbers, that describe how our offline work positively impacts our online successes."
• "Difficult to convey outwards what we are accomplishing as a movement"
  • "The main challenge these Global Metrics are trying to overcome is the limited ability observed in Wikimedia projects and programs to sum up inputs, outputs and outcomes in self-evaluation and thereby to give us all a more cogent sense of the collective impact of our work. We hope that more cohesive reporting will help us celebrate our successes as a global movement, but also point out where we are not making an appreciable difference"