User:JAnstee (WMF)/Sandbox/Draft/Outcome Mapping

Outcome Mapping is an evaluation approach developed in 2001 for evaluation which places people as the center and looks at changes in behavior to define outcomes.

Importantly, program outcomes are defined as “changes in behavior, relationships, activities, or actions of the “individuals, groups, and organizations with whom the program interacts directly” as well as “those with whom the program anticipates opportunities for influence” are considered “boundary partners” and brought to the center of the outcomes mapping approach (Earl, Carden, & Smutylo, 2001, p.1).

In this way, outcome mapping can help to get beyond the more direct program outcomes to deeper environmental outcomes to measure a program’s contribution to complex change processes.

In the Wikimedia world of programs, outcome mapping strategies can help us to gather data on the contributions that our programs make in terms of bringing about changes in our partners around the world that help to build toward greater engagement in open and free knowledge. Further, outcome mapping can also help us to evaluate both the intended as well as unintended results of our innovative Wikimedia programs.

WHAT	Outcome Mapping: A Social Cartography of Wikimedia Programs
WHO	Affiliates and program leaders of GLAM and Education,
WHY	Because we need to get beyond topline dashboard metrics to mapping the qualitative outcomes of core programs as movement drivers (rather than content metrics) for hundreds of implementations of core programs.
SO WHAT	Allows for qualitative outcomes and stories of our programs and helps to better surface our shared impact
HOW	Help to tell local program stories Honor the diversity of local programming by providing an evaluation framework that can engage across our landscape of programs Enable the sharing and capturing of social and environmental changes targeted by our program leaders and their partners. Emphasize qualitative processes and outcomes -- organically, holistically, and comprehensively

More details and some example tasks and potential timelines

Importantly, there are three basic stages to outcome mapping which form a continuous feedback loop for holistic program thinking and development. While it typically follows a linear pattern at the beginning, it often cycles back through the various planning activities as learning happens.

Stage 1. Intentional Design

Intentional Design is the initial planning stage of Outcome Mapping. During this stage, Wikimedia program and community leaders will work to reach consensus on two things:

1. The broad environmental changes (macro level change) that a program would like to help bring about in the world
2. The plans and strategies to provide support that is appropriate, based on resources and opportunities, to help create those changes

Much work has already been done to support our initial review by various program and community leaders who have shared their logic models to describe the theories of change behind their programs. The next step will be to look across these program models to focus on those changes in behavior, relationships, activities, or actions of the people, groups, and organizations are being targeted by our core programs which are not measured in direct program outputs or on-wiki metrics.

Key tasks

Describe the vision and identify:

The mission	Our partners in the work	The ideal changes in behavior, relationships, and/ or actions which we wish to influence in our partners
Examples: Increase coverage of cultural content on Wikimedia projects Increase access to cultural content available through open licensing of cultural content on Wikimedia Projects	Examples: Community partners GLAM institutions Educational partners	Examples: Partner promotion of Wikimedia values Partner engagement in developing Wikimedia projects Increased awareness of open knowledge projects and resources

After identifying mission, vision, partners, and desired changes, we will seek to map those expected changes collaboratively to identify a set of graduated progress markers which we can use to see if our partner-directed efforts are leading to the desired changes.

The outcome mapping framework prompts us to look at these desired changes in terms of:

The changes we expect to see	The changes we hope to see	The changes we would LOVE to see
Examples: Participation in partnership and planning meetings Development of shared language and skills for project/program Successful implementation of collaborative events	Examples: Partner articulation of movement values and strategies at the local level Expanding the partnership to additional partnerships, opportunities, or resources Partners changing their policies to better promote movement values	Examples: Partners playing a leading role in managing resources Partners sharing experiences and lessons learned Partners influencing policy at a larger scale

After mapping our desired changes, the outcome mapping framework guides the creation of a clear strategy map for each shared outcome that we wish to document together. In this step, we will develop matrices that categorize the strategies that similar programs employ to influence partners. In this step, each strategy is classified along two axes:
Target: Whether the strategy is aimed a specific individual or group OR the environment of a specific individual or group, and
Type of Influence: Whether the strategy is causal, persuasive, or supportive.

Example: Outcome Area: Strategies for the desired outcome of increasing awareness of open knowledge projects and resources

Target of Strategy	Causal Influence	Persuasive Influence	Supportive Influence
A specific individual or group	Strategies which have a direct effect on target or produce a direct output	Strategies which bring about new thinking or skills, based on those experienced and expert	Strategies to develop self-sustaining networks for involvement and support
Example: GLAMs	Digitization Support Workshops and edit-a-thons to add content to projects	Blog storytelling and infographics development Presentations to potential GLAM partners	Skill development for movement volunteers
The environment of a specific individual or group	Strategies which bring about change in social, physical, or policy environment	Strategies to create a persuasive environment and messaging system to disseminate information	Strategies to create an active learning network across partners to guide action
Example: Copyright Policy	Attend and organize physical meetings of policy makers	Gain presence at consultations, dialogues and events Consult with policy makers on free knowledge policy	Networking and coalition building

The final step of stage one is articulating the key actions that would demonstrate these engagement practices within organizations and mapping those potential measures which would also evidence desired changes. This is based on how organizations are working to:

• Seek new ideas and opportunities or feedback from key program targets and partners;
• Obtain support from those at the next level of organization (partnership development with partners at the next highest power);
• Assess the value of the activities, modify product/program design to increase value, and share wisdom about successful strategies; and
• Engage in organizational reflection and experiment to remain innovative.

From this mapping we can then move into the second stage to prioritize what outcomes we want to map and design strategies for doing so.

Potential timeline and examples:

• Collaborative identification of shared points of impact (Initiate at WMCON April 2016, push out to meta consultation as follow-up - May-June 2016)

Stage 2. Outcome & Performance Monitoring

The Outcome & Performance Monitoring Stage is the second stage of Outcome Mapping. During this stage, program stakeholders work to define and map strategies for the ongoing monitoring of:

1. The program's actions in support of the outcomes
2. Partners' progress towards the achievement of outcomes.

Stage two planning is based largely on setting up shared systems for systematic monitoring and self-assessment. Key tasks include setting priorities for monitoring and developing an outcome journaling system for those target outcomes.

Potential timeline and examples:

• Collaboratively design process (Follow-up online meetings with core stakeholder groups to respond to community consultation - bi-monthly following WMCon May 2016, July 2016)

Stage 3. Evaluation Planning

The Evaluation Planning Stage Stage is the third stage of Outcome Mapping. During this stage, program stakeholders identify evaluation priorities and develop an evaluation plan.

Potential timeline and examples:

• Map out our monitoring needs and options and agree upon a shared outcome and performance monitoring system and appropriate evaluation tools (e.g., tracking logs, reporting templates, survey templates, etc.) (September 2016, November 2016)
• Report back to community on progress (Dec 2016)

Potential Implementation Timeline Beyond Planning

• Pursue an agreed upon voluntary cooperative evaluation plan in 2017 with program/community leader partners syncing on critical points of change we will monitor and report (beginning January 2017)
• Monthly check-in meetings/Technical assistance office hours (with core volunteer working groups) (Jan-Mar 2017)
• Regroup in 2017 at existing convening for working sessions in 2017 (April 2017)
• Report on progress to community (June 2017)

By cooperatively identifying our shared impact story, we can better guide our reporting and storytelling toward shared practices and then better deliver the qualitative impact story that is not available through tracking behavior and metrics online. Together, we can begin to fill in the context and color of our shared strategic work through programs. This context will help us better understand and represent programs as they contribute to the larger forces of the open knowledge movement as a whole through a social cartography of our programs; a capturing the networks built and behavioral experiences on the front lines of our offline work.

Resources

• Outcome Mapping: Building Learning and Reflection into Development Programs
• Outcome Mapping Learning Community (An outcome mapping community of practice)