Grants:APG/Proposals/2016-2017 round 1/Wikimedia Serbia/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or other relevant funding	$125,398	$145,094	+15.71%
Budget	$105,562	$145,094	+37.45%
Staff (FTE)	5	5	0%

Overview of strengths and concerns

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

• Education program has potential to scale at the national level.
• WMRS is supported by a strong volunteer base deeply involved with their work.
• WMRS includes unusual and culturally relevant themes in their media work.

Concerns

• Large grant increase is not tied to impact or spending patterns.
• This list-like annual plan does not demonstrate good implementation of strategy.
• This plan lacks relevant metrics in quality and community engagement.

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

• Wikimedia Serbia has achieved national-scale success through integration of Wikipedia into Serbia’s national curriculum, and through collaborations with high-profile GLAM partners like University Library Belgrade. This is no small achievement in Serbia, where some GLAM partners may be less technologically savvy or resource-rich than in other parts of Europe, and where successful institutional partnerships may be highly dependent on the political environment. Internet access throughout Serbia is by no means universal, which may be an additional challenge in working with new and existing contributors.
• Wikimedia Serbia continues to be a community-driven organization at heart, and volunteers are central to to all of their program activities. WMRS pays close attention to developing volunteer leaders in their small community, with a personal approach.
• WMRS relies primarily on APG funding for its revenue, for both core operating expenses and programs. They have not prioritized local fundraising, which would be difficult in their context, but this may be something they need to consider if they have plans to grow.
• As a well-funded and well-staffed organization in Eastern Europe, WMRS has opportunities to lead in the CEE network, yet we are not seeing these fully realized.
• We do not yet see evidence that they are exploring opportunities to target gender-related work.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

• After succeeding in getting Wikipedia integrated into the national curriculum for secondary schools, WMRS succeeded in 2016 in establishing an accredited seminar to train teachers in using Wikipedia in the classroom. This important achievement has the potential to lead to impact on a national scale.
• By the middle of 2016, WMRS saw growth in its WLE photo competition from 831 to 1749 photographs, and their programs resulted in 5473 media uploaded to the Wikimedia projects, with a high proportion of 932 images in use by the end of 2016. They are not on track to meet their target of 5060 images in use by the end of 2016, however. This may indicate that WMRS is somewhat successful in generating useful content, although less successful than they planned.
• In 2016, WMRS achieved 3,506 articles added or improved to Wikimedia projects at the midpoint (not including articles that were improved solely by adding images), 2,280 of which were added as a result of Wikipedia in Schools. WMRS does not assess the quality of articles created through its education program work. Including articles improved by adding images this number is likely to be closer to 8,000 - 9,000 articles.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

• Wikimedia Serbia has been working over time to refine its education initiatives, and has adapted learning from Wikimedia Armenia (e.g. work with youth clubs) to make its program plan for the current year.
• WMRS has a dedicated and committed volunteer board, as well as a pool of core volunteers that actively lead, participate in, and support the chapter’s work. They derive their core capacity from their ability to engage with this active volunteer community on an ongoing basis.
• WMRS has a large staff relative to the size of their community. While it makes sense to have a dedicated staff person for the education program, which is a major focus of the organization, they also have significant staff resources in administration and communications. This lends their organization the capacity to maintain good financial systems for an organization of their size, and also to function outside of the traditional ED-led model that would be typical for a staff of this size.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan?

• WMRS has a [strategy plan in place https://rs.wikimedia.org/wiki/Wikimedia_Serbia_Strategy_2015-2017] until 2017. The strategic plan does not include long-term goals or a detailed analysis of risks and opportunities.
• This plan emphasizes partnerships, education work, and free content. While these areas are well-aligned with those in the proposal, mapping them can sometimes be difficult given the volume of activities described in the proposal. From reading the proposal, it is sometimes difficult to understand the relative important (in terms of strategic importance and resources) of the different activities described.
• WMRS did not mention specific plans to develop a strategy for 2018 and beyond in the current proposal, but we hope this will be an important focus next year. If WMRS continues in the FDC process, and especially if they request to grow their budget size, standards for strategic planning and implementation will increase.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

• In education, WMRS has fostered the growth of student ambassadors into program leaders. WMRS’s long-running education program is continuing to grow each year. In 2017, they are focusing on training teachers to use Wikipedia in the classroom through an accredited seminar.
• GLAM work is innovative, because it focuses on unusual GLAM partnership opportunities, work with galleries, theatres, and academic libraries. Digitization of techno culture is an ongoing initiative that may provide content in culturally relevant areas. Many projects include a colorful focus on topics like tombstones and magic trees, as well as interesting off-the-ground exploration of photography perspectives through underwater or aerial photography, and an exploration of seasonal photography projects.
• While the Wiktionary project was started by one dedicated volunteer, this work has yielded excellent results across several languages and includes collaboration with the Faculty of Philology and University Library. This is an example of WMRS’s approach of supporting volunteer-driven initiatives can lead to projects that grow and thrive.
• WMRS has set targets for 2017 that seem rooted in their ability to achieve, although they are not projecting to improve performance significantly over the current year in articles improved. Some targets are useful metrics for their work, such as their measure of images in use (included as a grantee-defined metric) or articles created and improved, while others, such as number of events (included as a grantee-defined metric) focus on outputs rather than outcomes. For example, while usage of photographs is a highly relevant metrics for WMRS due to this focus, number of events is unlikely to tell the story of what was achieved through these events, as this does not give us a good idea of event outcomes or serve as a good proxy for community engagement, organization activity, or other outcomes we could identify.
• In some areas, important metrics seem to be lacking. With this emphasis on education, we did not see an accompanying emphasis on or attempt to measure the quality of student work, which could be important to a deeper understanding of WMRS’s results and plans. Community support work in general lacks metrics that could tell us whether or not WMRS is successful in its efforts to grow and “reanimate” its communities.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

• WMRS has been increasing its grants year over year since entering the FDC process, but it is not clear that grant increases are leading to impact that is commensurate with funding. The $40,000 increase requested in 2017 is distributed across program areas, and does not appear to be linked to improved outcomes in any specific area. Expenses like printing and materials ($9,000) and travel costs ($8000) would be more scrutinized as part of the Simple APG process to clearly establish their link to impact. This process may offer other benefits for WMRS, since they are still developing their ability to implement strategy and spend on target.
• WMRS spent less than planned in 2014, 2015, and 2016. They project to be 85% spent by the end of 2016.
• WMRS depends entirely on APG funding. It is good to see that WMRS is listing in-kind donations. We encourage WMRS to develop a system for quantifying in-kind donations to more clearly show the value of these contributions. This is especially important in an environment where local fundraising might be difficult.
• WMRS maintains minimal reserves, at about $4,500.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Neither	WMRS has a strategic plan in place, but it is difficult to understand how this strategy has been implemented in its annual plan. This annual plan reads like a list of projects and activities without a coherent strategic vision, and it is difficult to understand what activities have the most strategic importance and why. This becomes more problematic as WMRS requests to increase its funding.
P2. Potential for impact at scale	Neither	WMRS is requesting an increase in funding, but we do not see targets increasing at the same scale. Education work could have impact on a national scale, although it seems unlikely to scale beyond Serbia's borders.
P3. Objectives and evaluation methods	Neither	Programs have metrics that are a mix of relevant outcomes-focused metrics and less relevant outputs-focused metrics. This is also reflected in WMRS's selection of grantee-defined metrics. The plan will be a big challenge to evaluate due to its length and lack of strategic distinction. There is a lack of a quality approach in the education focused work, although it is there in the media-focused work. There aren't any effective ways of measuring the impact of WMRS's community support work. They have a clear plan in place including a tracking sheet they have shared that shows metrics across all programs.
P4. Diversity	Neither	WMRS are located in a geography with an emerging community, but do not have programs or metrics that focus on diversity.
Organizational effectiveness
O1. Past results	Neither	In 2015 and 2016, WMRS did not meet or is not on track to meet targets for images in use, while performance in terms of articles has been improving. Some qualitative achievements, like setting up the accredited course, are good.
O2. Learning	Neither	We have not seen sufficient improvements in this area since WMRS joined the FDC process. Reflections are included in WMRS's proposal and reports, but are sometimes superficial. The list-like nature of WMRS's plans and lack of strategic approach make systemic reflection and quality reports more challenging.
O3. Improving movement practices	Neither	Last year we noted that despite significant staff resources, WMRS has stepped back from taking a leadership role in regional CEE work. Expectations in this area increase with funding requests.
O4. Community engagement	Strength	WMRS is a community-centered organization, that works with community members to design and implement its projects and activities. There is a healthy overlap among volunteers and online contributors. WMRS is making good progress engaging volunteer ambassadors through its education work, but with this funding increase we would like to see a deepening approach to sustaining community engagement over time.
O5. Capacity	Neither	The lack of strategic direction or prioritization in this plan will make it difficult to implement and evaluate. While WMRS's budget is increasing, staff capacity is not increasing. While active volunteers enable WMRS's program work, some projects run the risk of having a single point of failure.
Budget
B1. Past budgeting and spending	Concern	WMRS has a pattern of underspending year over year, despite steady revenue. Growth in staff and budget has not led to commensurate growth in impact.
B2. Budget is focused on impact	Concern	Staff resources target the education program, which is one of WMRS's high potential impact areas, but it is not clear how the significant grant increase requested will lead to more impact. Targets set for 2017 are low when compared with 2016 achievements, and amounts for travel, promotional materials, and printing, are distributed across program expenses, and may not be directly linked to improved outcomes.

This staff proposal assessment is the work of FDC staff and is submitted by: Winifred Olliff (WMF Program Officer) talk 21:28, 8 November 2016 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O2. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O3. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O4. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O5. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.