Grants:APG/Proposals/2016-2017 round 1/Wikimedia Argentina/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or other relevant funding	$232,500	$241,140	+3.72%
Budget	$216,170	$241,140	+11.55%
Staff (FTE)	3.5	4.0	+14%

Overview of strengths and concerns

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

• WMAR is developing systematic ways of measuring content quality.
• WMAR’s online course has potential to scale.
• WMAR is achieving improved results in important areas like articles improved.

Concerns

• This complex program plan is difficult to implement and evaluate.
• Community support work is not linked to online impact.

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

• WMAR can impact Spanish language Wikimedia projects, including Spanish Wikipedia, which is a large and growing project that can benefit hundreds of millions of people worldwide. Through their position in Argentina, they can increase the diversity of contributors to and content on Spanish-language Wikimedia projects, to reflect Latin American knowledge and expertise.
• WMAR is one of the most well-funded and established chapters in Latin America. Despite this, we have not seen them develop their leadership role in Iberocoop at the scale we had hoped, nor has the Iberocoop group been successful in generating activities or coordination in the region in the past year.
• WMAR experiences a duality in their interaction with the Spanish language Wikimedia community in Argentina: the people who are active online and offline seem to function like two different communities. WMAR is working to bridge that gap, which can present a challenge in implementing projects that link online and offline activities, such as education or GLAM program work.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

• WMAR’s complex approach to program design, implementation, and reporting, makes assessing their past results challenging.
• For 2016, WMAR set global metrics targets that were reasonable based on mid-2015 achievements, and exceeded expectations in most areas by mid-2016 (e.g. 357 active editors and 6,579 articles), with the exception of media in use (e.g. 56 media in use).
• WMAR developed new tools to assess quality of content in 2015, and began to implement them in the first 6 months of 2016, including a goal for quality articles in its education program (target of 100 quality articles, with 184 quality articles achieved by the midpoint of 2016). This will be an interesting area to track as WMAR moves forward with this quality emphasis.
• Some past activities such as “La mujer que nunca conociste 2016” work on content generation (1,262 articles improved or created) and address gender-related content gaps, with an emphasis on quality.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

• WMAR’s complicated plans raises concerns about the organization’s effectiveness, due to the time spent designing, implementing, and evaluating using this time-intensive approach. It appears that this approach may also be impeding the engagement of stakeholders in the plan’s development. This continues to be a concern, year over year. WMAR has good potential to design thoughtful plans, and clearly has an intent to emphasize learning. Sadly, important lessons learned and overall sense of priority is lost in the complexity of the current approach.
• WMAR continues to document its learning, and draws links from what they have learned to changes they have made to their program plan for 2017. While we appreciate this thoughtful approach and see that learning has been applied in some cases, WMAR sometimes repeats approaches that don’t seem to be effective. For example, WMAR has received feedback about their complicated plans and reports for several years, and has continued this approach year after year. Another example is WMAR’s continued emphasis on training teachers to become regular Wikipedia editors.
• WMAR is tracking increased volunteer involvement in its activities. Developing volunteer leaders, and having a strategy around leadership development, may be important to the long-term sustainability of WMAR, and we are glad to see this included as part of the community support program. Beyond this, it would be good to see clear integration of volunteer leadership roles into other aspects of WMAR’s work, like GLAM and education. Engaging online contributors will also be critical to WMAR’s future success.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan?

• WMAR’s two-year strategic plan covers 2017 and 2018. WMAR’s strategic plan emphasizes community participation, diversity, content, and partnerships and describes lines of actions for content, promoting free culture, the movement, and diversity through themes. It includes an analysis of risks and opportunities and maps various programs to strategic lines. While specific metrics are outlined for various programs, the plan does not include long term targets.
• Despite the program-specific information included in the strategic plan, WMAR’s complex approach in developing its annual plan using cross-cutting issues makes it difficult to understand how WMAR aligns its programs within its strategic framework, and dilutes WMAR’s strategic focus.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

• While WMAR does not have a specific program that focuses on diversity, addressing gender-related content gaps is addressed in their programs and approaches. One example is the editing challenge “La mujer que nunca conociste.” WMAR is also supporting an offline group of women, and have started to work with like minded groups in that area (Un Pastiche, Economia Feminista, Chicas Poderosas). This is particularly important, as these outside groups can offer expertise that the Wikimedia community does not have in working on gender issues.
• Online courses and activities have generated article achievements. WMAR has been doing foundational work in this area, and this emphasis will continue in 2017. The Wikilesa project is an interesting opportunity for WMAR to combine its expertise in education with its interest in partnering with human rights groups.
• WMAR is continuing its focus on training teachers to edit Wikipedia, although most education programs have shown that this is not an effective approach. WMAR is extending its support to activities that are occurring outside the classroom, applying learning from other models that work with youth, such as Wikimedia Armenia’s.
• In GLAM, we appreciate that WMAR plans to increase their emphasis on sister projects like Wikidata and Wikisource. WMAR is trying out an itinerant Wikipedian in Residence model, with an emphasis on digitization, to palliate the fact that Argentinian Cultural institutions cannot afford to fund a dedicated resident, which is an interesting new model.
• There is no clear link between WMAR’s community support work and online impact, as online impact is not emphasized in any of the many metrics provided for this program.
• WMAR includes many metrics in its plan. Some are thoughtful and highly relevant, while others are not useful. The sheer volume of metrics will make implementing and measuring this plan a challenge, and the complex design, implementation, and measurement may not be an effective use of WMAR’s resources. We have seen the results of this approach in WMAR’s reports, which do not effectively tell the story of WMAR’s important achievements.
• WMAR has outlined three qualitative metrics, and two related quantitative metrics that serve as their grantee-defined metrics. These include quality, measured through WMAR’s newly developed assessment tool, and a sustainability metric that looks at the ratio of new activity generated by past partners in relation to WMAR’s investment in new partners. In terms of qualitative emphasis, WMAR is emphasizing quality, gender, and diversity.
• Based on current achievements, WMAR’s target for 33,325 pages (includes Commons uploads) may be low, while their target of 2100 may be ambitious.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

• WMAR explains that the unstable financial situation of the country has made it challenging to plan and budget accurately each year, as exchange rates between the US dollar and the Argentine peso fluctuate rapidly, and because inflation is continuing to increase. As a result, spending patterns are difficult to track.
• WMAR has not provided a breakdown of staff expenses by program, but non-staff expenses are focused in the community support program ($34,500), which is likely to be the least impactful of Wikimedia Argentina’s three program areas. In terms of staff resources, however, WMAR has a full-time employee dedicated to the education program, while no other program area has a dedicated staff resource.
• WMAR is requesting to increase their staff by 0.5 FTE in 2017, for a total of 5 staff and 4.0 FTE; this is the addition of a part-time assistant. WMAR’s staff is heavily focused on operations, with a fulltime communications manager, a part-time administrative manager, a part-time staff assistant, and a fulltime ED. It is not clear that staff growth is leading to commensurate results.
• We are pleased to see WMAR tracking and reporting on in-kind donations. WMAR is projecting a value of $16,162 to be received in in-kind donations during 2017.
• According to current values, WMAR’s reserves will cover about 3 months of staffing expenses in an emergency.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Neither	WMAR has a strategic plan in place, and programs are aligned with this strategy in principle; yet, it is difficult to decipher strategic alignment in this convoluted plan. WMAR's strategic approach is strong, but its weakness lies in its implementation.
P2. Potential for impact at scale	Neither	Some programs, like the online course, may have potential to scale, and WMAR may have impact on Spanish Wikipedia. Other programs, like community support, are not clearly linked to results at scale. While results have improved significantly, they are still not at a scale that is commensurate with funding requested.
P3. Objectives and evaluation methods	Neither	WMAR has put significant effort into including an evaluation plan for each of its programs, including SMART objectives and many relevant targets with baselines. Despite this effort, this evaluation plan contains too many metrics for tracking at the program level, which raises concerns about the organizational effort it will take to implement this evaluation plan and inhibits good storytelling. The negative effects of this approach are evident in WMAR's reports and proposals, which are rich in information but lack a clear sense of priority and strategic direction.
P4. Diversity	Strength	WMAR is positioned to increase content related to Latin America, as well as increase contributors in Argentina. WMAR emphasizes the potential of their education program to increase diversity through attracting female contributors; while this may lead to increased amounts of content from female contributors it is unlikely to lead to results in terms of retaining female contributors. Gender is referenced in their content-focused programs, and some specific plans are described. They also emphasize gender in their qualitative metrics.
Organizational effectiveness
O1. Past results	Neither	WMAR's complex evaluation approach makes past results difficult to assess. WMAR's results have improved in the past two years, and their progress report shows them to be on track for 2016 in some areas and lacking in other areas (such as images in use for example). Despite improvements, results are still low with respect to funding received in the past.
O2. Learning	Neither	WMAR is documenting its own learning, but is not consistently incorporating feedback from other sources into its approaches.
O3. Improving movement practices	Neither	While positioned to be a leader in the Latin American region, and willing to participate in regional collaboration, regional efforts (e.g. WMAR's role in Iberocoop) have not progressed. WMAR has some good one-on-one relationships with other chapters, but we do not see evidence that this has resulted in improved practices.
O4. Community engagement	Neither	After conducting their first community survey, WMAR has adjusted its strategies to include contributors that are not currently active in the chapter, but they have ongoing challenges engaging online contributors. WMAR had limited participation from the community in the development of their proposal. They have included plans for leadership development in this year's proposal, although the role of volunteer leaders is not clear across all programs.
O5. Capacity	Concern	Last year we wrote that we are concerned about the feasibility of this complicated plan, particularly given the amount of evaluation and tracking required to complete it satisfactorily. This continues to raise capacity concerns, considering the resources needed to create, execute, and evaluate this plan.
Budget
B1. Past budgeting and spending	Neither	WMAR faces major challenges with accurate budgeting due to fluctuating exchange rates and inflation. Considering this, WMAR has done a good job budgeting effectively and spending on target. In the past, funding has not led to commensurate impact, although results are improving.
B2. Budget is focused on impact	Concern	We are concerned with the use of staff time to create, execute, and evaluate this complex plan. Staff expenses are not directly linked to highest impact areas, and neither are non-staff program expenses. We continue to have concerns that impact will not be commensurate with funding.

This staff proposal assessment is the work of FDC staff and is submitted by: Winifred Olliff (WMF Program Officer) talk 21:28, 8 November 2016 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O2. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O3. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O4. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O5. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.