Grants:APG/Proposals/2016-2017 round 2/The Centre for Internet and Society/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or other relevant funding	$157,500	$180,000	+14.29%
Budget	$160,500	$186,810	+16.39%
Staff (FTE)	9	11	+22%

Overview of strengths and concerns

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

• CIS-A2K operates in a region with high potential for involving new editors and readers in the Wikimedia movement.
• CIS-A2K excels at documenting learning and adapting its programs.
• This plan emphasizes linguistic, geographic, and gender diversity.

Concerns

• It is good that CIS-A2K is describing their work using program verticals, but difficult to map targets and activities across the verticals and FLAs.
• While it is good that CIS-A2K is formulating a global initiative, this program is still too speculative to be impactful.
• CIS-A2K is operating in a complicated environment in India, and criticism from some members of the communities in India is an ongoing concern.

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

CIS-A2K operates in a country with an enormous potential for increase in active contributors and readers of the Wikimedia projects. India’s diversity is only partly and still inadequately represented in the Wikimedia projects, and CIS-A2K is an important agent for increasing this representation.

CIS-A2K has historically focused on specific languages, but this year's annual plan shows that they intend to support communities in more language areas. CIS-A2K faces ongoing criticism in some communities but continues to document thoroughly its interactions with active editors and stakeholders, and to seek community input to develop their programs and activities.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

CIS-A2K continues to have good results and achieve its targets, year over year.

Participation in CIS-A2K programs has already been exceeded at the 2015-16 midpoint with a result of 2,688 participants for a target of 2,643 participants, demonstrating significant involvement from communities in their activities. CIS-A2K has achieved 878 new editors involved at the midpoint, including more than 400 new editors participating in gender gap programs, a result which already exceeds their performance in 2014-15.

CIS-A2K is also delivering good results in the area of content. Digitization work has yielded more than 3,700 proofread folios added to various Wikisource at the midpoint, which makes this upcoming year’s target of 5,000 proofread folios almost conservative. With 5,405 articles improved at the midpoint, their content-focused activities have delivered results which support their ambitious target of 12,754 articles in the current year.

Past performance indicates that CIS-A2K has the potential to deliver on the targets it has set for itself in the proposed year, at least in the areas it has measured in the past.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

CIS is an effective organization, with a proven track record of managing funding and achieving results. The integration of the CIS-A2K team, which is focused on delivering programs associated with the Wikimedia mission, within an effective mission-aligned organization like CIS, is a benefit for the Wikimedia movement. It also grants the movement access to an important network of like-minded organizations and people.

The CIS-A2K team has been evolving to include in a mix of people with diverse skills. Most staff focus on language areas and are bringing different program skills. The leadership of CIS has helped the A2K team to grow and flourish, with opportunities for internal growth and leadership development. The reorganization of the team to include a program manager, program officers and community advocates, puts CIS-A2K in a good position to deliver the ambitious annual plan it has designed.

The complexity of CIS-A2K’s plan presents a challenge in terms of evaluation and implementation, as well as effective communication within the team. While these approaches are still being refined each year, we have observed consistent improvements in these areas.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan?

CIS-A2K has a brief strategic plan spanning 2016-2018 with four priorities and three strategies. The plan lacks a detailed risk and opportunities assessment but lays a foundation for implementing good strategies that serve these four priorities: new editors, existing editors, content, and skill building. It is good to see the emphasis on leadership development included in the strategic plan.

CIS-A2K’s proposal is accompanied by an annual plan that describes the many different approaches to the work they are doing. Due to the complexity of this plan, it is sometimes difficult to map approaches and outcomes across verticals and focus language areas. Some of the verticals included in this plan are well-aligned with CIS-A2K’s strategy, while others lack depth and real potential for implementation.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

CIS-A2K has developed relevant and innovative grantee-defined metrics. The Footprint Tracker will help CIS-A2K measure their geographical reach and diversity of communities supported through events, bringing in a good measure of the breadth of their impact. The deleted bytes metric might be much more difficult to implement, but it shows a necessary focus on quality of the content supported by their activities.

There is some confusion in CIS-A2K's plan as to the division of their work into programs and verticals. The 4 main programs and the 5 verticals are unevenly developed. Some overlap, others do not fit in with the whole, but others, like the Bridging the Gender Gap Initiative, are well-structured. We acknowledge that CIS-A2K has been working on improving the way they present their work for several years, and we see that this plan is a step in the right direction. CIS-A2K can clarify the integration of their annual plan and work plans to make planning and reporting more effective and less time-consuming.

Skill building activities address community needs and will benefit the Wikimedia projects. These activities include train the trainer programs that invest in leaders with the potential for long term engagement in the movement, and Mediawiki trainings offer technical skills to communities that have identified this need. It is not clear what the relationship between this program and the vertical "Skill building Initiatives" is, or why it makes sense to have a program and a vertical that are describing the same approach.

The inclusion of GLAM and Institutional Partnerships as separate programs is confusing, as is the fact that the Institutional Partnerships Program is referred to throughout the documents as the Education Program (WEP). There is also some overlap between the GLAM program and the Content Enrichment Activities program. There may be good opportunities in GLAM, especially digitization work and copyright release. Education is a more problematic area in India, as there are often conflicts between community members that want to do more education program work and those who have concerns about poor quality and plagiarism having a negative impact on the Wikimedia projects. While these are both interesting work areas, they may not be as impactful as CIS-A2K’s other community-focused approaches.

CIS-A2K has done useful background work to analyze and assess the opportunities and challenges in the area of gender gap work. This work focuses on improving the environment and practices, which may lead to sustainable results. For example, work on policy and a code of conduct, a training session for sensitization on Gender gap issues, and collaboration with local groups. More gender balance in the staff would be a good next step!

We are conscious that becoming more integrated with the wider Wikimedia movement has been a request from the FDC process over the past few years, and we are grateful that CIS-A2K has started this approach of looking up potential programs to experiment or adapt in their geography, but the plans articulated in the global engagement initiatives vertical are not yet concrete or realistic. We are looking forward to working with CIS-A2K to develop some of those programs and ideas.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

CIS is a frugal organisation, with impressive results that are commensurate with the funding they have received. CIS-A2K budget shows that the bulk of their expenses goes to programs, followed by a large portion of their annual plan's budget focused on staff, which may be a good investment considering that they are operating in a very large country with a great deal of linguistic and cultural diversity, and that their plan calls for a variety of skills needed to implement their programs. Staff costs in this year's proposal have also been significantly reduced from last year's proposal (from about US$102,000 to about US$85,000 with community associates), which shows their reorganization is focused on lean but adequate staffing.

One of CIS-A2K’s largest budget line items, "skill building," pertains to one of their most innovative, successful and interesting programs, which has a high potential for impact. Digitization program costs are linked to high impact in the indic languages projects. Overall, CIS-A2K is a focused and impact-driven team, and their budget reflects that.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Strength	CIS-A2K has a clear strategic plan in place which serves as the backbone of their annual plan. Programs and activities are aligned with strategic priorities and strategies.
P2. Potential for impact at scale	Major strength	Working in India with several languages that represent large numbers of speakers, CIS-A2K has great potential for impact at scale. Their development of customized approaches for specific Focus Language Areas and their pioneering work in areas like skill-building are likely to achieve good results, which may also benefit the movement at large.
P3. Objectives and evaluation methods	Strength	CIS-A2K has defined innovative and thoughtful grantee-defined metrics, which may also be a challenge to monitor. CIS-A2K is meticulous about measuring and documenting the results of their work.
P4. Diversity	Major strength	CIS-A2K has designed their gender gap work in a thoughtful way, and it is generating results in the current year. The upcoming plan emphasizes this area, and also includes work with a diverse range of language communities. CIS-A2K's organizational culture also fosters an atmosphere of diversity and inclusion.
Organizational effectiveness
O1. Past results	Strength	CIS-A2K is likely to meet or surpass many of its targets in the current year and has also performed well against its targets in past years. Their past performance indicates that they will be able to achieve their projections. CIS-A2K succeeds in generating significant amounts of content in many different languages, as well as engaging new and existing contributors with its work.
O2. Learning	Major Strength	CIS-A2K continues to document their learning, and strives to integrate community feedback into their program design. In a few cases, however, CIS-A2K could be more thoughtful in the way that they apply learning. For example, CIS-A2K has attempted to respond to some feedback by developing its Global Engagement Initiatives, yet this program is still largely speculative. There have also been issues with the implementation of education program work in India, which CIS-A2K is addressing by thoughtfully reflecting on next steps.
O3. Improving movement practices	Neither	CIS-A2K is developing stronger connections and showing interest in other initiatives in the Wikimedia movement. We hope they continue their good work in this area. The Global Engagement Initiative would benefit from being developed into a more concrete plan.
O4. Community engagement	Strength	We continue to appreciate CIS-A2K's proactive engagement with communities through wishlists and requests for feedback. They have shown patience and dedication in integrating this feedback in their plans. We hope they continue their good work in this area, as maintaining good relationships with all of the communities they work with is likely to be an ongoing challenge.
O5. Capacity	Major Strength	The CIS-A2K integration within CIS continues to be a strength. The reorganisation of the CIS-A2K team is likely to lead to good results, and we appreciate the good work of the CIS-A2K staff team. We see that the staff team is growing in their leadership and expertise. The relationship between verticals, Focus Language Areas, and programs may still need to be clarified.
Budget
B1. Past budgeting and spending	Strength	While financial progress throughout the year is sometimes difficult to follow, CIS has always been a frugal organization, and past spending has been focused on direct impact on the projects.
B2. Budget is focused on impact	Major strength	CIS-A2K has developed a plan and budget that ensures maximum focus on results, backed with an adequately sized staffing plan. The integration of CIS-A2K within CIS also increases CIS-A2K's ability to focus funding on impactful areas.

This staff proposal assessment is the work of FDC staff and is submitted by: Delphine (WMF) (talk) 18:54, 8 May 2017 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O2. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O3. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O4. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O5. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.