Jump to content

Grants:APG/Proposals/2015-2016 round 2/The Centre for Internet and Society/Staff proposal assessment

From Meta, a Wikimedia project coordination wiki

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

[edit]

Summary

[edit]

Current (projected)

Upcoming (proposed)

Proposed change (as a +/- percentage)

FDC or PEG funding

$175,080.00 $153,195.00 -13%

Budget

$827,811.02 (CIS full budget) $910,592.12 (CIS full budget) 10% (CIS full budget)

Staff

23 (CIS full org) 24 (CIS full organization) 4% (CIS full organization)

Overview of strengths and concerns

[edit]

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

[edit]
  • Strong performance in the current year indicates potential for future success
  • The Centre for Internet and Society (CIS) offers expertise and capacity to the Access to Knowledge (A2K) program
  • There are significant opportunities for impact in India

Concerns

[edit]
  • Challenging to understand work across both programs and Focus Language Areas (FLAs), particularly metrics
  • Should aspire to more nuanced targets and evaluation plans, especially around skillbuilding
  • Target for images in use may not be achievable based on past performance

Staff proposal assessment narrative

[edit]

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

[edit]

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

[edit]

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

  • Last year we noted that CIS-A2K is well-poised to impact significant global content gaps and to increase participation through its programs focusing on smaller language Wikipedias. We also note that by serving large numbers of people speaking the languages of India and surrounding areas, this in an excellent opportunity to increase the reach of our projects. CIS-A2K has maintained and refined this focus on small language Wikipedias.
  • At the same time, working across so many languages makes program design, implementation, resource allocation, and evaluation a major challenge.
  • Last year we noted that CIS has had success in implementing programs in partnerships with other institutional donors, and has significant non-APG funding that enables its institutional infrastructure in support of its A2K program. The complex legal and regulatory environment in India presents a significant challenge for any organization, yet we note that CIS is experienced and well-equipped to address them.
  • Last year we noted that CIS-A2K has improved in its engagement with local communities. This has also improved in the current year, as CIS-A2K has increased its efforts to systematically document its responses to community feedback, including detailed needs assessments to respond to information received from the community.
  • Distinct from last year, Wikimedia India’s work has been stalled by serious regulatory and governance challenges. Although some emerging informal user groups are becoming more active, CIS is currently the only formal organization doing active Wikimedia program work in India, and the only one able to receive foreign funding. CIS plays a critical support role in the region.

Past performance

[edit]

Will this organization's past performance with program implementation enable this plan to succeed?

At the midpoint of the current grant year, we are seeing good progress toward CIS-A2K’s overall participation goals, which it appears on track to meet or exceed. At the midpoint, CIS-A2K has involved 1,441 individuals of the 2,525 targeted. Achievements with active and new editors at the midpoint seem on track to meet or exceed targets, with 1,580 new editors engaged of 2,065 targeted and 168 active editors engaged of 289 targeted.

CIS-A2K’s achievements in the area of content generation show good potential for future impact in this area. CIS-A2K has supported the creation of 4,231 articles at the midpoint (not including folios) through its FLAs. At the midpoint, Kannada has been the highest performing FLA in terms of articles improved, while Telugu has been the highest performing FLA in terms of images added and active editors involved. Marathi has greatly exceeded the targets set for active editors and total editors involved at the midpoint, and all FLAs are on track to exceed current targets in terms of article count. Without folios, CIS-A2K’s article target for the proposed year is 12,740, which is ambitious and yet appears achievable based on its current progress.

CIS-A2K projects to increase its training activities in the second two quarters of FY 2015-2016, but has achieved 9 trainers conducting outreach events of 25 targeted and 7 editors involved in the train a wikipedian program of 20 targeted. Through content acquisition and language content digitization programs, CIS-A2K has enabled the digitization of 4,130 books at the midpoint, and expects to focus on digitization in last two quarters of FY 2015-2016 in order to digitize 50 more books. Finally, significant resources have been created in each language targeted by an FLA, including videos and handbooks in different languages explaining editing and copyright.

The target for images added to articles of 9,720 in the proposal seems high relative to CIS-A2K’s current achievement of 1,060 images added to articles at the midpoint. This may be because CIS-A2K is expecting its work in Education and GLAM to generate a greater number of images in the proposed year. It is not clear that this is a reasonable expectation of either program, however. Of current APG organizations, only Wikimedia Deutschland has met or exceeded 9,000 images in use. 4,231 articles at the midpoint are impressive when compared with other APG organizations, with only Wikimedia Deutschland, Wikimedia Osterreich, Amical Wikimedia, and Wikimedia CH exceeding 4,000 articles.

CIS-A2K’s longer term goals of achieving healthy and self-sustaining communities are more difficult to track, but meeting its yearly goals in participation and content per language area may be a good indicator of progress.

Organizational effectiveness

[edit]

Will this organization's overall effectiveness enable this plan to succeed?

CIS is a healthy and well-functioning organization, and housing A2K within CIS is a great opportunity for the movement to benefit from this experienced and well-networked organization.

The CIS-A2K team has remained strong during the transition of its leadership. The team is currently being led by CIS’s executive director, and this new structure is working very well. The ED is building capacity within the growing team, which includes a rich array of community and language-specific expertise that should enable the success of this plan. CIS has added a number of Program Associate positions to its A2K team, who have been selected through a community process. Since the A2K team is hosted by CIS, staff growth does not create significant capacity concerns. Furthermore, we believe the staff resources and expertise of the A2K team will enable this plan to succeed.

Last year we noted that CIS-A2K has been responsive to critiques and suggestions from the community and the FDC, and we appreciate that they are continuing to seek out and incorporate feedback. For example, in the progress report they included a detailed table showing how they have responded to the FDC’s and the community’s feedback. CIS-A2K has also been offering support to individuals and groups within the movement in India (for example, helping community members with grant applications, hosting community meetups at the CIS office). CIS-A2K will need to continue to diligently monitor this area and keep a close eye on community dynamics that may be affected by its work with specific language communities, and be especially aware of the impact of hiring staff from the community, which may negatively impact volunteers or community dynamics.

Last year we noted that CIS-A2K excels at documenting results and applying learning, and we have seen them continue to be strong in this area. CIS-A2K has shown willingness to change and adapt its programs and approaches according to feedback and results, and has continued clear documentation of its program work on multiple levels. CIS-A2K also greatly improved in the timeliness of its reports to the movement. We encourage CIS-A2K to grow in the area of evaluation, which it has acknowledged is a challenging area for the team. In the programs section, we have pointed out a few areas where we think CIS-A2K can improve.

Strategy and programs

[edit]

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

[edit]

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan? We appreciate CIS-A2K’s strategic approach to working with communities to achieve results in terms of content and participation. We like how CIS-A2K clearly documents community needs and designs programs according to the needs and desires of the communities it is working with, to tailor language-specific plans and strategies to those communities. We appreciate that working with so many communities at once can be a challenge, considering that each project has its unique ways of working and special challenges and opportunities.

CIS’s A2K program work is clearly aligned with Wikimedia goals around participation, content, and especially reach, as it may bring access to higher quality content to large numbers of people speaking the languages covered by these FLA plans.

We have been monitoring the changes to CIS-A2K’s plans and approaches over time, and believe the A2K program is moving in a consistent strategic direction.

Fitting the CIS-A2K proposal into the FDC process has been an ongoing challenge. This year, CIS-A2K’s approach was to present a focused program plan that is not inclusive of the sum of its A2K activities, which are outlined in its detailed work plan, per focus area. This was done at the advice of a former FDC member. This created some confusion in understanding how its FLA targets related to the programs presented. At FDC staff’s request, CIS-A2K provided some estimates of per program metrics to help staff better understand how programmatic impact was distributed. Working across these multiple axes or ways of organizing CIS-A2K’s programs has been challenging.

Programs

[edit]

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

In terms of specific program performance, tracking becomes more difficult. CIS-A2K has traditionally structured its work (including its budget and evaluation plan and staffing structure) according to its language focus areas rather than the programs presented in the FDC proposal. This makes tracking performance and targets by program area a challenge. At the request of FDC staff, CIS-A2K sent estimates of its targets by program area. According to this analysis, Edit-a-thons are projected to be the most effective program at engaging active editors and producing articles, with targets of 550 active editors and 6140 articles created or improved. The Wikipedia Education Program will be the most effective program at generating images in use, with 4,000 images in use. Other images in use are projected to be produced through other Focus Language Area activities not included in the main program groups. The Wikipedia Education Program will also involve a high number of active editors at 500 active editors. According to the targets provided, the GLAM and Skill Building programs will be less prolific content generation engines.

We see high potential in the Skill Building program, which is designed to build the capacity of individual community members to 1) contribute to Wikipedia; 2) conduct offline outreach; 3) participate in technical work on the projects. We are encouraged by progress in the current year, and would like CIS-A2K to move beyond tracking the number of Wikipedians trained. We would like more details about the specific skills being targeted, and to see CIS-A2K measure skills acquisition or relevant attitudinal shifts in order to better understand the real impact of this valuable work. We also encourage CIS-A2K to set targets in these areas.

While retention is a specific goal of the Edit-a-thons program and a key aspect of CIS-A2K’s overall strategy, we note that metrics around retention are not included in the SMART objectives for this program.

Through its GLAM work and as part of its education program, CIS-A2K is targeting a large number of institutions, all of which seem to be at different stages of partnership. We wonder if pursuing so many institutions at once will hinder CIS-A2K from meeting some of its ambitious targets in the coming year. We hope CIS-A2K is working toward deepening the results from these individual partnerships instead of focusing on the number of partnerships. At the same time, we are encouraged by the documentation that CIS-A2K has been doing in this area, and look forward to learning more.

Budget

[edit]

Is this plan and budget focused on the programs with the highest potential for online impact?

CIS-A2K has had some challenges articulating costs by program area, since they approach budgeting and evaluation by Focus Language Area. This creates a difference in the way they normally budget for its work and the way the proposal to the FDC is presented. At the request of FDC staff, CIS-A2K has provided a per program cost breakdown, which we can use to understand how program expenses are being routed to each program.

CIS-A2K lists its total program cost at around $69K, with Skill Building and GLAM the most expensive programs at about $19K each, followed by Edit-a-thons at $15K and Education at $14K. While metrics for the skill building program need to be refined, this program has good potential for impact.

Staff are assigned to Focus Language Areas rather than programs, and so it is more difficult to precisely map staff and operating expenses to specific programs. Overall, staff expenses at nearly $100K or 36% make up a large portion of total A2K expenses, and operational expenses make up 38%. Most of these operational expenses are specific to A2K work, as only 11% overall will go to support CIS’s general operating expenses. We have asked CIS for more clarification on the nature of these A2K-specific operating expenses, to better understand the definition they are using and determine whether more details are needed.

CIS is reducing its reliance on APG funding while expanding the organization’s emphasis on A2K. CIS-A2K requesting a $22K reduction in its grant request, amounting to a change of -14%. Including in-kind contributions, they are matching APG funding at a rate of 80%, with requested APG funding making up 56% of its A2K budget. Beyond this, housing the A2K project with CIS provides a great deal of expertise, infrastructure, and reputational advantage within India and internationally.

A2K’s approach is staff intensive, although staff costs are relatively inexpensive in India. CIS is able to take on the burden of housing A2K staff within its organization, thus minimizing capacity concerns that could be raised by having such a large team.

CIS has robust financial systems in place, and a history of frugal budgeting and spending.

Summary of expert opinions (if applicable)

[edit]

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

[edit]

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

  • Major strength
  • Strength
  • Neither a strength nor a concern
  • Concern
  • Major concern

Criterion

Assessment

Description

Program design

P1. Strategy

Strength CIS-A2K's programs are well aligned with movement priorities of increasing participation, reach, and quality, and CIS's approach of Focus Language Areas is focused on online impact. Priorities are well-explained in the accompanying workplan, and this includes information about how decisions about priorities were made.

P2. Potential for impact at scale

Major strength Through the Focus Language Area approach, programs could lead to significant impact and correspond to the funding amount requested (which is less than last year). Targets for articles in the current year are ambitious and yet achievable at 12,740 (not including folios). CIS-A2K's focus on building capacity within its communities may yield long term results. We can continue to learn from CIS-A2K's well-documented work with smaller language communities and projects. CIS-A2K's past performance and outstanding performance as a learning organization make this plan more likely to achieve impact. Finally, opportunities for impact in India are immense.

P3. Objectives and evaluation methods

Strength Each Focus Language Area has a clear and detailed plan for measuring results. Each program includes SMART objectives and targets, and logic models. CIS works in a complex program environment, with its focus on 5 languages and several programs that operate across these language communities, which presents an evaluation challenge. The CIS A2K team is growing their capacity in this area, to more effectively address these complexities in coming years.

P4. Diversity

Strength Programs will expand participation and reach through support to underserved language areas in our movement, encouraging more people to contribute to Wikimedia projects in their own languages, making content available to large numbers of people in relevant langauges, and making important resources and content available to the Wikimedia movement and world at large. CIS-A2K has done some work on the gender gap, which is included in its Odia Focus Language Area Plan, and we encourage CIS-A2K to continue to develop a gender-focused lens for their work.

Organizational effectiveness

O1. Past results

Strength CIS-A2K is on track to meet and exceed its targets in the current year, and its midpoint achievements compare favorably to other APG organizations. Highlights include 1,060 images in use, 4,231 articles created or improved, and 168 active editors engaged at the midpoint. Given the continued FLA approach and strong capacity of the CIS-A2K team, these achievements are likely to predict future success.

O2. Learning

Major strength CIS-A2K is addressing risks and challenges effectively, is learning from and documenting its experiences on wiki, and is applying learning to improve its programs. CIS-A2K has included detailed needs assessments summarizing what it has learned through consultations with the community, and has used these to show how changes have been made to each of the Focus Language Area plans and has also documented complaints and feedback from the community as well as their responses. CIS-A2K has shown flexibility in adjusting its overall program strategy (see the current year's reorgnization) as well as specific program details in order to hone their results as they work. For example, CIS's partnership team has worked to document the results of each partnership they pursue through case studies, and use this learning to adapt their partnership strategies.

O3. Improving movement practices

Neither CIS has improved in its ability to learn from and share with the broader Wikimedia Movement, and is working on building and improving relationships with other movement organizations and volunteers and staff in the international community, in order to facilitate more learning and sharing at the movement level. CIS is making an effort to become more engaged in international forums (such as mailing lists and wikis) in order to make its work more visible and to better understand the work of others and integrate this learning into their own plans. At the same time, as a non-Wikimedia organization, this will continue to be a challenging area that will require CIS's attention and dedication.

O4. Community engagement

Strength CIS-A2K has done foundational work to engage communities and volunteers in planning through their needs assessments in each Focus Language Area, as well as by asking the community to make hiring decisions for the Program Associates and systematically addressing comments and complaints from the community. Focus Language Area plans are tailored to the specific needs and preferences of the communities they serve, in order to engage these communities through this work. CIS-A2K provides support to individuals and emergent groups in India who are interested in developing their activities independently of CIS-A2K (for example, by applying for grants from the Wikimedia Foundation or hosting community meetups at their office space).

O5. Capacity

Major strength CIS is a strong organizational backbone for A2K, and has the resources and expertise to do the plan proposed. During last year's leadership transition, the team has maintained strong leadership through CIS's ED and has been increasing the leadership capacity of its growing team. CIS-A2K has actively been working to build their capacity in community engagement, and is working to build the team's capacity in evaluation. Housing A2K within CIS provides additional benefits, such as a strong organizational reputation in India and internationally.

Budget

B1. Past budgeting and spending

Strength CIS has a history of budgeting realistically and managing funds effectively in the past. Housing the A2K project within CIS offers the project significant expertise and human resources in the areas of accounting and financial management, including access to significant expertise with navigating the complex regulatory environment in India.

B2. Budget is focused on impact

Major strength Based on CIS-A2K's performance and current targets, as well as a 12% decrease in movement funds requested, funds are allocated to programs and activities with corresponding potential for programmatic impact. The bulk of program expenses are devoted to edit-a-thons and skill building, at around $19K each. While metrics for skill building programs need to be improved, we believe this is a high potential area.

This staff proposal assessment is the work of FDC staff and is submitted by: Winifred Olliff (WMF Program Officer) talk 23:29, 9 May 2016 (UTC)

Staff proposal assessment framework

[edit]
  • Major strength. This is something the organization does very well, and this is a strong indicator of future success.
  • Strength. This is something that the organization does well, and this could indicate future success.
  • Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
  • Concern. This is something that the organization does not do well, and this could make funding the application a risk.
  • Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion

Description

Program design

P1. Strategy

The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.

P2. Potential for impact at scale

Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested

P3. Evaluation methods

Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.

P4. Diversity

Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.

Organizational effectiveness

O1. Past results

This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.

O3. Learning

This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.

O4. Improving movement practices

This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.

O5. Community engagement

This organization effectively engages communities and volunteers in the planning and implementation of its work.

O6. Capacity

This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.

Budget

B1. Past budgeting and spending

This organization has a history of budgeting realistically and managing funds effectively in the past.

B2. Budget is focused on programmatic impact

Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.