Grants:APG/Proposals/2017-2018 round 2/The Centre for Internet and Society/Staff proposal assessment

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview[edit]

Summary[edit]

Current (projected)

Upcoming (proposed)

Proposed change (as a +/- percentage)

FDC or other relevant funding

$180,000 $223,334 24.07%

Budget

$186,810 $244,735 31%

Staff (FTE)

9.0 12.0 33.33%

Overview[edit]

This section summarizes the themes that emerged from the proposal, which are explained in the detailed narrative section of this assessment.

Themes[edit]

  • CIS-A2K is adapting approaches in a contextually relevant way, including increasing their focus on supporting indic languages communities and capacity building.
  • CIS-A2K has a strong focus on diversity but less on gender gap than in the past. A stronger focus on gender issues would be interesting in their context.
  • CIS-A2K has generated strong performance year over year, and is improving at setting targets accurately.

Staff proposal assessment narrative[edit]

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness[edit]

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment[edit]

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

CIS-A2K operates in a country with great potential for growth, as we noted last year. The multiplicity and diversity of the languages of India and the sheer potential for contributors in India is a good foundation for CIS-A2K’s operations and activities.

In this proposal, CIS-A2K continues to broaden its focus in India from specific focus language areas to supporting activities that span many languages, while maintaining a customized approach to work in each language. CIS-A2K has been responding to a need expressed by communities to be trained, both in wiki skills and in more technical topics, and has continued to develop its competence in providing this kind of support. Their constant needs assessment exercises with communities help them stay aware of the needs of communities and respond to opportunities in their environment.

Past performance[edit]

Will this organization's past performance with program implementation enable this plan to succeed?

CIS-A2K has been meeting or exceeding their targets year over year, which makes us confident that they have the ability to achieve what they set out to achieve. Furthermore, we see their competence at setting accurate targets in some areas (such as participants, new editors, and the footprint tracker) also improving.

At the midpoint of the current year they were on track to achieve their targets for participants and new editors, as well as the footprint tracker metric, and had already greatly exceeded their target for content pages. These achievements are impressive, with 18,951 content pages, 853 newly registered users, and 2,132 participants.

CIS-A2K has replaced the bytes deleted grantee-defined metric, which was was identified as complicated and difficult to interpret, with a community evaluation metric that reinforces their plan to listen to the communities and uses that feedback to adapt their programs and activities accordingly. This rather qualitative metric might be difficult to implement and interpret as it is currently formulated, but might be interesting to refine with time to make it really useful.

Organizational effectiveness[edit]

Will this organization's overall effectiveness enable this plan to succeed?

CIS is an effective organization, that has proven its ability to manage funding and achieve results. As we have pointed out in past years, the inclusion of A2K as part of the larger CIS organization continues to bring benefits to the movement.

In the past year, CIS-A2K continued to go through many staffing transitions, and we hope the team will be able to stabilize in order support programs even more efficiently.

While CIS-A2K’s plan is complex, as is their context, they have successfully managed similar plans in the past. CIS-A2K has worked to produce a more coherent program structure, but still has room to grow in the area of evaluation.

Strategy and programs[edit]

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy[edit]

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan?

CIS-A2K's strategic plan has been in place since 2016 and is supposed to continue until July 2018. CIS-A2K has been very active in collecting feedback and ideas from the communities of India during Phase I of the movement strategic direction, and we hope they will continue to be active in Phase II. This proposal does not specify whether CIS-A2K is looking at working on a new strategic plan to follow-up on their 2016-2018 endeavour, but their work plan and this proposal already gives insights about their strategic thinking, especially in their approach to community support.

Programs[edit]

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

CIS-A2K has clarified their program structure in this proposal, which better demonstrates their focus on providing support to communities and leaders in the region.

In the area of content enrichment, CIS-A2K is working with the Punjabi communities to improve the quality of Punjabi Wikisource. They are planning to work in consultation with the communities to select other Focus Project Areas this year, and will also continue their education program work.

CIS-A2K’s work in education is highly aligned with the movement strategy, as it involves contributing in local languages. However, the presentation of their educational activities in this plan is not always very clear and it is difficult to make out what their exact focus in that field is going to be.

CIS-A2K is working to build the capacities of contributors through their skill development and leadership development programs. These programs are thoughtfully designed in consultation with the communities and based on needs assessments, and are likely to continue to have direct benefits on the Wikimedia projects in India. They take into account the specificity of the region as well as the diversity of actors present in the Wikimedia ecosystem in India, working with user groups and language communities to develop training schemes adapted to each need.

We like that CIS-A2K’s evaluation plan, including their selection of grantee-defined metrics, reflects their shifting approach. Yet, for some important activities such as their skill-building initiatives, metrics do not adequately reflect the depth of their work. Many are focused on the number of events conducted or people trained, but do not reflect the impact these events may be having on participants. Partnerships metrics are also shallow, focusing on things like the number of partners engaged rather than the longer term results of this work. As CIS-A2K deepens their work in these areas, we hope that they also deepen their approach to evaluation.

Budget[edit]

Is this plan and budget focused on the programs with the highest potential for online impact?

CIS-A2K is requesting an increase, and we believe the funding invested is likely to lead to commensurate impact based on the plan presented and past results. Staffing costs and program costs seem appropriately directed to impactful work.

Summary of expert opinions (if applicable)[edit]

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework[edit]

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

  • Major strength
  • Strength
  • Neither a strength nor a concern
  • Concern
  • Major concern

Criterion

Assessment

Description

Program design

P1. Strategy

Strength We see that CIS is refocusing their efforts in line with their context and the lessons they have learned, and this is well explained throughout their plan.

P2. Potential for impact at scale

Major strength Working in India with many languages with large numbers of speakers, CIS-A2K has great potential for impact at scale. Their development of customized approaches for specific language areas and the ongoing development of their skill-building work may lead to long term results.

P3. Objectives and evaluation methods

Strength We like the way CIS-A2K has adapted their evaluation plan to their new approach. We find the selected metrics relevant and believe that CIS-A2K has the capacity to implement the evaluation plan they have proposed. They appear to be increasing their capacity to set accurate targets.

P4. Diversity

Strength The upcoming plan continues to emphasize work with a diverse range of language communities, and an emphasis on gender gap work is included throughout the plan. At the same time, the bar is being raised in the area of gender gap work in the movement and CIS continues to struggle with engaging more female contributors in the communities, and on their staff.

Organizational effectiveness

O1. Past results

Strength CIS-A2K is likely to meet or surpass many of its targets in the current year and is also on track to surpass last year's performance.

O2. Learning

Major Strength CIS-A2K continues to document their learning, and this plan is the most comprehensive to date in showing how they apply learning. CIS-A2K strives to integrate community feedback into their program design.

O3. Improving movement practices

Strength CIS-A2K is putting a stronger emphasis on their role as a regional actor in this plan, and we think this is something that can benefit the movement in the long term. We would still like to see CIS-A2K engage more at the international level.

O4. Community engagement

Strength CIS-A2K has made progress in this area, and has produced a plan and approach that emphasizes community engagement. They are continuing to build skills in assessing community needs and listening to communities.

O5. Capacity

Strength CIS-A2K has proven to be an organization with strong capacity, despite challenges, for example turn over in some positions which have been set backs to implement long term actions in some areas. CIS-A2K must continue to work on accompanying growth and development in order to stabilize their team and take advantage of lessons learned even more effectively. As an example, improving the gender imbalance on the team could ais them in tackling gender gap work effectively.

Budget

B1. Past budgeting and spending

Strength Past spending has been focused on direct impact on the projects and this has not changed.

B2. Budget is focused on impact

Major strength The plan and budget are focused on results that will support effectively programs and activities in India.

This staff proposal assessment is the work of FDC staff and is submitted by: Delphine (WMF) (talk) 06:59, 15 May 2018 (UTC)

Staff proposal assessment framework[edit]

  • Major strength. This is something the organization does very well, and this is a strong indicator of future success.
  • Strength. This is something that the organization does well, and this could indicate future success.
  • Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
  • Concern. This is something that the organization does not do well, and this could make funding the application a risk.
  • Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion

Description

Program design

P1. Strategy

The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.

P2. Potential for impact at scale

Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested

P3. Evaluation methods

Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.

P4. Diversity

Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.

Organizational effectiveness

O1. Past results

This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.

O2. Learning

This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.

O3. Improving movement practices

This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.

O4. Community engagement

This organization effectively engages communities and volunteers in the planning and implementation of its work.

O5. Capacity

This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.

Budget

B1. Past budgeting and spending

This organization has a history of budgeting realistically and managing funds effectively in the past.

B2. Budget is focused on programmatic impact

Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.