Methodology behind the 2013-2014 Round 2 FDC staff assessments
The FDC staff assessments provide a snapshot of the staff review of Annual Plan Grant (FDC) proposals, and identify both strengths and concerns with the work proposed.
- We did this through both portfolio analysis - looking across proposals and calibrating across different organizations - as well as assessing the context of each individual organization and its proposal.
- Inputs into the staff assessment process this round include:
- The proposals themselves, with the attached documents (strategic plans, annual plans, detailed budgets, supplementary documents)
- Past and current reports (quarterly progress reports on annual plan grants, other grant reports, annual financial and narrative reports)
- A portfolio view of financials across proposals for both rounds this year Grants:APG/Proposals/2013-2014_round2/Financial_overview and Grants:APG/Proposals/2013-2014_round1/Financial_overview.
- Internal financial, program evaluation, and grants compliance inputs from the WMF Finance, Programs, Legal, and Grantmaking teams.
- Detailed comments and questions collected during the Community Review period, which are also summarized in these staff assessments.
- We used a few key principles for the assessment, across all proposals. These are not new, and are detailed in the narrative and scoring sections of the assessment. They include:
- Impact on Wikimedia projects and the global movement, commensurate with funds requested. We looked for the rationale behind the work organizations do, aligning them to the Wikimedia strategic priorities and goals. We looked particularly for direct and indirect impact of this work on Wikimedia projects; for example, growth in contributors and content donation. Most importantly, we looked at whether the funds requested were justified in the context of current and potential impact of the proposed work.
- Organizational strategy, leadership, and governance. We looked for evidence that an organization had a good understanding of its own context, and the needs, opportunities and challenges of the communities it seeks to serve or partner with. We looked for board, staff and volunteer leadership that is committed, effective and engaged with these communities and practising movement values of openness and transparency.
- Rates of growth and financial sustainability. In this round, no applicants asked for funds within the recommended maximum growth rate in FDC funding of 20%, which is itself a significant rate in many contexts. The minimum growth rate in the FDC/movement funding proposed is 22.7% and the maximum proposed is 216%. Given the challenges faced by some organizations with raising funds externally, there are significant concerns about the long term financial sustainability of this level of dependence on movement resources in areas where other funding may be available.
- Program design, learning, and evaluation. We recognize and appreciate that organizations in this round aim to link their work directly to outcomes on the Wikimedia Projects they are targeting and to work toward aligning their high-level goals across programs; at the same time, this work does not always coincide with clear links between these high-level targets and the direct outcomes of program activities, which may make it difficult to understand the effectiveness of some program activities. We appreciate the good work that organizations have done in this area so far, and urge organizations to continue to work toward making the links between program activities, objectives, and high-level targets clear.
- Inclusion of exceptional organizations. Two organizations were included in this round that required exceptions. The Centre for Internet and Society is the first non-Wikimedia organization to participate in the FDC process, and was made eligible through a resolution by the WMF Board of Trustees. As a result, detailed attention was paid to the financial data presented by CIS to understand the work specific to the Wikimedia movement that will be funded by the FDC, in addition to understanding their broader budget context, and only this work was described in detail in the CIS proposal and analyzed in the staff assessment. Other than this, the process for assessing the CIS proposal has been the same as that for other proposals. The Wikimedia Foundation also participated in this round of the FDC process; at the FDC’s request we did not complete a staff assessment for the Wikimedia Foundation.
Note: We shared our nearly final draft assessments with all FDC applicants a day ahead of publishing them, so that organizations had some lead time for sharing these with their Board and staff. We then took all responses into consideration where possible, making factual changes and clarifying our statements where necessary. We encourage organizations to note all substantive comments and concerns on the discussion pages of these assessments.
|Name of eligible organization|
|Summary staff assessment [substantial concerns: none/some/many]|
|Total amount proposed to the FDC for the upcoming year annual plan in currency requested|
|Estimate of total amount proposed to the FDC for the upcoming year annual plan in US dollars|
|Exchange rate used to calculate estimate in US dollars and date accessed (mm/dd/yy)|
|Date of proposal submission|
|Adherence to proposal process [yes/no]|
|Adherence to legal, regulatory, and policy regulations [yes/no]|
|Adherence to US anti-terrorism laws [yes/no]|
A detailed summary of the financial information from all proposals is here: financial overview document.
|Projected 2013||Proposed 2014||Proposed change (as a +/− %)|
|FDC or WMF funding|
Overview of staff assessment
What are the top strengths of the plan?
What are the top concerns with the plan?
To what extent does the plan address the strategic priorities of the movement?
Alignment with the strategic priorities
Program objectives and measurable outcomes
Context of the organization and community
To what extent does the context of the organization (e.g., community participation, fundraising context, sociopolitical environment) enable or inhibit the organization's potential for impact? To what extent does the organization's experience enable or inhibit the organization's potential for impact?
Online focus and potential for global impact commensurate with requested resources
Financial sustainability and proposed growth
Feasibility and risks for the annual plan
How feasible is the annual plan? Specifically:
- To what extent are the budget, staffing, availability of volunteers, and other resources realistic for achieving the anticipated outcomes?
- Are the timelines realistic for the proposed initiatives and associated activities?
- Are there acknowledged and/or unacknowledged risks in the proposal?
Summary of expert opinions and/or community commentary
Summary of the opinions of subject-matter experts on the plan
No expert commentary or opinion was requested for this plan.
Summary of community commentary on the plan
Scores, comments and feedback
Scores provide only a preliminary assessment for consideration by the FDC. The rubric for scoring is below.
|Dimensions||Criteria||Score (1–5)||Comments and feedback|
|Potential for impact against strategic priorities||(A) The plan addresses the Wikimedia movement strategic priorities of the Wikimedia movement (infrastructure, participation, quality, reach, and innovation)|
|(B) The initiatives in the plan have the potential to lead to movement strategic priorities, if executed well|
|(C) The initiatives in the plan have the potential to lead to significant impact, given context and amount of funds proposed|
|Ability to execute||(D) The organization has the resources, skills, and capacity (staff, volunteer, non-staff) for the plan to succeed|
|(E) The organization has a record of success in similar initiatives|
|(F) The organization's leadership is effective, committed, and relatively stable|
|Efficient use of funds||(G) The funding requested is reasonable given the proposed initiatives|
|(H) The organization has a record of using funds efficiently and staying on budget|
|Quality of proposed measures of success||(I) Clear indicators of success are outlined in the plan|
|(J) A plan is in place to track the proposed metrics|
|(K) The organization has a feasible plan in place to track the proposed metrics|
|Potential to add knowledge and/or other benefits to the movement||(L) The initiatives in the plan, if successful, could be helpful and/or productively adapted elsewhere in the movement|
|(M) The organization is ready to share with the movement the lessons learned from the initiatives|
|Dimensions||Evaluation criteria||1 = Weak or no alignment with criterion||3 = Moderate alignment with criterion||5 = Strong alignment with criterion|
|Potential for impact against strategic priorities||(A) The plan addresses the Wikimedia movement strategic priorities||The plan will not directly address any strategic priorities||The plan will partially address strategic priorities or other goals that are closely related to the strategic priorities||The plan will directly address several strategic priorities|
|(B) The initiatives have the potential to lead to the strategic priorities, if executed well||The initiatives have no clear connection to strategic priorities||Some or all of the initiatives have an unclear relationship to strategic priorities||The initiatives have been shown (by the organization or the movement) to lead to strategic priorities|
|(C) The initiatives have the potential to lead to significant impact, given context and amount of funds proposed||The initiatives will have marginal impact in the local community and the broader Wikimedia community, given context and amount of funds proposed||The initiatives proposed may have a significant impact in the local community, given context and amount of funds proposed||The initiatives could have significant impact at scale, both in the organization's local community and in the broader movement, given context and amount of funds proposed|
|Ability to execute||(D) The organization has the resources, skills, and capacity (staff, volunteer, non-staff) for the plan to succeed||The organization has neither the number nor the amount of resources, skills, and capacity needed for success.||The organization has some relevant staff and volunteer capacity, but may be under-resourced for some initiatives||The organization is well-resourced with the relevant skills and capacity for success|
|(E) The organization has a record of success in similar initiatives||The organization has had trouble succeeding in similar initiatives||The organization has not engaged in similar initiatives in the past||The organization has conducted similar initiatives in the past and has been successful in achieving desired results|
|(F) The organization's leadership is effective, committed, and relatively stable||Leadership has been unstable (e.g., changing on an annual basis)||Current leadership is committed to the organization; some concerns exist around stability or effectiveness of leadership||Current and past leaders have demonstrated the ability to develop and execute on strategic plans; leadership team has been stable and transitions have been smooth|
|Efficient use of funds||(G) The funding requested is reasonable given the proposed initiatives||Based on FDC experience and relative to other proposals, the plan contains significant over- or under-budgeting, and/or has not thoroughly accounted for costs||Based on FDC experience and relative to other proposals, the budget is generally consistent with the initiatives; some initiatives are likely to be over- or under-budgeted||Based on FDC experience and relative to other proposals, the budget is reasonable, and rigorous cost projections have been presented|
|(H) The organization has a record of using funds efficiently and staying on budget||The organization has a poor record of budgetary management: e.g., consistently and significantly below budget (such as 50% underspending), and drawing on organizational reserves due to overspending||The organization has a middling record of budgetary management; e.g., generally on-budget for its biggest initiatives and cost items, properly recording any over- and under-spending, and with plans in place for improvements||The organization has consistently strong record of financial management; e.g., consistently staying on or below budget (within ~5% of budget), and putting surplus funds towards the organization's and/or movement's goals|
|Quality of proposed measures of success||(I) Clear indicators of success are outlined in the plan||Initiatives have no associated indicators of success, or the proposed indicators are unrealistic and/or unmeasurable||Each initiative has indicators of success, but at least some indicators are unrealistic or are not clearly connected to the initiative||Each initiative has a realistic and carefully framed set of indicators of success associated with them, which are rigorous and time-bound|
|(J) A system is in place to track the proposed metrics||The plan does not include methodology or timing for tracking and collecting metrics to measure progress against proposed indicators of success||A proposal to track progress against indicators exists, but has flawed methodology or unrealistic timelines and initiatives||The plan includes timing and tracking of proposed metrics, and additional initiatives (e.g., surveys, research) to track indicators|
|(K) The organization has a feasible system to track the proposed metrics||No systems are in place to track metrics, or staff or volunteer capacity to manage metrics tracking||There is some overall capacity (e.g., staff, volunteers) to track and monitor metrics, but the organization does not clearly have the relevant skills to do this||High-quality systems, and the necessary skills and capacity to manage these systems, are in place|
|Potential to add knowledge and/or other benefits to the movement||(L) The initiatives, if successful, could be helpful and/or productively adapted elsewhere in the movement||The initiatives would not be relevant to other movement groups||Some initiatives may be relevant to other movement groups||Many initiatives proposed are new or have seldom been tried elsewhere in the movement; if successful, other movement entities would benefit from doing similar initiatives|
|(M) The entity is ready to share with the movement the lessons learned from the initiatives||The entity is isolated from other movement groups and rarely shares information; the entity's leaders and members seldom participate in movement-wide functions||The entity is somewhat active within the movement, and is developing more relationships with other movement entities with which it could share lessons learned||The entity has regular communication with other movement entities, is a regular participant in Wikimedia events, and has a track record of sharing with others|
This form was created by: