Grants:APG/Proposals/2013-2014 round2/Wikimedia Foundation/Proposal assessment by Wikimedia Deutschland e.V.

Add links
From Meta, a Wikimedia project coordination wiki

Introduction[edit]

In order to avoid a potential bias of WMF in assessing their own proposal, FDC have asked Wikimedia Deutschland (WMDE) to do the staff assessment of the WMF's proposal.

We feel honoured and have gladly agreed to help assessing the proposal. Besides our own involvement, we appreciate that community members and other chapters took the the chance to add their assessment and comments to the talk page.

It was clear to all parties that WMF cannot be compared to other movement entities both due to its size and its role: it is tasked with different duties than the other applicants.

Only after we read the proposal carefully and put it into context with the strategic plan, the complexity of the matter became apparent. It can only be reviewed in the context at large, refined with the strategy (which again is refined by the narrowed focus) as well as the operational work (which is laid out in the proposal’s appendix and its several subdocuments).

Because of the extent and complexity of the matter, and given the short time frame, we decided to assess the “Infrastructure” and “Mobile” part of the proposal.

Our Head of Software Development (Abraham Taherivand) and Head of Evaluation (Manuel Merz) assessed consistency of the Infrastructure and Mobile parts in themselves and with the given strategic plan and the clarity for decision-makers (FDC, board). We focussed on the interconnection of the narrative told in the available documents.

The next pages will explain how we proceeded with the review and give an overview of the results. We will end with a summary and conclusion.

Method[edit]

We focused our review on two of the four crucial initiatives: “Infrastructure” and “Mobile”. We conducted a methodological expert review only, analyzing the clarity and consistency of the proposal:

  • Clarity of Theory of Change: Are the causal links between the goals 2014-15 and the intended long-term outcomes explained in a way that is coherent for decision-makers?
  • Clarity of goals and meaningfulness of metrics: Are the outcome and output goals for 2014-15 explained in a way that is coherent to decision-makers? Does the proposal focus on metrics that are able to measure the intended outcomes of the activities?
  • Consistency with strategy: Do the goals of the proposal 2014-15 directly address the strategic priorities of the Wikimedia Strategic Plan 2011 or are there any unexplained inconsistencies?

What we have not assessed is the content of the proposal itself, the prioritization of the key activities, whether they are the right ones or not, or whether they are well chosen or suited to achieve impact. We think that the WMF staff with all their background knowledge and experience is in a better position to do so. Further, we did not assess whether the funds proposed for the initiatives are proportional to the overall context and plan. Again, we think that we are not in the position to do this.

This is the information that we used for the analysis:

Proposal

Strategy

Results[edit]

The results section includes an outcomes-level analysis as well as a short output-level analysis.

Outcomes-level analysis[edit]

The WMF’s Theory of Change is given directly in the Wikimedia Strategic Plan 2011. It is important to point out that the Wikimedia Strategic Plan 2011 also contains output-level goals. As a consequence the WMF proposal mainly adopted its goals directly from the Strategic Plan 2011. That means that in most cases only output goals and no outcome goals were included in the proposal. Because of this, rather than testing the output goals of the strategy, for the purpose of this review we take these goals as given.

Clarity of Theory of Change[edit]

Both initiatives are based on a comprehensible Theory of Change. While the Theory of Change of “Mobile” is explained directly in the proposal, in the case of “Infrastructure” it is only given indirectly and briefly in the Wikimedia Strategic Plan 2011. We recommend that the proposal/Annual Plan itself should explain how the proposed outcomes and key activities will address a corresponding Theory of Change. This will help decision-makers to assess the proposal.

Clarity of goals and meaningfulness of metrics[edit]

There were outcome goals given in the proposal for “Mobile” (e.g. “Increase the number of editors who contribute to Wikimedia projects” and “Increase mobile readership”), including proposed outcome-focussed metrics (e.g. “number of editors active on mobile” and “Wikipedia Zero pageviews”). This was very positive to see. At this stage it is only natural that concrete targets are still to be developed. As a future enhancement we would recommend to also consider outcome-metrics that represent changes that can be directly attributed to the work done by WMF (e.g. the “Wikipedia Zero pageviews” are strongly correlated to external factors such as the general number of mobile phone users with internet access).

In the context of “Infrastructure” the WMF uses and gathers complex metrics. We highly value these analytics efforts and the transparency they provide. However, due to the operational nature of the goals given in the Wikimedia Strategic Plan 2011, there were hardly any outcome goals (and metrics) given for “Infrastructure”. We would recommend that the proposal should include outcome goals and metrics, if possible, instead of solely giving key activities. This will help decision-makers to assess the proposal from a non-technical perspective. At the same time we acknowledge the challenge of setting outcomes-based goals for highly complex software and infrastructure development programs. A general approach for all software development activities within the movement could be useful.

Consistency with strategy[edit]

For the “Mobile” section of the proposal, we could test its outcome goals for consistency with the strategy. The outcome goals in the “Mobile” section (and their high priority) were highly consistent with the Wikimedia Strategic Plan 2011 (section “Increase reach”, on page 13, 2. “Invest in mobile products…”), especially considering the Narrowing Focus document. We suggest linking to the Narrowing Focusdocument, in the “Mobile” introduction section of the proposal to make clear that this is the main strategic link. This helps readers to get a clear overall understanding of the consistency towards the strategic plan.

As explained, no clear outcome goals could be identified in the “Infrastructure” section. Therefore we worked with the listed “key activities” (output goals) and their descriptions alongside the output goals of the Strategic Plan 2011. Given that, this section of the proposal was highly consistent with the Strategic Plan 2011.

A more detailed overview of the analysis can be found in the appendix of this review.

Outputs-level analysis[edit]

The plan contained general descriptions of key activities and direct products. We tested these output goals for consistency with the given outcome goals (only applicable for “Mobile”, as the output goals of “Infrastructure” had to be tested directly against the Wikimedia Strategic Plan 2011, as explained above). The output goals for “Mobile” were generally consistent with the given outcome goals from an expert perspective.

To enhance the general clarity we recommend to follow a common structure for sections and subdivisions in the proposal and appendix concerning the described “key activities”. We would also like to address that the proposal did not contain SMART output goals. Instead, detailed output goals will be developed by the corresponding departments and teams in the course of the WMF's “annual goal-setting process in May-June and iterated on a quarterly basis, following the model of the current fiscal year’s engineering goals”. We think this approach is appropriate for movement organizations. The iterative planning makes it possible to manage the complexities a movement organization has to deal with. Planning iteratively enables the integration of community feedback and gives the flexibility to react properly in the fast-paced environment of our movement. This raises the question, whether a yearly proposal assessment fits to this kind of iterative planning.

Summary[edit]

Overall, we are quite impressed by the proposal and the work that has gone into it. All of the reviewed parts of the proposal were highly consistent with the strategy and movement priorities included in the Strategic Plan 2011. There is however no clear prioritization visible in the described "key activities". We assume, that the detailed prioritization is done during the quarterly planning process, supported by agile methods as mentioned by the "Mobile" team e.g. SCRUM. From the reviewers’ perspective this iterative goal-setting process is seen as very suitable.

For the “Infrastructure” part of the proposal we recommend that the proposal itself should explain how the proposed outcomes and key activities will address a corresponding Theory of Change. We also recommend that the proposal should include outcome goals and metrics instead of solely giving key activities. Both will help decision-makers to assess the proposal from a non-technical perspective. It was very positive to see that both points were already addressed in the “Mobile” part of the proposal. Because of its clear descriptions and the outlined outcome goals the “Mobile” section of the proposal was very transparent and comprehensible even for a non-expert reader.

We hope that our recommendations can help to further develop the annual plan. We would have loved to do a more comprehensive assessment, but that would have needed more time and more preparation. We will be available in future rounds to help out, of course.

Manuel and Abraham are available to answer further questions, either online or in person or via video call, if necessary.

APPENDIX[edit]

A) Glossary[edit]

  • Outcome [1]
  • Output [1]
  • Theory of Change [1]

[1] compare: https://meta.wikimedia.org/wiki/Programs:Evaluation_portal/Library/Glossary

B) Consistency with strategy[edit]

“Infrastructure” (key activities)[edit]

  • Site operations - Consistency with strategy (compare “Stabilize infrastructure” section 1.* in Wikimedia Strategic Plan 2011)
  • Site performance and architecture - Consistency with strategy (compare “Stabilize infrastructure” section 1.* in Wikimedia Strategic Plan 2011)
  • Wikimedia Labs - Consistency with strategy (compare “Encourage innovation” section 2.* in Wikimedia Strategic Plan 2011)
  • Data dumps - Consistency with strategy (compare “Encourage innovation” section 2.* in Wikimedia Strategic Plan 2011 and “Narrowing Focus” document)
  • Code review - Consistency with strategy (compare “Stabilize infrastructure” section 1.* and “encourage innovation” section 2.* in Wikimedia Strategic Plan 2011)
  • Security and privacy - Consistency with strategy (compare “Stabilize infrastructure” section 1.* in Wikimedia Strategic Plan 2011)
  • Deployment - Consistency with strategy (compare “Stabilize infrastructure” section 1.* in Wikimedia Strategic Plan 2011)
  • APIs- Consistency with strategy (compare “Stabilize infrastructure” 1.* and “Encourage innovation” section 2.* in Wikimedia Strategic Plan 2011)
  • Release management - Consistency with strategy (compare “Stabilize infrastructure” 1.* and “Encourage innovation” in Wikimedia Strategic Plan 2011)
  • Test infrastructure - Consistency with strategy (compare “Stabilize infrastructure” section 1.* Wikimedia Strategic Plan 2011)
  • Shell requests - Consistency with strategy (compare Stabilize infrastructure section 1. in Wikimedia Strategic Plan 2011)
  • Search, authentication, etc. - Consistency with strategy (compare “Stabilize infrastructure” section 1.* in Wikimedia Strategic Plan 2011)
  • Quality Assurance - Consistency with strategy (compare section “Stabilize infrastructure” in Wikimedia Strategic Plan 2011)
  • Reporting and documentation - Consistency with strategy (compare section “Encourage innovation” section 2.* in Wikimedia Strategic Plan 2011)
  • Outreach and development - Consistency with strategy (compare section “Encourage innovation” section 2.* and 3.* in Wikimedia Strategic Plan 2011)
  • Upstream collaboration - Consistency with strategy - Although the activity is not explicitly mentioned in Wikimedia Strategic Plan 2011 the technical reader knows that it is important to address it as a key activity. For participants in the open source ecosystem it is clear that it is essential. However, detailed descriptions about the scope are missing in the proposal.
  • Analytics - Consistency with strategy (see “Encourage innovation” section 1. in Wikimedia Strategic Plan 2011)
  • Research - Consistency with strategy (compare “Encourage innovation” section 1.* & 2.*” in Wikimedia Strategic Plan 2011)
  • Internationalization - Consistency with strategy (see “Encourage innovation” section 1.* and “Increase reach”)

“Mobile”[edit]

  • Overall consistency with strategy (compare “Increase reach” section 2.* in Strategic Plan 2011 and “Narrowing Focus” document)
Retrieved from "https://meta.wikimedia.org/w/index.php?title=Grants:APG/Proposals/2013-2014_round2/Wikimedia_Foundation/Proposal_assessment_by_Wikimedia_Deutschland_e.V.&oldid=15333663"