Grants talk:APG/Funds Dissemination Committee/Advisory Group/Recommendations/2014/ED Response

From Meta, a Wikimedia project coordination wiki

Anasuya or Katy may post responses from Lila[edit]

Responses to the questions on this page are from Lila, but may be posted by Katy or me for convenience. Thanks, ASengupta (WMF) (talk) 07:41, 5 November 2014 (UTC)[reply]

Impact as documented and in reality[edit]

Thanks for this report that gives a good summary of what we have learnt this far, and gives, in my opinion, a fair set of guidelines for future funding and reporting.

The core of the message is around Impact: impact reporting and funding in proportion to impact. And I believe this is right and I am also happy to see that a set of Global metrics is emerging, just what has been most missing for the movement to be able to have an impact driven funding.

In this context I would like to highlight, though, what I believe has been, and is, a deficiant mechanism for documenting impact. That there is a big difference in impacts actually taken place in communities, project and interested parties around our projects and what has been properly documented and reported. I know this first-hand, as I am the second most active Wikipedian on svwp, and actually follows all edits 24/7 and fixes problems in all new articles, when needed. I also follow first-hand most of the programs the chapter runs and can observe its impacts. And I see that sometimes programs run by chapters takes 2-3 three years before the real positive impact can be seen in our projects (like promoting experts in Glams to improve article content quality). Other programs has been critical in stopping our community to break down in internal bickering (and this alternative reality of a dysfunctional community can of course never be measured).

This reflection does not change my support to focus funding on impact and that efforts shall go into better understanding and reporting impact. I think, though, that a recognition of that fact that we are still very immature in understanding and reporting impact and that impacts in reality is actually more positive than can be seen in the reporting. So I would wish this report to be seen more as a guideline, and that it will be a year or two before translating it into operative guidelines for APGfunding.

Another comment on the report, is that I notice WMF budget will be outside the FDC scope. I personally would like it to be in the same as other entities but can live with it. But then would it not be proper to demand the same set of impact reporting and funding in proportion to funding for the WMF efforts as for other entities?

Anders Wennersten (talk) 07:42, 2 November 2014 (UTC)[reply]

I am very glad to have your feedback, Anders. Thank you for your commitment to our movement as a top contributor to svWP, WMSE and as a member of the FDC. I agree with you that our movement is still learning what works and what doesn’t. In addition, some of the most important partnerships work by Wikimedia organisations take time to show impact, and some qualitative work is difficult to measure. Therefore, we will be opening up any possible changes to the annual plan grants process to community consultation over the next year, and will operationalise only in 2016.
At the same time, it seems clear to me and my team that our movement needs more non-monetary support; for e.g. more tools and practices that will support impact on policy and partnerships, on evaluation and learning. Anasuya and her team are keen to move their focus towards facilitating the sharing of best and good practices across the movement, and working with experts within and outside our communities to create significant resources for us to do, share and measure our work better.
I also agree that WMF needs to hold itself up to high standards of planning, implementation and measurement given its key role in the movement and its budget. This is precisely what I am focusing my attention on right now, and I believe you will begin to see the results as we share our strategic planning process with you. I would ask you to continue to hold us accountable to these standards, and give us your suggestions and advice if you feel we are not being transparent and impactful enough. The operational processes and reporting is changing as well, including quarterly cadence of metrics. thanks, (for Lila) ASengupta (WMF) (talk) 07:42, 5 November 2014 (UTC)[reply]
Thanks Lila/Anausya for your thoughtful answers which I find quite satisfactory.
  • I like very much your thoughts on giving more hands-on support in helping the entities to work with evaluations and impact oriented plans and reports
  • I also like and appove your timing in planning to have a year of discussion and consultation before any major changes in the the annual plan grants process
  • for the budget (and planning) process for WMF, I intend to follow it closely and hope it will, as you state, show you live up to being held accountable to the same standards as you require from the entities
  • And I am thinking of and expect the FDC session in 10 days time will be as successful (and stimulating process) as it was the two years I participated!Anders Wennersten (talk) 17:29, 5 November 2014 (UTC)[reply]
Hi Anders, glad to know you're pleased with our current thoughts and direction. And yes, we're all very much looking forward to the FDC meeting in 10 days! We'll miss you, though. :-) ASengupta (WMF) (talk) 19:20, 6 November 2014 (UTC)[reply]
Hi Anders Wennersten, I'd just like to note here that in my WMF role, my work is a part of what Lila noted as "tools and practices that will support impact on policy and partnerships, on evaluation and learning". I'm working on the Learning Patterns Library. I hope to develop this resource to help affiliates and individuals with many aspects of programmatic work, including tools for streamlining and enhancing their reporting about financial and programmatic inputs and outputs. I also hope to help affiliates and individuals to share their "best and good practices across the movement" for everyone's benefit. Regards, --Bgibbs (WMF) (talk) 20:00, 5 November 2014 (UTC)[reply]
Thanks for leting us know this. Your work is very important and will be much apprceiated and eagerly awaited by all involved. Welcome to the entitities support realm!.Anders Wennersten (talk) 20:57, 5 November 2014 (UTC)[reply]

Pine's questions[edit]

I am asking these questions in my personal (not WMF) capacity, and I hope to use these questions and your responses for a Signpost report. Secondarily, the responses will inform my thinking about Cascadia Wikimedians, and my work in WMF Learning and Evaluation.

Thanks for all these thoughtful questions, Pine. I'm posting Lila's responses below on her behalf. Do note she hasn't managed to complete your long list yet. :-) ASengupta (WMF) (talk) 15:18, 5 November 2014 (UTC)[reply]

APG/FDC questions:

  1. Going forward, will comprehensive use of SMART goals be required for APG applicants? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    We have already implemented global metrics and are asking APG applicants to be clear in their use of SMART goals for specific programs. We are also encouraging the use of combined quantitative and qualitative goals for initiatives; for instance, like policy advocacy work. Most importantly, we are asking all applicants to be clear about their rationale and reasoning behind specific programs in their plans. (for Lila) ASengupta (WMF) (talk) 15:18, 5 November 2014 (UTC)[reply]
  2. Per my earlier conversation with Jessie Wild: because smaller, more targeted grants often show better results than large general-support grants, would it make sense to reorganize FDC grants to fund specific programs run by affiliates instead of providing general support for affiliates as a whole? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    We are shifting towards a more focused approach. We are certainly investing in programmatic success: looking for high potential or high impact programs, while encouraging organizations to modify or stop unsuccessful programs. At the same time, some level of flexibility in decision-making (offered by general support grants) means that organizations working over long time-horizons like a year or more, can shift strategy and be opportunistic when they see clear openings to do so (for e.g. unexpected support from local GLAM institutions or government).
    In addition, we have been encouraging an integrated, focused approach to work based on the key strengths of an organization. For instance, if a chapter has high opportunity and expertise in science education, different activities can be focused towards a clear education initiative, rather than dissipating energy having multiple programs with stretched resources across all of them.
  3. Following up on recent discussions in the Research showcase and on the Research mailing list: what hope does WMF have of increasing editor retention and diversity on English Wikipedia if WMF does not devote significant resources to assisting the community to evolve from a "crushing beaurocracy" (in the words of MIT Technology Review) to a welcoming, flexible, and adaptable organism/community (see Aaron Halfaker's work)? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    I agree that editor retention and diversity would be better supported by a welcoming and inclusive editing environment. While editorial policies are the responsibility of our communities, I think the WMF needs to take a more active role in facilitating necessary change. And I hope that as we think about these issues (including in our strategic planning process), community members will continue to offer specific suggestions on what can be done productively and constructively and help us prioritise.
  4. What evidence do we have that the $6 million annually in APG funding is money well spent? I glance through some periodic reports come through Wikimedia-l, but I know of little significant online impact outside of Wikidata and a number of GLAM partnerships that could probably be run directly and economically through contracts by WMF. --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    We do not have a lot of evidence and I am looking to make this a fundamental part of our grantmaking process. We believe that local partnerships - for content or policy (like GLAM institutions or government bodies) - are best understood, nurtured and grown locally. This helps us sustain and expand the motivation of our local language communities, through focusing on the specific needs and interests of readers and contributors in those contexts.
    This is also where the expertise of local volunteers and organizations can serve as multiplier effects for increasing the quality of content globally and challenging current content gaps. For e.g. content acquired, uploaded and used by local organizations and communities in Latin America can then be used across the world. I agree with you, however, that the connections between and the impact of, offline activities on our online environments needs to be clearly established and monitored. Global metrics, and the continuing focus of Wikimedia organisations to make those connections clearer and more specific, will help with this.
  5. Let me ask about a hypothetical situation. Suppose that WMF decided to cap APG spending at $100,000 per organization per year and $1 million total, WMF instructed affiliates that they are required to fund-raise from diverse sources for any additional funds, and WMF redirected the $5 million saved from FDC funding to spending on research, engineering, and product development efforts that scale easily. What would be the pros and cons of that decision, and would it be better in the long run to move $5 million annually from APG spending to other priorities? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    This is exactly the kind of scenario forecasting that I would like the FDC, the APG applicants and our communities at large to discuss further, and will be part of our community consultation. Let’s keep in mind that not everything the WMF does scale and our ability to evaluate technical impact is just as important as evaluating local programs.
  6. Would it make sense to run all APGs on a biennial cycle instead of an annual cycle? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    At a time when the movement is still learning and adapting quickly, annual cycles are more useful for periodic sharing around progress and impact -- we do not have enough data to make a full change. However, we are experimenting with a pilot on two-year grants this APG round, and we will keep the community posted on lessons learned from this.
  7. What is the rationale for requiring that thematic orgs complete two PEGs before applying for APG funding? I believe that organizations that are currently ineligible for APG funding sometimes try to get FDC-like grants through PEG until they become APG eligible. Why not allow all of them to apply for APGs from the beginning? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    APG, by design, is an “at-scale” program that is designed for organizations that have a proven, successful model and are ready to produce larger impact. The original eligibility requirements for the APG funding were to ensure that organizations could meet the rigour of the APG process, and were also coming to the process at the right moment in their community and organisational context. Our team currently works in close partnership with organizations that want to get APG grants to ensure that they only apply when they feel ready for it, and have the right strategic approach for it. It may well be that if we modify the APG grants process over the next few years, based on our community consultation and internal thinking, then these eligibility requirements will also change. However, the most important criterion is that groups should not apply for significant sums of money, taking on the liability of full time staff, without showing early success with smaller pilot programs and adequate preparation/scoping of the cost-benefit analysis to themselves and the volunteer communities they serve.
  8. You emphasize the funding of programs that are believed to be effective as a priority over innovative projects. Can you further explain your reasoning for this decision? As you know, IEG often funds innovative projects, yet as a whole I would argue that the IEG portfolio is more cost-effective than the APG portfolio which is geared toward more conventional grants, so it seems to me that we should be funding more innovation rather than less, while we should also scale down or eliminate programs that repeatedly demonstrate low impact. --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    We may indeed need to move more of the funding towards the early research/seed grants to feed the innovation. However, the question is about the overall cost-benefit analysis of the significant funds/large grants. Individual Engagement grants are an excellent example of the process of seeding and supporting innovative and possibly risky ideas at low cost. I would encourage larger (APG or other) grants and organizations to follow this model where the bulk of the money is in proven programs that scale well and sustainably, while experimenting with pilots that try new things at lower cost to the movement.
  9. I was under the impression from the last Grantmaking quarterly review that you want changes in the strategy of the Grantmaking department, but this review of APGs is generally consistent with the direction that Grantmaking has been heading for some time, I believe at least since the time of Anasuya's arrival. Could talk about your strategic goals for the Grantmaking department? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    I think a shift in our strategy in Grantmaking has to happen over time. We need to become impact driven, programmatic, proactive in our thinking and accountable for successes/failures. We need to shift our approach towards well-focused and appropriate non-monetary support for Wikimedia programs, i.e. learning tools, practices and facilitation that will help our movement’s programs do better. Anasuya and her team have been moving towards this for a while now, but I have asked them to be much more intentional and focused on their strategy towards this. Other shifts and changes can only be discussed in consultation with our grants committees and community members, before finalising, which is why we have suggested a timeline for community consultation.
  10. In a mobile-first world, what changes should thematic organizations and FDC consider for their existing programs and for the creation of new programs? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    We will need to think about the different kinds of contributions that are enabled in a mobile-first or mobile-focused world. This is a great place for innovation. But even so some of the established work can be scaled as well: local partnership work is possible with telecommunication carriers, contests like WikiLoves heavily leverage mobile. We need to be creative in our ability to document parts of world and history where references are scarce. Mobility can help us do this.
  11. You talked about the Global South's small proportion of grants. We've known about this statistic for years. How does WMF plan to make changes that will significantly affect this statistic? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    The current Grantmaking team has already shifted the focus towards the Global South considerably. In the past year, we’ve doubled (from 10 to 20%) the total *amount* going to the Global South and the total *number* of grants (we currently give about half the number of our total grants to individuals and groups in the GS). However, we continue to need to support the environments for community growth and scaling with successful programs in these emerging Wikimedia regions and languages, and our strategy will be to focus much more on non-monetary strategic support to ensure that this is possible. In addition, we are looking at much more significant coordination between the work of the language engineering, mobile and Wikipedia Zero teams with the programs of the Grantmaking team. The bulk of our funding is currently to large organizations in Europe; we will continue to work with them as they diversify their own resource bases and increase support for diversity, with (for example) expansion of content related to the Global South in their own contexts, work with diaspora communities or well-focused offline distributions.
  12. We've also talked for years about the gender gap among contributors, and no prior efforts have led to sustained and noteworthy progress on a large scale. Going forward, what will WMF in general and Grantmaking in particular do differently to make progress? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    We hope to be trialing a campaign on gender-related contribution and content later this year, and have already - in collaboration with community members - pulled together some key program areas in which volunteers would like to focus their energies. I believe that a more proactive strategy is required on this issue, balanced by a responsive one that keeps in mind community contexts and good opportunities for change. We will also hope to be working with more external partners who can support the movement on this. At the same time, I’d like us all to ask ourselves these questions in our daily volunteer work: how are we supporting female contributors to do their work better, or helping expand female-related content? thanks, ASengupta (WMF) (talk) 22:25, 12 November 2014 (UTC)[reply]

WMF Annual Plan questions:

  1. What are the plans for the community review of the WMF Annual Plan? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
  2. What are the SMART goals for the community review of the Annual Plan? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
  3. Will the WMF Annual Plan make comprehensive use of SMART goals? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
  4. If there will be emphasis placed on quarterly reviews and making midcourse adjustments, then what relevance does the Annual Plan have? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
  5. Other than fixing errors, what specific changes were made to the 2014-2015 Annual Plan as a result of FDC feedback? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
  6. Other than fixing errors, what specific changes were made to the 2014-2015 Annual Plan as a result of community feedback? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
  7. Going forward, what will be the relationships between the Strategic Plan update, the Annual Plan, and the quarterly departmental plans? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
  8. Will there be a WMF-wide quarterly report that tracks progress against SMART goals? --Pine✉ 19:38, 2 November 2014 (UTC)[reply]
    Hi Pine, thanks again for your questions on this. I'm not sure Lila has the time to get to them right now, but I believe these are good considerations for us all to think about as we work through our current strategic planning and how we integrate with our next annual plan. We'll be coming back to the movement with a timeline and more details as we get more clarity on both. Appreciate the thoughts! ASengupta (WMF) (talk) 22:28, 12 November 2014 (UTC)[reply]

Thank you very much, --Pine✉ 19:38, 2 November 2014 (UTC)[reply]

  • Hi @Risker: I am currently working on a temporary contract for WMF as an intern in Learning and Evaluation. The focus of my work is the learning patterns library. You can read more about me and my scope of work at Bgibbs (WMF). --Pine✉

First N[edit]

Wikimedia budget

«The first four chapters receive 60% of all funding to Wikimedia organizations». I understand this is a document focused on APG, but I find this emphasis misplaced, and "all funding to Wikimedia organizations" a misnomer. It would be more appropriate to say «The first five entities receive 95% of all funding to Wikimedia organizations», by including the WMF which consumes 88 % of the money. --Nemo 08:36, 3 November 2014 (UTC)[reply]

Hi Nemo. Yes, I agree WMF should hold itself accountable and to higher standards. As I said above to Anders, this is a priority for me and the organization. (for Lila) ASengupta (WMF) (talk) 07:44, 5 November 2014 (UTC)[reply]

Re: open process into its yearly, quarterly and strategic planning[edit]

The time is soon coming when WMF starts drafting the 2015-16 plan, judging from Wikimedia budget; WMF management is already producing some strategy documents/drafts/discussion proposals. I wish to know the approximate dates by which:

  1. first WMF annual plan draft is shared on Meta-Wiki for comments;
  2. first strategy thinking is shared on Meta-Wiki.

--Nemo 09:28, 3 November 2014 (UTC)[reply]

We are currently working to finalize the timeline. I am hoping that we will share our first strategic thinking over the next few months (Q3) and the first draft of the next annual plan shortly after that (Q4). We will be in communication with the movement to clarify this by the end of this year. thanks, (for Lila) ASengupta (WMF) (talk) 07:46, 5 November 2014 (UTC)[reply]
Thanks for giving an approximate timeline, very appreciated. However Q4 is a bit generic, as – for the sake of paradox – it may mean sharing the annual plan on June 30, one day before it enters into force. If 3 months are enough to discuss an annual plan, that's fine; but the plan must be shared on Meta as soon as it's shared with WMF employees (wmfall@ and officewiki?) and/or the WMF board. --Nemo 08:53, 6 November 2014 (UTC)[reply]

Several questions[edit]

The recommendations are good but I have few concerns and I agree in some responses.

  • Continuity: a project and a plan can have an impact if it has a continuity... to assure a continuity it's important to have a local organization, more than a local community. I have seen several good projects demonstrating a good impact but, applying the same metrics some months after, I have read that the impact is disappeared... the projects have received a big temporary stimulation but a lot volatile and without a longtime impact. This is happening frequently. In any of these recommendations I don't see a good stress in this point and in a creation of "programs" having long duration in order to support the initial stimulation.
  • Diversity: It's important to stress the diversity but it's also important to stress that the Wikimedia projects have had success and are still valid thanks to some communities which need a constant support. There is no sense to food some communities with steroids and to suppose that some other communities (considered mature and with a sufficient capacity to survive) can live with small support. The risk is to sacrifice volunteers in oldest communities and to don't acquire sufficient volunteers (in the name of the diversity) in the new ones in order to keep the same level of contribution. In my personal opinion it's sufficient to say that Wikimedia offers the same level of access to the grants for all communities and to keep a neutrality in the evaluation and to communicate appropriately in order to don't disadvantage any of them.

These are personal comments and personal suggestions as a "normal" volunteer. --Ilario (talk) 10:18, 3 November 2014 (UTC)[reply]

Thank you for your thoughtful comments, Ilario. We do need to have programs in communities, and I agree that programs do need some support to sustain and grow impact. However, I do not feel that this support is necessarily financial; my conversations with our team here and preliminary discussions with volunteers makes me believe that a significant gap is *non-monetary* support, i.e. ways in which to focus and deepen program strategy and impact, and to share that learning across contexts for adaption and further innovation. If you consider the impact you are seeing as a GAC and IEG Committee member, I am sure you will agree that it is not simply money that our volunteers need and seek, nor is it money that is the primary motivating factor: acknowledgement, appreciation and resources/tools beyond money to do work effectively is what seems to be a key need. I think we should pivot towards focusing on that.
About diversity, I think our movement needs to start thinking in terms of trade-offs and scenarios: if this money for annual plan grants were to be used differently, what would it be used for? We have contexts where certain Wikimedia organizations and communities are already established, and can get multiple kinds of support (not only funds but non-financial resources). We also have other contexts where our communities and organizations are only just starting to grow and show great promise (besides being critical to fulfilling our vision). Where should the balance of movement resources go? I agree with you that we should continue to support and sustain existing and older communities, but again, this sustenance is not always financial. I would very much like to hear from community members like you what you feel are the most important ways for us to sustain and grow these older communities and to attract new volunteers into the mix. thanks, (for Lila) ASengupta (WMF) (talk) 07:49, 5 November 2014 (UTC)[reply]


  • Adding to second bullet: the document notes «the rate of growth of readership and contributorship in the Global North is either stable or declining». In fact, even some "top Wikipedias" are collapsing, with two-digits yearly decreases in activity: the data calls to action, not to surrender. --Nemo 07:40, 4 November 2014 (UTC)[reply]

Tracking in-kind resources[edit]

The ED mentions this twice:

"Wikimedia organizations should ... track the in-kind resources they receive."
and
"Reports will begin measuring in-kind resources as well as funds".

It seems entirely appropriate. However, a number of issues may need to be clarified for WMF affiliates and other WMF grant recipients:

  1. Are in-kind resources somewhere defined/exemplified, and if so, is it in the right level of detail (and in particular should they include paid and volunteer labour)?
  2. Are there guidelines for how to express the equivalent monetary value of in-kind resources?
  3. Should estimates of in-kind resources be set out in grant proposals as well as the reports the ED has specified?

Tony (talk) 09:24, 8 November 2014 (UTC)[reply]

Hi Tony, thanks for your excellent question. We found the lack of measuring in-kind resources to be a critical gap across our proposals and reporting this past year, and so this year, we've begun to explore exactly what this means. Before defining it too definitively (sic), we've asked for the APG applicants to offer their current categories of in-kind resources, and an estimation of what they might have cost if monetised. This is likely to be much more useful if based on local rather than global estimates of costs, since they will vary greatly across context. We've also piloted this in our current round of voluntary reporting: we're asking about an estimation of volunteer hours, meeting/event spaces, food/drinks, prizes/giveaways, and materials/equipment. It's likely these as well as additional categories (like pro bono communications or tech support) might be the most helpful. Hope this helps, ASengupta (WMF) (talk) 18:51, 12 November 2014 (UTC)[reply]
Retrieved from "https://meta.wikimedia.org/w/index.php?title=Grants_talk:APG/Funds_Dissemination_Committee/Advisory_Group/Recommendations/2014/ED_Response&oldid=10510110"