User:MartinPoulter/Tales of Evaluation

Here's a story linking events that have happened to Wikimedia UK. We'll draw lessons from it about how we plan and evaluate the activity of a Wikimedia organisation.

A series of unlikely events

Back in early 2011 we held the first Bristol Wiki Academy; a day event at the University of Bristol. As well as local (and one international) Wikipedians, we invited anyone with any interest in Wikipedia - not separate, specialised events as we'd do nowadays. People came wanting to learn to edit, to possibly share content from their workplace, or to learn how Wikipedia works. We hoped to get new Wikipedia editors from the local Somali community, or to recruit new Campus Ambassadors. We didn't really achieve that, but useful partnerships came out of the day, including with the conservation charity ARKive and with Girl Geeks. Matt Jukes, at that time working for the Joint Information Services Committee, attended out of curiosity and wrote a blog post.

Later in the year, Matt moved from JISC to the Medical Research Council. He and his colleagues saw media reports that Wikimedia UK had been training staff at Cancer Research UK so he asked me for a similar event. A team of Wikipedians spent the day at MRC headquarters. We got people to take initial steps as Wikipedia contributors, but didn't overhaul a lot of articles or create many dedicated new Wikipedia contributors that day.

Side note: Frank Norman, a medical librarian, attended and worked on an article that day. He was later one of the instigators of a series of Women in Science editathons at his workplace, the National Institute of Medical Research. When I bumped into him in my capacity as the Jisc Wikimedian Ambassador, he suggested starting events again, and we ran a training workshop for medical researchers. Frank is now a very useful contact at the Crick Institute.

One of Matt's former JISC colleagues, Philip Pothen, was now working for the Arts and Humanities Research Council. He came along to the MRC event, just for the initial presentation on Wikipedia's impact. I had a meeting with him on a later date and he started a proposal to use AHRC money on a project with Wikimedia. Negotiations continued with Fae and the British Library, and Philip released money to fund a Wikimedian In Residence at the Library.

Side note: Matt later moved on to the Office of National Statistics. Once again, he asked Wikimedia UK in, and John Cummings gave a presentation and workshop. Initially interested in Wikipedia, the ONS staff were more interested to learn about Commons and Wikidata, and how their graphs and data could reach a huge audience with relatively little work.

As the WIR, Andrew Gray (for it was he) not only worked in the Library but visited universities around the UK to give training workshops. One of these was Oxford University. At Oxford, the Bodleian Libraries (the second biggest library in the country) were interested to hear what the British Library (the biggest library in the country) were doing with Wikipedia. Women in Science editathons became a regular part of the calendar and the Bodleian eventually, with funding from Wikimedia UK, employed their own Wikimedian, officially uploading some of their digital content to Commons for the first time.

Side note: One of the instigators of the Ada Lovelace editathons at Oxford, Melissa Highton, went on to be head of IT at the University of Edinburgh, and has since energetically promoted Wikipedia contributions by staff there, doing great work to counter the gender gap. She created the university's Wikimedian In Residence post in 2016, recruiting Ewan McAndrew, with all sorts of positive results within and beyond Edinburgh.

Another organisation watching the British Library was JISC, which had been re-incorporated as a charity and renamed "Jisc". Jisc and Wikimedia UK had worked together a few times since Matt Jukes was in Bristol, most notably at a World War I editathon and at the EduWiki conference 2012 in Leicester. With matched funding from Wikimedia UK, they employed me as their Wikimedia Ambassador. One of the activities was to run events with institutions that had had Jisc funding. The Wellcome Library was one such institution: a group of Wikipedians and Wellcome Librarians had a rewarding day running a public training event. The experience left us all energised and keen to do more for mutual benefit.

Side note: Similar, branching stories of contact leading to other activities could be told about Wikimedia UK's relationship with Jisc, or with the Wellcome Trust, or with Imperial College, London, or other institutions I know less about.

So I find myself in the Summer of 2015, employed by the Bodleian Libraries, having a platform to reach out to many prestigious libraries and museums who hadn't thought of working with us. In my volunteer life, I'm organising a science conference which has been made possible by the enthusiastic co-operation of the Wellcome Library and Wellcome Trust, benefiting from contacts at the Crick Institute and at Cancer Research UK, with speakers from the University of Edinburgh and from Wikipedia scientists that I've worked with in the past. It's all been made easy for me by fellow volunteers, WMUK staff and various contacts in partner institutions.

A series of obvious observations

• No one would have seriously proposed a paid Wikipedian post at the British Library as an outcome, even an indirect one, of a Medical Research Council workshop. Just to be clear, the British Library post came about through Fae's work, but Matt inviting Philip to attend my presentation was an instigating factor. There is much less work involved in proposing a project than in negotiating and planning the details, but for these projects to happen someone has to be persuaded they are a good idea.
• No one would have seriously proposed that running an open event in Bristol would result (indirectly) in National Statistics contributing to Wikimedia projects. No one from the Office of National Statistics was aware of the event at the time.
• ...and so on. You can make similar observations about the other outcomes.
• A lot of these activities didn't achieve much on their prospective measures of success but, in retrospect, achieved things we hadn't dared to imagine.

So why do things keep going unexpectedly well for us?

Black Swans

Nassim Nicholas Taleb has written a series of books about errors in decision-making that plague both individuals and organisations- even (and especially), efficient, rationally-managed organisations. The tendency to ignore possibilities because they are very improbable draws particular ire from Taleb. These "Black Swan" events are so unlikely that we don't plan for them, but there are so many such events that it is common for some sort of Black Swan to be happening.

Taleb's advice is most often applied to investors. For an investor with a well worked-out portfolio, Black Swan events are things like a bank collapse, or a national currency default, that might wipe out your whole investment. These things might be wildly improbable at any given time, but the fact that their occurrence would wipe you out is a reason to model them and hedge against them. You can get a very rose-tinted picture of your investments if the main downside risks are things you don't take into account.

Black Swan events aren't necessarily catastrophes. They can also be eucatastrophes; unexpected positive events. In a system which is well understood, the Black Swans are negative events. In a system which is new and open-ended, there are more opportunities for unexpected benefits.

For Wikimedia organisations, so far, the Black Swan events are usually positive; the "risks" are all on the upside. If you aim to train five newcomers to edit Wikipedia, the worst that can happen is that they don't edit ever again. The best that can happen is... well, the sky's the limit. Maybe none of the trainees will ever edit again, but they'll persuade their boss that open knowledge is important enough a cause to spend tens of thousands of pounds on. Maybe they'll not touch Wikipedia but release all their work under Wikipedia-compatible free licences from then on.

Conclusion

It's a welcome development that we measure the impact of our activities, so we can judge if best use is made of staff time and donors' money. I welcome that the evaluation is often not numeric but in narrative form. The metrics have to be sensible and have to actually measure the success of what we do.

In the UK university sector, the focus on league table positions and similar metrics is being actively discouraged by the funders, who recognise that perverse incentives from these metrics can be irrelevant to, or go against, making a better university.

If we were measuring the impact of ways of running a jam factory, we would just count the number of pots of jam coming off the end of the conveyor belt in a day. Our situation is as different from this as it's possible to imagine. The main positive outcomes of our work are unpredictable, causally distant from the work itself, and not well-proportioned to the effort put in.

One of my Wikimedia UK colleagues has suggested that financial planning is simple because we know what everything costs, so it's just a matter of looking at how much money we have and what we want to do. We're actually in as different a situation from that as it's possible to imagine. Maybe the next big partnership will come from an expensive outreach project, or from getting a volunteer to a meeting by funding their train ticket.

Our situation is more like a venture capitalist in a new industry, where we know that a proportion of money spent (but not which money) will be wasted, while another proportion of money will generate fantastic returns. Beyond performing due diligence that the money is spent on the right sort of things by competent, trustworthy people, we can't know what is wasteful and what is a productive investment; before, during, or even some time after the activity.

Intermediate outcomes

One way to simplify the problem of evaluation is to concentrate on intermediate outcomes which are hard to quantify but which don't take years to materialise. Another of my colleagues has suggested that the metrics of success for Wikimedia UK are just the number of edits made and number of files uploaded. This is exactly the wrong approach. The things Wikimedia UK produces are less tangible: a more favourable national media environment, greater appreciation of Wikimedia projects among educators and researchers, greater openness to including Wikimedia projects in cultural or academic workflows, and so on.

Changing an researcher's attitude from hostility to general positivity will not result in any files uploaded or other visible outcomes, but the final change from positivity to enthusiasm might. The first activity might be necessary for the second, so the effect of the first has to be measured, and that's why we have to be in the messy business of measuring attitudes and intentions, setting changes in these as measures of our impact.