WikiLearn - Introduction to Partnership Building (computer-graded exercises) - video: Evaluation methods

@metadata	sourceLanguage "en" priorityLanguages "es" "fr" "ar" "ru" "zh" allowOnlyPriorityLanguages true description "video in Introduction to Partnership Building (computer-graded exercises) - Basics of partnership building for Wikimedia volunteers, with computer-graded exercises." label "WikiLearn - Introduction to Partnership Building (computer-graded exercises) - video: Evaluation methods"
display_name	"Evaluation methods"
subtitle-2210-13340-1	"So"
subtitle-14330-18080-2	"this lesson is not really a course on evaluation."
subtitle-18830-23450-3	"That's a whole field that can be gone into at some length."
subtitle-23450-30080-4	"And we actually have some good trainings and tutorials and recorded videos of trainings on"
subtitle-30080-32810-5	"evaluation by my colleagues across the movement."
subtitle-34340-39170-6	"And I'm linking to them in the resources page at the end of the slides at the end of"
subtitle-39170-40190-7	"the presentation."
subtitle-40400-47660-8	"So don't forget to look if you don't see it right now during the presentation, don't"
subtitle-47660-54410-9	"forget to download the slides at the end and look at these resources so that you can"
subtitle-54410-56090-10	"familiarize yourself with it."
subtitle-56900-63320-11	"To your question: No, a logic model is not necessary in evaluations."
subtitle-63920-69920-12	"It is a popular tool popularized by some of the professional evaluators in the movement."
subtitle-70610-72590-13	"But it is not absolutely necessary."
subtitle-72590-74120-14	"It is a tool."
subtitle-74120-75290-15	"It is one tool."
subtitle-75560-83600-16	"And I want to stress: when we evaluate, we are basically trying to understand whether"
subtitle-83600-90590-17	"our investment, what we put in the time, the effort, the equipment, whatever, is achieving"
subtitle-90590-93200-18	"our goals, our outcomes."
subtitle-93920-99860-19	"And one way to do that is by linking it in a visual diagram that's popularized under the"
subtitle-99860-101120-20	"name "logic model"."
subtitle-101120-103280-21	"I don't think it's a good name, but that's the name."
subtitle-104450-108800-22	"And remember that we already have a lot of the groundwork for this already made in our"
subtitle-108800-113330-23	"theory of change. We have indicators, we have thresholds of success, and we know the"
subtitle-113330-116540-24	"group in which we measure that indicator."
subtitle-118040-122660-25	"So I am going to introduce the logic model, but it is not the only way to do it."
subtitle-124220-129290-26	"What we should always do is collect the data in real time."
subtitle-129290-135260-27	"If we want to evaluate later, at some point, we need to have the data to evaluate and it's"
subtitle-135260-139250-28	"a lot easier if we collect the data in real time."
subtitle-140150-143780-29	"If we don't collect it in real time, it will be very difficult to have a proper and"
subtitle-143780-146930-30	"representative evaluation later."
subtitle-146930-153500-31	"So in order to collect in real time as painlessly as possible, I encourage you to"
subtitle-153500-158990-32	"prepare a convenient, customized way to record the information."
subtitle-158990-161240-33	"And this can be whatever works for you."
subtitle-161240-166040-34	"It can be a little notepad, a little physical notepad that you scribble into."
subtitle-166070-167930-35	"Doesn't work for me, but maybe it works for you."
subtitle-168410-170510-36	"It can be a neat spreadsheet."
subtitle-170540-176390-37	"It can be a task management system like Asana or Trello or whatever, whatever works."
subtitle-176390-180500-38	"But something that is easy to use in real time that wouldn't be too bureaucratic so"
subtitle-180500-185630-39	"that you can just feed data into it as it happens, while still preserving the minimal"
subtitle-185630-190490-40	"context like a date and an amount, so that later when you have an aggregate, when you"
subtitle-190490-193610-41	"have a collection of such data, it's not a mess."
subtitle-194030-195680-42	"You know, this is from that date."
subtitle-195680-196700-43	"This is from this date."
subtitle-197360-203840-44	"That's what you need. So suppose you are trying to track the indicator "Do we have"
subtitle-203840-209900-45	"more than 50% female attendees at a Wikipedia training", because let's say that's"
subtitle-209900-216080-46	"a focus of my particular program to specifically target women contributors."
subtitle-217370-224300-47	"So to know how I'm doing, I obviously need to count the total number of attendees, and"
subtitle-224300-226790-48	"then I need to count the number of women attending."
subtitle-228690-234000-49	"And then I need to document these two numbers, plus the date and maybe the name of"
subtitle-234000-238500-50	"the event. So it's as simple as that; that's collecting the data."
subtitle-243190-246580-51	"Like I said, logic models, I think they have a scary name."
subtitle-246580-253900-52	"It's basically a way to visually express the causality, the cause and effect that we"
subtitle-253900-259420-53	"expect. Leading from the investment, from what we put in, to the outcomes."
subtitle-259420-264520-54	"And we've already seen a kind of outcome map in our theory of change earlier."
subtitle-264760-268030-55	"And a logic model is another way to present it."
subtitle-272080-280910-56	"And logic models can help us realize that our inputs have not been leading to the"
subtitle-280910-282620-57	"outcomes we were hoping for."
subtitle-282710-289760-58	"And then the logic model is a visual way to examine where is the weakest link, where is"
subtitle-289760-295370-59	"the assumption that we made about cause and effect that isn't true?"
subtitle-295400-300650-60	"Where the cause, what we thought was the cause isn't having that effect."
subtitle-301640-308540-61	"So we may be investing certain inputs that are not achieving the outputs that we expect."
subtitle-308840-313940-62	"Logic models make this distinction between outputs, which are kind of immediate things"
subtitle-313940-319820-63	"that happen, events that we do, products that we create, documents that we create, the"
subtitle-319820-321770-64	"people that we talk to or reach."
subtitle-321770-323640-65	"These are outputs."
subtitle-323640-330200-66	"But outcomes are about the goals are about changed behavior or changed society or skills"
subtitle-330200-333410-67	"that are being put to use beyond the event."
subtitle-334010-337880-68	"So maybe our inputs aren't even achieving the outputs."
subtitle-338210-345770-69	"So we think we're training, but we're not actually reaching people or we have a"
subtitle-345770-350150-70	"promotional campaign, but people aren't actually seeing it for whatever technical"
subtitle-350150-357740-71	"reason, or we are achieving the outputs, we are running workshops, we are having events,"
subtitle-357740-362900-72	"but we're not achieving the outcomes, meaning that the people go through the"
subtitle-362900-367070-73	"training, but the outcome of having more editors is not achieved."
subtitle-367070-368840-74	"They don't continue to edit."
subtitle-370520-374870-75	"Again, this distinction between outputs and outcomes has confused many, many people"
subtitle-374870-376160-76	"across the movement."
subtitle-376160-377930-77	"I think better names would have been less confusing."
subtitle-378740-381470-78	"The point is, the output is easy to measure."
subtitle-381470-382490-79	"It's what you do."
subtitle-382790-384800-80	"If you had an event, that's an output."
subtitle-384800-387050-81	"If you produce a document, that's an output."
subtitle-387050-393080-82	"Whereas the outcomes are the things we expect to see happen in the world, beyond our"
subtitle-393080-394160-83	"direct control."
subtitle-396630-403440-84	"So these logic models can help us debug our programs or just track them to see that they"
subtitle-403440-408050-85	"are successful. So I'm going to show you two kinds of diagrams very briefly."
subtitle-408060-415170-86	"Again, there are full instruction sessions already on Meta and on Commons, in slides and"
subtitle-415170-419280-87	"in video, which is why I'm not going into it in detail today."
subtitle-419520-425160-88	"But the simplest diagram is called A Stairway to Impact, and it's a simplified"
subtitle-425160-429990-89	"logic model in that it tracks a single chain of causality."
subtitle-430950-434790-90	"So just a single chain of "because of this, that", etc."
subtitle-434940-436770-91	"and this is what it can look like."
subtitle-436800-439110-92	"You start reading it from the bottom."
subtitle-440730-442980-93	"So there is some kind of an external situation."
subtitle-443910-448770-94	"And because of that external situation, we have decided to put certain inputs or"
subtitle-448770-458230-95	"resources: Time, money, volunteer effort, into performing certain activities"
subtitle-458950-460060-96	"like events, etc.."
subtitle-460070-461350-97	"These are outputs."
subtitle-461380-471010-98	"Activities. The participation in our work, trainees showing up; Partners; Anyone"
subtitle-471010-474010-99	"participating, is also a kind of output."
subtitle-474340-480280-100	"And if we have some kind of satisfaction survey at the end of the activity, that is"
subtitle-480280-482140-101	"also an output because it's immediate."
subtitle-482140-483790-102	"It's right after the activity."
subtitle-484300-490630-103	"All of these can be kinds of outputs of my input, of what I put in, the effort I put in."
subtitle-491050-496570-104	"But then, after the event or the intervention or the action is over, there are"
subtitle-496570-500410-105	"the outcomes that I'm expecting, the desired changes I'm expecting."
subtitle-500410-505960-106	"So, for example, I'm hoping that my target group is applying the skills that they've"
subtitle-505960-511390-107	"gained or is engaging in editing more than they edited before, or that they are now"
subtitle-511390-514750-108	"contributing pictures to Commons, which they didn't before."
subtitle-515380-521440-109	"Or that now that they understand Notability better, a greater percentage of their new"
subtitle-521440-524950-110	"articles survives deletion discussions."
subtitle-525490-535390-111	"There's all kinds of changes we might define and these changes can be short term"
subtitle-535390-536770-112	"or longer term."
subtitle-536980-542530-113	"Ideally, we want our investment to keep on giving, in the sense that the people we have"
subtitle-542530-547000-114	"directly influenced are already influencing other people, etc.."
subtitle-547720-553780-115	"So this is an abstract description of this diagram and here is a concrete example of"
subtitle-553780-556420-116	"this exact same diagram about a concrete project."
subtitle-556900-562300-117	"In this case, the example is an in-person Wikidata editing workshop."
subtitle-562810-565390-118	"So what is the input in such a case?"
subtitle-565390-570130-119	"It is our staff and volunteer time, the costs of the venue, the hosting, the"
subtitle-570130-571360-120	"material, the curriculum."
subtitle-571360-576980-121	"These are all things we put in to the Wikidata editing workshop."
subtitle-576980-583120-122	"And then the actual output, the thing that happened in the world is a half day Wikidata"
subtitle-583120-585850-123	"editing workshop that takes place."
subtitle-586330-592660-124	"And the participation aspect of it, is that 18 people in this case took part in the"
subtitle-592660-598060-125	"editing workshop, and we did a satisfaction survey of them and they gave positive"
subtitle-598060-599320-126	"feedback on the workshop."
subtitle-599320-602170-127	"So they liked it, they enjoyed it, they thought it was useful."
subtitle-602950-609670-128	"So far I only know what I put in and what I was able to make happen in the world."
subtitle-610150-614950-129	"I got these 18 people in a room and I trained them on Wikidata."
subtitle-615630-618780-130	"I don't know whether I actually reached my goals."
subtitle-619860-621690-131	"This is the outcomes, in red."
subtitle-622590-627360-132	"Are the participants motivated to use Wikidata for Wikipedia and know how to do it?"
subtitle-627630-628950-133	"I need to measure that."
subtitle-629430-633570-134	"Do they incorporate Wikidata items in their daily Wikipedia work?"
subtitle-635050-639210-135	"Do they gain greater benefits from using Wikidata for their work on Wikipedia?"
subtitle-639220-645910-136	"And finally, the multiplier effect: Do they themselves now cause other people to get"
subtitle-645910-647200-137	"engaged in Wikidata, etc.?"
subtitle-647200-649360-138	"And these are of course harder to measure."
subtitle-649360-653200-139	"Outcomes are always harder to measure than the outputs."
subtitle-653200-662920-140	"But this is an example of a linear chain of of causality called the Stairway to Impact."
subtitle-664450-666340-141	"It's quite limited."
subtitle-666970-671530-142	"So if we want a fuller view of a partnership that can have multiple activities going on"
subtitle-671530-681520-143	"and multiple goals, we use the fuller logic model diagram, which looks a little busier,"
subtitle-681520-683590-144	"and this one is read from left to right."
subtitle-683590-684910-145	"But the principle is the same."
subtitle-684910-686650-146	"There's a whole bunch of inputs."
subtitle-687220-693070-147	"What we invest: time, planning, money, knowledge, expertise, materials, all the"
subtitle-693070-700330-148	"costs, and then there's what we do and create: The documents, the articles, the"
subtitle-700330-708130-149	"templates, the the people who participate, and those ideally lead to all kinds of"
subtitle-708130-716620-150	"outcomes: new skills being applied, new behavior that we can track, decisions being"
subtitle-716620-723070-151	"made that were not made before, things beyond the actual intervention, beyond the"
subtitle-723070-729130-152	"event, or beyond our direct efforts, things that keep on giving are these outcomes."
subtitle-729670-735640-153	"Again, this is an abstract diagram, and here it is in concrete, a concrete example again"
subtitle-735640-740950-154	"for the Wikidata editing workshop: we put in the time and the hosting and the material,"
subtitle-741130-750910-155	"and then a bunch of editors are reached, and we document these, so that we know the effort"
subtitle-750910-757090-156	"that we put in, and how many people attended, what they learned, etc."
subtitle-757630-763510-157	"But ideally what we want is to measure the outcomes with the indicators that we set."
subtitle-765130-771400-158	"Is this knowledge that we gave them actually utilized, are they motivated to use Wikidata,"
subtitle-772210-773290-159	"etc., etc.?"
subtitle-775210-777310-160	"I know this doesn't make sense to everyone."
subtitle-777310-783490-161	"I have seen people really love logic models and feel that it really helps organize their"
subtitle-783490-785320-162	"thinking around evaluation."
subtitle-785320-789970-163	"And I have seen other people become really miserable when they thought they had to use"
subtitle-789970-792250-164	"them. You don't have to use them."
subtitle-792250-798400-165	"If there is some other way that makes sense to you to organize your evaluation, that is"
subtitle-798400-800440-166	"absolutely fine."