WikiLearn - Introduction to Partnership Building (computer-graded exercises) - video: Specifying outcomes for evaluation

@metadata	sourceLanguage "en" priorityLanguages "es" "fr" "ar" "ru" "zh" allowOnlyPriorityLanguages true description "video in Introduction to Partnership Building (computer-graded exercises) - Basics of partnership building for Wikimedia volunteers, with computer-graded exercises." label "WikiLearn - Introduction to Partnership Building (computer-graded exercises) - video: Specifying outcomes for evaluation"
display_name	"Specifying outcomes for evaluation"
subtitle-30-10580-1	"Now"
subtitle-10580-14810-2	"it's time to zoom in. So we have some kind of list of outcomes and interim outcomes"
subtitle-14810-17450-3	"that, theoretically, one leads to the other."
subtitle-17750-20180-4	"Now we can break it down and define it better."
subtitle-21110-28560-5	"We want to define the outcome, using an indicator, a measured group and a threshold"
subtitle-28560-36060-6	"of success. The indicators are measurable quantities – אhey have to be measurable –"
subtitle-36690-41280-7	"that can tell us how close we are to the desired outcome."
subtitle-42180-46650-8	"And [someone] immediately asks "Are you saying that all positive outcomes are"
subtitle-46650-49860-9	"necessarily measurable?" No."
subtitle-49860-52890-10	"There could be outcomes that are difficult to measure."
subtitle-52890-58920-11	"But I am saying that [for] the outcomes we cannot measure, we also cannot tell whether"
subtitle-58920-60270-12	"we have achieved them or not."
subtitle-61170-66450-13	"If you feel that you can define an outcome that is on the one hand not measurable, but"
subtitle-66450-71580-14	"on the other hand you will know it when you see it, then I will suggest to you that you"
subtitle-71580-73080-15	"probably can measure it."
subtitle-73080-78660-16	"Whatever your intuitive understanding of "Oh, we have achieved this", it can probably"
subtitle-78660-88380-17	"be somehow emulated or communicated using a quantifiable measure, even if a quantifiable"
subtitle-88380-94590-18	"measure is a survey of people's intuition, because that survey could at least come up"
subtitle-94590-98940-19	"with some kind of number that could then be compared to that same number from last year,"
subtitle-98940-100950-20	"etc.. It's a whole art."
subtitle-101220-107190-21	"Evaluation is a whole art that we won't go into, but theoretically you can come up with"
subtitle-107190-113790-22	"a measurable aspect of things, even if it doesn't capture every nuance of the thing."
subtitle-113790-119160-23	"You can come up with some measurable aspect, and we will need to come up with a measurable"
subtitle-119160-121860-24	"indicator of the outcome of the desired outcome."
subtitle-122130-127380-25	"Again, even if it's a very qualitative thing, you can derive a quantitative thing"
subtitle-127380-132870-26	"from it using surveys, for example, like the survey, I give this class every week."
subtitle-133680-139050-27	"It's not a science, how good the lesson was, but if you are asked to rate it on a scale of"
subtitle-139050-141660-28	"1 to 5, then I do end up with an average number."
subtitle-142800-145770-29	"And I can compare that number to last week and to next week."
subtitle-146370-148230-30	"Does it capture all the nuances of my teaching?"
subtitle-148860-153930-31	"It does not, but it still tells us something about are things improving or not, etc.."
subtitle-154380-159840-32	"So again, there's a whole art and a whole discussion about proper evaluation, coming up"
subtitle-159840-162840-33	"with good metrics, etc., that we cannot go into right now."
subtitle-163110-169350-34	"The point is that the indicator is what we're looking at when we want to see how far"
subtitle-169350-172680-35	"we've come, or not, in our pursuit of this outcome."
subtitle-173820-178200-36	"And this indicator ideally should be specific and measurable and achievable and"
subtitle-178200-180900-37	"relevant and time bound, also known as SMART."
subtitle-182940-184680-38	"But again, we won't go too deeply into it."
subtitle-184680-186090-39	"We will give some examples."
subtitle-186390-191850-40	"The measured group defines what this indicator applies to."
subtitle-192000-196560-41	"So the indicator may be, for example, number of edits made."
subtitle-197610-199410-42	"Number of edits made by whom?"
subtitle-199860-201630-43	"By all editors?"
subtitle-201630-204860-44	"By French Wikipedia editors?"
subtitle-204860-208860-45	"By new French Wikipedia editors, meaning people who have been around for less than six"
subtitle-208860-213930-46	"months? We need to define what is the group that I am measuring the indicator for."
subtitle-214050-215820-47	"So that's what the measured group is."
subtitle-216330-222660-48	"It's what the indicator is measured for, or in the context of what it is measured."
subtitle-222660-226260-49	"And the threshold of success is again related to the indicator."
subtitle-226260-231960-50	"It's the number, the goal that we're hoping for or we're hoping to exceed for the"
subtitle-231960-236820-51	"indicator. So when the indicator in the measured group reaches the threshold of"
subtitle-236820-242310-52	"success, then we say, "okay, we have achieved this outcome." So it's like three"
subtitle-242310-247500-53	"different things. But actually it's an indicator with kind of two qualifying or"
subtitle-247500-251850-54	"contextualizing components."
subtitle-255630-259440-55	"[someone] says I'm wedded to the quantitative paradigm."
subtitle-259680-264810-56	"You could say that; I suppose I am teaching you how to build a theory of change that,"
subtitle-264810-272100-57	"yes, has quantitative elements that tell you how close you are to the outcome."
subtitle-272730-280200-58	"Again, I say there is a world of nuance and discussion on how to turn qualitative aspects"
subtitle-280200-283450-59	"into quantitative indicators."
subtitle-283470-286010-60	"We cannot expand on that, unfortunately."
subtitle-286020-289260-61	"So it's okay if you think that's a limited perspective."
subtitle-290220-298260-62	"But that's what we'll be teaching here: how to have theory of change with outcomes that"
subtitle-298260-299580-63	"are measurable."
subtitle-301230-309120-64	"By all means, augment this with storytelling, with inspiring examples, with"
subtitle-309120-310500-65	"videos, etc."
subtitle-310500-312240-66	"but that is in addition."
subtitle-313380-319370-67	"So here are examples of specifying outcomes."
subtitle-319370-324110-68	"And I've marked the indicators in blue, the groups in orange, and the thresholds in"
subtitle-324110-329300-69	"green. So taking the previous slide, I'm saying, okay, what if I say I want to"
subtitle-329300-332030-70	"increase active editors per month?"
subtitle-332030-333560-71	"That's my indicator."
subtitle-334290-340440-72	"Among the total registered accounts in Wikipedia in a specific Wikipedia, probably"
subtitle-340440-345300-73	"in a specific language, but this was from the Foundation's perspective, so actually it"
subtitle-345300-346500-74	"was completely global."
subtitle-347280-350430-75	"And I want to increase this, say, to 90,000 per month."
subtitle-351150-355020-76	"Today, we don't have 90,000 active editors per month."
subtitle-355350-357330-77	"I think it's something around 70,000."
subtitle-358260-360210-78	"So let's say we want to increase it to 90,000."
subtitle-361260-365610-79	"That's the overall indicator and the overall measured group."
subtitle-365610-368340-80	"And I have defined a threshold of success."
subtitle-368340-369780-81	"By the way, how did I define it?"
subtitle-369780-371580-82	"How did I pick 90,000?"
subtitle-371790-372870-83	"I just picked it."
subtitle-372900-375480-84	"It is my wish."
subtitle-375510-377130-85	"It should be a realistic number."
subtitle-377340-382200-86	"I mean, I could have also written I would like there to be 3 million active editors per"
subtitle-382200-384830-87	"month, but that's an unrealistic number."
subtitle-384830-389940-88	"Based on everything we know about, the dynamics of Wikipedia so far in all of its"
subtitle-389940-396630-89	"history, including its peak, we'll probably never reach 3 million editors, at least not"
subtitle-396630-398070-90	"in the coming, let's say, three years."
subtitle-398070-399330-91	"So I could have said that."
subtitle-399330-405180-92	"But it is better to pick a number that is at least conceivably achievable."
subtitle-406020-408630-93	"And then we break it down into the focus areas."
subtitle-409140-416160-94	"And in order not to make the slide too busy, I've specified only two of the interim"
subtitle-416160-420240-95	"outcomes, but of course all of them could be and should be specified."
subtitle-420240-421830-96	"So let's say awareness."
subtitle-422610-428520-97	"The indicator for awareness that I'm proposing is "percent aware Wikipedia is"
subtitle-428520-436050-98	"editable, among the surveyed students." And maybe I should be specific about which"
subtitle-436050-437640-99	"students, university or whatever."
subtitle-437850-440520-100	"And I want that percentage to be at least 70%."
subtitle-442560-447780-101	"Now, again, you could define other numbers or other populations, but the point is that"
subtitle-447780-453900-102	"this combination of three elements – what I am measuring, among whom or in what, out of"
subtitle-453900-458790-103	"what group, and what I'm hoping the measurement would be – that together is"
subtitle-458790-467580-104	"giving me a measurable indication of how well I'm doing in this particular aspect of"
subtitle-467580-474170-105	"awareness. Let's look at mentoring and help ."
subtitle-474270-476950-106	"Again, I'm going with an awareness indicator."
subtitle-476960-483380-107	"The percentage of people who are aware of mentoring options among three-month newbies,"
subtitle-483650-487000-108	"among people who have only been around for three months or less."
subtitle-487010-491750-109	"I want at least half of them to be aware that there are mentoring options, for"
subtitle-491750-498140-110	"example. And I want to make sure that the percentage of people who have used a"
subtitle-498140-502820-111	"mentoring option among the six-month newbies is at least 10%."
subtitle-502850-504410-112	"Again, I made up these numbers."
subtitle-505010-507580-113	"To do it properly would of course need some more thought."
subtitle-507590-513950-114	"But these are potentially-- these are conceivable indicators and thresholds of"
subtitle-513950-516770-115	"success for these outcomes."
subtitle-516770-521650-116	"And I want to demonstrate that not all of the interim outcomes are easy to quantify."
subtitle-521660-524210-117	"For example, motivation to join."
subtitle-525220-526510-118	"Motivation to join."
subtitle-527740-529510-119	"What does that mean exactly?"
subtitle-529510-530800-120	"How can I measure that?"
subtitle-530800-535330-121	"Exactly as [someone] notes, not everything is easy to quantify."
subtitle-535600-540670-122	"So maybe we need to break that down into an additional interim outcome."
subtitle-541380-544700-123	"And then that interim outcome may be easier to quantify."
subtitle-544710-548630-124	"Or maybe I just need to think harder and figure out a way to measure it."
subtitle-548640-551430-125	"If I don't measure it, then I may or may not achieve it."
subtitle-551430-557010-126	"I will never know. But it doesn't mean that it's not important, right?"
subtitle-557010-559590-127	"It just means I need to work harder to measure it."
subtitle-561100-563530-128	"Okay. So that's one example."
subtitle-565850-570110-129	"Here is another example going back to our Ghanaian youth example."
subtitle-570470-577070-130	"So I can say that my long term goal about Ghanaian youths is that, let's say, at least"
subtitle-577070-583790-131	"75% satisfaction rate in a survey, among the trained volunteers."
subtitle-585320-589130-132	"I'm not even sure that's the best measurement, but it's one possible"
subtitle-589130-594410-133	"measurement, right? So I will measure whether they have better job prospects by"
subtitle-594410-598260-134	"asking them if they feel they have better job prospects."
subtitle-598700-603110-135	"It's not exactly the same as having better job prospects, but it's an indication if they"
subtitle-603110-608480-136	"feel at least that they have better job prospects, then the training will have done"
subtitle-608480-609860-137	"something presumably."
subtitle-610550-611810-138	"We could think of other things."
subtitle-612020-617810-139	"We could, say, try to track the number of students, of trained volunteers who have"
subtitle-617810-622490-140	"landed a job in their profession, in software."
subtitle-623090-626330-141	"I mean, that may be harder to actually do, to track."
subtitle-626330-631610-142	"But if I am able to track it or if I may ask them really nicely to please report to us"
subtitle-631610-635510-143	"when they've found a job, then that could be a very good indicator, right?"
subtitle-637670-642610-144	"In terms of Wikipedia's volunteer base, that's probably easier to measure."
subtitle-642620-651020-145	"I can just count how many of the new active technical volunteers in 2021 are from Africa."
subtitle-651020-652730-146	"Let's say that's the indicator I want to track."
subtitle-653030-659750-147	"I want to see that my efforts in Ghana have actually made, created, more active technical"
subtitle-659750-662840-148	"volunteers from Africa in 2021."
subtitle-662840-667130-149	"I want to make sure at least ten of the new volunteers are from Africa."
subtitle-667610-669110-150	"That's one way I could measure it."
subtitle-670660-676120-151	"And then the interim goal, I can more concretely measure and say, I want to measure"
subtitle-676120-681610-152	"how many people I actually trained in Media wiki development and I want to make sure that"
subtitle-681610-687280-153	"I have trained at least 400 people among young software students."
subtitle-687580-691660-154	"So I'm taking out of all the software students in Ghana, I want to make sure I"
subtitle-691660-693370-155	"reach at least 400 people."
subtitle-693370-700420-156	"I would consider that a success Of course, these numbers depend on my capacity, right?"
subtitle-700420-705340-157	"If I have a lot of training capacity, maybe I want to target 4000 students."
subtitle-705580-712630-158	"But I hope you see how I'm breaking it down into measurable indicators so that I know how"
subtitle-712630-715090-159	"I'm doing on these outcomes."