@metadata | sourceLanguage | "en" |
---|
priorityLanguages | |
---|
allowOnlyPriorityLanguages | true |
---|
description | "video in Introduction to Partnership Building (computer-graded exercises) - Basics of partnership building for Wikimedia volunteers, with computer-graded exercises." |
---|
label | "WikiLearn - Introduction to Partnership Building (computer-graded exercises) - video: Evaluation plan example" |
---|
|
---|
display_name | "Evaluation plan example" |
---|
subtitle-600-3720-1 | "So here's a very simple example." |
---|
subtitle-4860-9210-2 | "Let's say we're evaluating a series of editing workshops at the museum." |
---|
subtitle-9870-15390-3 | "So our plan might look something like this, and I kept it simple so it fits on a slide." |
---|
subtitle-16110-21720-4 | "Every workshop, we would ensure that participants are registered in the course in" |
---|
subtitle-21720-23550-5 | "the Programs and Events Dashboard." |
---|
subtitle-24300-27570-6 | "So this is something that has to happen every workshop." |
---|
subtitle-28460-35180-7 | "Then, at the end of each workshop, we're going to poll all the participants using an" |
---|
subtitle-35180-39470-8 | "online survey to gauge their satisfaction." |
---|
subtitle-39560-44600-9 | "And the questions in the survey would be, and here you should have some detail." |
---|
subtitle-44630-48770-10 | "I didn't go into it in this example, but obviously this survey needs to be planned." |
---|
subtitle-49070-52550-11 | "What exactly are you going to ask them, after every workshop?" |
---|
subtitle-54270-58620-12 | "Additionally, every two months, we will gather statistics from the Programs and" |
---|
subtitle-58620-59830-13 | "Events Dashboard." |
---|
subtitle-59830-63300-14 | "That dashboard, once the people are registered, those statistics are just" |
---|
subtitle-63840-64920-15 | "available any time." |
---|
subtitle-64920-70560-16 | "So every two months we're going to collect these numbers, record them somewhere, and" |
---|
subtitle-70560-77130-17 | "then we would compare the number of editors retained to our planned thresholds of" |
---|
subtitle-77130-82980-18 | "success. Because we were hoping in these editing workshops that, let's say, 10% of the" |
---|
subtitle-82980-89190-19 | "editors are retained, 10% of the people we train, continue to edit after two months," |
---|
subtitle-89190-90720-20 | "after four months, etc." |
---|
subtitle-91260-95100-21 | "By the way, 10% is a pretty good number of retention." |
---|
subtitle-99200-104390-22 | "And another thing we will do every two months or four months, whatever, we're going" |
---|
subtitle-104390-105800-23 | "to use ORES." |
---|
subtitle-106070-110930-24 | "For those of you who don't know ORES is a very exciting technology of machine learning" |
---|
subtitle-110960-119420-25 | "that is able to use machine learning to assign a predicted quality score to an" |
---|
subtitle-119420-126500-26 | "article. So it can predict this article looks like "B class" or "C class" or a "good" |
---|
subtitle-126500-127640-27 | "article" or whatever." |
---|
subtitle-128240-133190-28 | "And it is not better than a human assessing the article, but it is a lot faster." |
---|
subtitle-133610-142400-29 | "So ORES is useful to do lateral assessment of a large group of articles." |
---|
subtitle-142400-148280-30 | "Suppose we have 300 articles or 500 articles and we want to see how, whether and how, they" |
---|
subtitle-148280-149300-31 | "have improved." |
---|
subtitle-149300-152720-32 | "We can use ORES in the supported languages." |
---|
subtitle-152780-157310-33 | "English is supported, a bunch of other languages are supported, and any language can" |
---|
subtitle-157310-160700-34 | "be supported, but you need to do some work for it." |
---|
subtitle-160700-169940-35 | "You need to help by having humans classify manually several thousand articles so that" |
---|
subtitle-169940-171380-36 | "the machine can learn from it." |
---|
subtitle-171680-174050-37 | "And this was already done in English and other languages." |
---|
subtitle-174050-175300-38 | "So it's available." |
---|
subtitle-175300-179930-39 | "If you come from a smaller language where ORES doesn't work, you can get in touch with" |
---|
subtitle-179930-186170-40 | "the ORES developers -- I can help you do that, let's talk later -- and you can help" |
---|
subtitle-186170-190100-41 | "them train the machines so that your language is also supported." |
---|
subtitle-190190-195980-42 | "But anyway, in this example, let's say we're going to use ORES to assess the quality of" |
---|
subtitle-195980-201110-43 | "all the articles in the museum's list of articles they care about, list of relevant" |
---|
subtitle-201110-207440-44 | "articles. And then we can see which of these articles has improved in these two months." |
---|
subtitle-207440-214430-45 | "And then we can go and see: "Okay, this article has improved from stub to B class." |
---|
subtitle-214790-220370-46 | "I'm going to check whether the improvement involves our volunteers", because of course" |
---|
subtitle-220370-224690-47 | "someone else might have improved the article and that's great, but it wouldn't be impact" |
---|
subtitle-224690-225740-48 | "of my program." |
---|
subtitle-226460-230600-49 | "So this is an example of something very specific that I'm going to do in order to" |
---|
subtitle-230600-237470-50 | "attempt to show that these editing workshops actually improved the quality of the articles" |
---|
subtitle-237470-239210-51 | "that the museum cares about." |
---|
subtitle-239510-245360-52 | "So you can see it's a very relevant metric and I am planning, I'm putting it in my plan," |
---|
subtitle-245360-252920-53 | "how I'm going to measure it, how I'm going to try to show whether and how much I am" |
---|
subtitle-252920-259670-54 | "achieving this goal of improving quality of material that the museum cares about." |
---|
subtitle-260650-262720-55 | "I hope this was a clear example." |
---|
subtitle-263110-267100-56 | "Again, the evaluation plan doesn't have to be much more complicated than this." |
---|
subtitle-267100-269280-57 | "I really just simplified it with..." |
---|
subtitle-269280-271810-58 | "I left out dates and and the details of the survey." |
---|
subtitle-272230-276790-59 | "But something like this, a page, a page and a half, can be an evaluation plan." |
---|
subtitle-276790-280030-60 | "It doesn't have to be a 30 page bureaucratic nightmare." |
---|