WikiLearn - Introduction to Partnership Building (computer-graded exercises) - video: Evaluation plan example

@metadata	sourceLanguage "en" priorityLanguages "es" "fr" "ar" "ru" "zh" allowOnlyPriorityLanguages true description "video in Introduction to Partnership Building (computer-graded exercises) - Basics of partnership building for Wikimedia volunteers, with computer-graded exercises." label "WikiLearn - Introduction to Partnership Building (computer-graded exercises) - video: Evaluation plan example"
display_name	"Evaluation plan example"
subtitle-600-3720-1	"So here's a very simple example."
subtitle-4860-9210-2	"Let's say we're evaluating a series of editing workshops at the museum."
subtitle-9870-15390-3	"So our plan might look something like this, and I kept it simple so it fits on a slide."
subtitle-16110-21720-4	"Every workshop, we would ensure that participants are registered in the course in"
subtitle-21720-23550-5	"the Programs and Events Dashboard."
subtitle-24300-27570-6	"So this is something that has to happen every workshop."
subtitle-28460-35180-7	"Then, at the end of each workshop, we're going to poll all the participants using an"
subtitle-35180-39470-8	"online survey to gauge their satisfaction."
subtitle-39560-44600-9	"And the questions in the survey would be, and here you should have some detail."
subtitle-44630-48770-10	"I didn't go into it in this example, but obviously this survey needs to be planned."
subtitle-49070-52550-11	"What exactly are you going to ask them, after every workshop?"
subtitle-54270-58620-12	"Additionally, every two months, we will gather statistics from the Programs and"
subtitle-58620-59830-13	"Events Dashboard."
subtitle-59830-63300-14	"That dashboard, once the people are registered, those statistics are just"
subtitle-63840-64920-15	"available any time."
subtitle-64920-70560-16	"So every two months we're going to collect these numbers, record them somewhere, and"
subtitle-70560-77130-17	"then we would compare the number of editors retained to our planned thresholds of"
subtitle-77130-82980-18	"success. Because we were hoping in these editing workshops that, let's say, 10% of the"
subtitle-82980-89190-19	"editors are retained, 10% of the people we train, continue to edit after two months,"
subtitle-89190-90720-20	"after four months, etc."
subtitle-91260-95100-21	"By the way, 10% is a pretty good number of retention."
subtitle-99200-104390-22	"And another thing we will do every two months or four months, whatever, we're going"
subtitle-104390-105800-23	"to use ORES."
subtitle-106070-110930-24	"For those of you who don't know ORES is a very exciting technology of machine learning"
subtitle-110960-119420-25	"that is able to use machine learning to assign a predicted quality score to an"
subtitle-119420-126500-26	"article. So it can predict this article looks like "B class" or "C class" or a "good"
subtitle-126500-127640-27	"article" or whatever."
subtitle-128240-133190-28	"And it is not better than a human assessing the article, but it is a lot faster."
subtitle-133610-142400-29	"So ORES is useful to do lateral assessment of a large group of articles."
subtitle-142400-148280-30	"Suppose we have 300 articles or 500 articles and we want to see how, whether and how, they"
subtitle-148280-149300-31	"have improved."
subtitle-149300-152720-32	"We can use ORES in the supported languages."
subtitle-152780-157310-33	"English is supported, a bunch of other languages are supported, and any language can"
subtitle-157310-160700-34	"be supported, but you need to do some work for it."
subtitle-160700-169940-35	"You need to help by having humans classify manually several thousand articles so that"
subtitle-169940-171380-36	"the machine can learn from it."
subtitle-171680-174050-37	"And this was already done in English and other languages."
subtitle-174050-175300-38	"So it's available."
subtitle-175300-179930-39	"If you come from a smaller language where ORES doesn't work, you can get in touch with"
subtitle-179930-186170-40	"the ORES developers -- I can help you do that, let's talk later -- and you can help"
subtitle-186170-190100-41	"them train the machines so that your language is also supported."
subtitle-190190-195980-42	"But anyway, in this example, let's say we're going to use ORES to assess the quality of"
subtitle-195980-201110-43	"all the articles in the museum's list of articles they care about, list of relevant"
subtitle-201110-207440-44	"articles. And then we can see which of these articles has improved in these two months."
subtitle-207440-214430-45	"And then we can go and see: "Okay, this article has improved from stub to B class."
subtitle-214790-220370-46	"I'm going to check whether the improvement involves our volunteers", because of course"
subtitle-220370-224690-47	"someone else might have improved the article and that's great, but it wouldn't be impact"
subtitle-224690-225740-48	"of my program."
subtitle-226460-230600-49	"So this is an example of something very specific that I'm going to do in order to"
subtitle-230600-237470-50	"attempt to show that these editing workshops actually improved the quality of the articles"
subtitle-237470-239210-51	"that the museum cares about."
subtitle-239510-245360-52	"So you can see it's a very relevant metric and I am planning, I'm putting it in my plan,"
subtitle-245360-252920-53	"how I'm going to measure it, how I'm going to try to show whether and how much I am"
subtitle-252920-259670-54	"achieving this goal of improving quality of material that the museum cares about."
subtitle-260650-262720-55	"I hope this was a clear example."
subtitle-263110-267100-56	"Again, the evaluation plan doesn't have to be much more complicated than this."
subtitle-267100-269280-57	"I really just simplified it with..."
subtitle-269280-271810-58	"I left out dates and and the details of the survey."
subtitle-272230-276790-59	"But something like this, a page, a page and a half, can be an evaluation plan."
subtitle-276790-280030-60	"It doesn't have to be a 30 page bureaucratic nightmare."