00:00:03.580 --> 00:00:05.839 Liam Wyatt: All right. Hello, everyone.

2 00:00:05.890 --> 00:00:09.040 Liam Wyatt: My name is Liam Wyatt. I will be

3 00:00:09.100 --> 00:00:15.159 Liam Wyatt: hosting this session. This is the Wikimedia Foundation. Future audiences. Discussion

4 00:00:15.440 --> 00:00:17.430 Liam Wyatt: approximately held monthly

5 00:00:17.660 --> 00:00:24.930 Liam Wyatt: and we'll be uploaded to Wikimedia Commons very soon after the call. Today is the fifteenth of February, 2,024,

6 00:00:25.160 --> 00:00:29.069 Liam Wyatt: and we have an agenda on the on the screen here which I'll read out.

7 00:00:30.370 --> 00:00:36.659 Liam Wyatt: Welcome what we're doing here, what is future audiences and what is happening past and future.

8 00:00:36.770 --> 00:00:39.669 Liam Wyatt: Then 2 major software development

9 00:00:39.680 --> 00:00:43.220 Liam Wyatt: reports, one about Chat Gpt Plugin.

10 00:00:43.440 --> 00:00:49.310 Liam Wyatt: It's a conclusion. And what happens next? Hint? It's called citation needed.

11 00:00:49.580 --> 00:00:53.840 Liam Wyatt: and we'll be hearing from that from Mike. So Mariana and Mike

12 00:00:54.100 --> 00:01:08.839 Liam Wyatt: followed by open QA. We. If there are any questions between each of these sections, please ask them then, because that is more relevant at the time. And we can share the video and the associated questions directly

13 00:01:08.850 --> 00:01:13.270 Liam Wyatt: to Meta, to Wikimedia community social media channels, etc.

14 00:01:13.600 --> 00:01:22.329 Liam Wyatt: So with no further ado Mariana, would you like to kick us off with what is future audiences, past, present, in the future.

15 00:01:22.380 --> 00:01:34.209 Maryana Pinchuk: Yes, I would love to thank you, Liam. Hello! For those of you who don't know me. Marianna Pinchuk, product manager at the Wikimedia Foundation. I lead the future audiences. Initiative. And I will be taking you through

16 00:01:34.240 --> 00:01:38.330 Maryana Pinchuk: what might be a reminder for most of you, but might be new for some of you.

17 00:01:38.370 --> 00:01:57.509 Maryana Pinchuk: What is the future? Audiences? Initiative? Why are we here? So just as a reminder. This is a new initiative that just kicked off in this fiscal year the Wikimedia Foundation fiscal year starts in July. So we've been operating as a team since July.

18 00:01:57.520 --> 00:02:20.669 Maryana Pinchuk: Not that much time. Actually, but the mandate of this team is as follows, we are trying to understand and identify strategies to pursue to arrive in a changing Internet and test our assumptions about those strategies with quick experiments.

19 00:02:20.670 --> 00:02:37.660 Maryana Pinchuk: So that is the set of objectives that we have but what does that actually mean? So what is a strategy? Well, a strategy that our movement has basically benefited from for the last 20 years or so

20 00:02:37.660 --> 00:02:50.959 Maryana Pinchuk: is a a interesting symbiosis with the way that technology has worked for the last 20 years. And the way that many people have used technology to find information

21 00:02:50.970 --> 00:03:17.420 Maryana Pinchuk: and what I mean by that is for the last 20 years there have been a large group of people globally, who, when interested in finding out a fact or learning more about a topic, would open up a web browser and go to a search engine, usually Google and would type in some question and would get a set of results and often the top result, or in the top results would be Wikipedia

22 00:03:17.550 --> 00:03:29.080 Maryana Pinchuk: and that is how Wikipedia has thrived in in readership and in building communities in various languages for the last 2 decades.

23 00:03:29.080 --> 00:03:50.269 Maryana Pinchuk: But that is a a very specific set of behaviors. That is changing. We know that. You know the world looks different now than it did in 2,001. There are a lot more places online to go and get knowledge. There are a lot more ways online to go and get knowledge. You can listen to a podcast you can watch a short video.

24 00:03:50.270 --> 00:04:04.829 Maryana Pinchuk: You can do all kinds of things. Now. To get information. Go to all kinds of places. And we know with generative AI. The whole concept of of getting information. Knowledge is in in flux.

25 00:04:04.830 --> 00:04:31.590 Maryana Pinchuk: So what we're trying to do as a team really is to look ahead and see what other strategies we could pursue as a movement to continue to be the incredible knowledge service to the world that we've been the last 20 years and to engage new audiences who we actually haven't been engaging over the last 20 years. Not everyone in the world knows about Wikipedia or uses Wikipedia.

26 00:04:31.670 --> 00:04:37.539 Maryana Pinchuk: So how might we see this change in technology as an opportunity. So

27 00:04:37.780 --> 00:05:05.809 Maryana Pinchuk: at a high level, II think 2 ways you could look at what a different strategy might look like. And these are kind of the areas we're poking around with experiments in to better understand both the opportunities and the the risks, the costs. Of each. These 2 areas are growing as a knowledge destination. Which just means that instead of relying on

28 00:05:05.840 --> 00:05:31.160 Maryana Pinchuk: a search engine or some other platform which, or technology to to bring people to our platform, to our projects. We could instead envision a world in which anytime anyone thought of anything they wanted to know more about, they would go directly to Wikipedia. That might bring, when many, many more readers to our projects. And potentially more folks who can contribute directly

29 00:05:31.350 --> 00:05:48.570 Maryana Pinchuk: that comes with a whole set of assumptions and risks and costs that we're that we're trying to understand experimentally. Another approach might be to instead look outward at all of the places. Now that people are getting information.

30 00:05:48.570 --> 00:06:12.569 Maryana Pinchuk: Whether that's video platforms like Youtube or Tiktok, whether that's third party chat bots like Chat Gbt, anywhere in the world that people are getting information and looking to see, could there be an opportunity for us to present our information more clearly, with attribution, with pathways to contribution. In all of those places

31 00:06:12.800 --> 00:06:18.359 Maryana Pinchuk: and I see a hand already. So I'm gonna stop and field any questions that people have.

32 00:06:21.150 --> 00:06:22.200 Liam Wyatt: Anton.

33 00:06:23.590 --> 00:06:24.570 Hello.

34 00:06:25.320 --> 00:06:28.419 Антон Обожин: May speak.

35 00:06:29.140 --> 00:06:31.310 Антон Обожин: Yes, we can hear you just go

36 00:06:33.720 --> 00:06:42.250 Антон Обожин: about that issue to become a source of knowledge

37 00:06:43.580 --> 00:06:47.109 Антон Обожин: itself instead of being surge driven, I can give an example

38 00:06:47.140 --> 00:07:13.019 Антон Обожин: which is relevant to our local community. And we have raised this question is an example that if you are using Ukrainian set to operational system windows. If, your browser is set to language, and your preferences in Google are set to bring in languages. If you Google, something new cranium languages, if

39 00:07:13.140 --> 00:07:21.839 Антон Обожин: gives you a results in Russian Wikipedia, first, then in English, and only then in Ukrainian. So

40 00:07:22.980 --> 00:07:24.110 Maryana Pinchuk: hmm

41 00:07:24.380 --> 00:07:50.059 Антон Обожин: have ways to manipulate what they are showing manipulate which content from Vicky, which is not enter, in which priority so deprive them. Of this possibility can be shown on the Wikipedia. And what can be?

42 00:07:50.080 --> 00:08:08.970 Антон Обожин: There may be an popular article with 100,000 views per day which would be easily given in a search engine like Google. And there may be an article with one view per day.

43 00:08:08.990 --> 00:08:24.040 Антон Обожин: and it will be completely ignored by search engine. And it may be a wonderful article. But no one ever sees it because Google never shows it in its search results. That is a problem.

44 00:08:25.160 --> 00:08:34.919 Maryana Pinchuk: Yeah, absolutely. Yeah. I think one of the one of the things that we're doing is trying to not just understand what are the good things about

45 00:08:34.919 --> 00:08:55.839 Maryana Pinchuk: each of these different directions, but what are potentially the risks and the costs. And you're certainly right. That, relying on other platforms to syndicate our content, which we have for the last 20 years again. Really does come with that risk that the different platforms, choices of what? Content to show

46 00:08:55.860 --> 00:09:16.320 Maryana Pinchuk: potentially their political affiliation, whatever might influence, what kind of information is is available to to readers. And that's a big danger. It's been a big danger with Google, and it will continue to be a big danger as long as we allow other platforms to be our main mode of syndication. Absolutely

47 00:09:20.340 --> 00:09:21.930 Maryana Pinchuk: any other questions

48 00:09:22.270 --> 00:09:37.049 Maryana Pinchuk: or thoughts about. And yes, Sandy, thank you for linking to the the retro board. So in December we had a a chat about sort of the different opportunities and risks we see in all of the different

49 00:09:37.150 --> 00:10:03.180 Maryana Pinchuk: potential strategies that we might undertake. And I think there were some good pros and cons. Very, very significant pros and cons for for undertaking either of these strategies the destination strategy, while it does have the benefit of allowing us total control over what we showed to to readers as a as a movement. Comes with also a number of really big unknowns. Cost

50 00:10:03.180 --> 00:10:11.550 Maryana Pinchuk: whether people will change their behavior to look for knowledge in a specific place.

51 00:10:11.580 --> 00:10:12.400 Maryana Pinchuk: yeah.

52 00:10:17.510 --> 00:10:21.569 Liam Wyatt: Anton, I see your hand up again. Is that the same hand? Or is that a different question.

53 00:10:26.890 --> 00:10:28.669 Maryana Pinchuk: I think that might have been a legit.

54 00:10:29.390 --> 00:10:34.460 Liam Wyatt: Pepsi. Okay. Mariana, please. Okay. Well.

55 00:10:34.530 --> 00:11:03.319 Maryana Pinchuk: yes, I will continue. So so yes. So our team. As I said, we are a testing team and a learning team. We aren't building fully finished products. We aren't putting a big bet on either of these directions yet, because we don't know whether these are even possible to pursue. So so I really just wanna stress that everything that you're seeing and everything that we'll be talking about. I'm gonna hand it over to Mike in a second

56 00:11:03.320 --> 00:11:07.959 Maryana Pinchuk: is experimental work. It's not meant to be like.

57 00:11:07.960 --> 00:11:32.019 Maryana Pinchuk: This is the new big 1 million dollar bet from Wikimedia about how we're going to. Use AI it, we're really just trying to learn and understand the opportunities and the risks of of these different strategies and make recommendations for other teams at the Wikimedia Foundation to pursue bigger investment if we feel like it is an important and

58 00:11:32.370 --> 00:11:34.310 Maryana Pinchuk: potentially fruitful area.

59 00:11:35.120 --> 00:12:00.379 Maryana Pinchuk: Okay, so I'm gonna hand it off to Mike now to talk about our very first completed experiment which we started the year with which was building a Plugin for Chat Gp. That would pull knowledge specifically from Wikipedia and summarize it, and you'll see the screenshot here shows essentially how it looked. So these are our key research questions that we had going in

60 00:12:00.380 --> 00:12:16.300 Maryana Pinchuk: this experiment. In the future. Audiences kind of framework was trying to help us understand whether Chat Gp is becoming the new Google and if so, what are those implications for? For how our knowledge is showing up there to readers?

61 00:12:16.300 --> 00:12:31.270 Maryana Pinchuk: And how might we deal with that? And there is a lot of learning and insight. That I'm excited for you to hear about. So, Mike, please take it away. I'm gonna stop sharing my screen so you can share your slides. Yeah.

62 00:12:31.810 --> 00:12:39.089 miishen: sounds good. Thanks. Great Hi, everyone. Please bear with me oops

63 00:12:39.420 --> 00:12:44.629 miishen: I'm I had Covid recently, so I'm like I might have a brain for it once in a while.

64 00:12:45.120 --> 00:12:48.140 miishen: okay, can everyone see this?

65 00:12:48.480 --> 00:12:50.159 miishen: I'm just gonna use this mode.

66 00:12:50.310 --> 00:13:06.369 miishen: Alright. So I'm not gonna go through every single slide in detail here, just out of the interest of time. But I think I'm gonna try to do a quick overview of the things we learned the link to this. This is on our Meta Wiki page. So anyone can go look at these slides as well.

67 00:13:06.810 --> 00:13:19.980 miishen: so I won't recap too long over what the Wikipedia Plugin for Chatty Bt. Was just because I think most people on this call are somewhat familiar with it. Basically, it was a plugin for Chat Gbt, that would

68 00:13:20.120 --> 00:13:36.590 miishen: be activated if Chat Gt. Thought that the query was related to general knowledge, and something that Wikipedia could answer, and then it would go search Wikipedia, and then summarize the content that we return back to it and provide links and references and some disclaimers.

69 00:13:36.660 --> 00:13:42.350 miishen: and basically just marking that the information was coming from Wikipedia in a very conspicuous way.

70 00:13:43.180 --> 00:14:07.089 miishen: So I think overall, this is kind of the executive summary of this experiment. We wanted to know if Lms and chat assistance like Chat Gbt, were the new paradigm for information seekers. If you can imagine. A year ago, when everyone was talking about chat Gvt, this is there was this kind of impending, almost doom feeling of like, oh, everything's gonna go away as we know it, and it's going to be replaced by Chat Gbt.

71 00:14:07.140 --> 00:14:14.040 miishen: so there's a lot of urgency behind trying to learn more about whether this is going to happen, or how or what that world looks like.

72 00:14:14.050 --> 00:14:18.879 miishen: And that was kind of a huge motivator. For why we wanted to do this experiment quickly.

73 00:14:18.930 --> 00:14:36.589 miishen: and we can say that it hasn't happened. It may still happen. We don't really know. But that same level of urgency is kind of diminished a little bit. We haven't seen like Google disappear. For example, we haven't seen, you know, as we'll see in other slides like web traffic to Wikipedia go down, or anything.

74 00:14:36.720 --> 00:14:44.320 miishen: But I don't think we can dismiss it completely. The the industry is changing very quickly. People are making improvements. Things could still change in the time.

75 00:14:44.690 --> 00:15:11.909 miishen: And similarly, like, I said, our chat assistance displacing traffic to Wikipedia. The answer is, no, there's no evidence that any of the Chat. Gbt. Anything has caused traffic to Wikipedia to change in any noticeable way, mostly because there's just so much traffic to our website that, like it's, you know, every chat Gbt user was like not using Wikipedia. I don't even know that would make it dead honestly. So that's kind of a good thing in a way.

76 00:15:12.370 --> 00:15:17.460 miishen: and then we were also interested in these Lm chat assistance. We're reaching new audiences.

77 00:15:17.480 --> 00:15:29.219 miishen: And we don't have complete data. But in general, from what we've seen is that it's not really reaching new people. It seems to be reaching a lot of the same people that currently use Wikipedia.

78 00:15:29.890 --> 00:15:39.090 miishen: But we would also say, a big part of this is Chatty, Pbt. To get to the Plugin and install it. You have to pay for it, anyway, so that Paywall is going to introduce a lot of barriers to

79 00:15:39.310 --> 00:15:53.070 miishen: new audiences. Who, you know, may live in person or over. They don't have enough money, basically And then, on the technological side, we're interested in how accurate the Lms were in retrieving and summarizing answers.

80 00:15:53.150 --> 00:15:55.040 miishen: And actually, they were pretty good.

81 00:15:55.060 --> 00:16:20.630 miishen: They're not perfect. They make mistakes, they do weird things. And these mistakes are constantly changing as they update the models. But I think overall from our internal auditing we saw that the results are pretty good most of the time, and they're probably only going to get better. We assume and then, I think, equally importantly, is beside the actual like performance of it. What is perception of that credibility like? What did people think about the accuracy and the relevancy.

82 00:16:20.700 --> 00:16:34.619 miishen: And in general we saw that people generally trust the information more when they know it's from Wikipedia, and so there's some strength there in the brand of Wikipedia as a source of reliable information. And that's something that we're kind of interested in exploring further as well.

83 00:16:34.750 --> 00:16:45.060 miishen: But okay, so just real quick. We will go through some data. Once again feel free to ask questions. I won't spend too much time on any individual thing.

84 00:16:45.260 --> 00:16:58.760 miishen: The I just labeled the boxes or the data in different colored boxes based on where it's coming from. Yellow is from logging data. We did a survey with about 71 participants and self reported, so that's green boxes.

85 00:16:59.810 --> 00:17:23.979 miishen: Okay. So we look at the users and the messages. We ran it for about 2 quarters. So about, let's say, 6 months, and we saw that after about a month into August we got some publicity. And so we got this kind of spike in users. Everybody suddenly knew about it. People were downloading it and solving it, and we see this kind of rise up in unique users and message count kind of more or less follows that trend

86 00:17:24.060 --> 00:17:30.589 miishen: And then, sometime around October, we saw the usage start to go down again.

87 00:17:30.750 --> 00:17:38.049 miishen: and that doesn't line up exactly with Chat Gbt. Moving away from plugins to Gpts. But I think it's pretty close

88 00:17:38.230 --> 00:17:40.350 miishen: and

89 00:17:40.450 --> 00:17:51.219 miishen: and then we ended the experiment in January. So just last month so we kind of see this rise, and we kind of see the a fall, and it seems kind of like top-out kind of around October.

90 00:17:53.590 --> 00:18:03.970 miishen: We looked at logging data and self reported data. It's kind of close, not exactly the same, but we would say about 2 thirds of the users were located in Europe and North America.

91 00:18:04.400 --> 00:18:08.319 And then, if we look at the top languages by number of messages.

92 00:18:08.380 --> 00:18:17.970 miishen: then English is always on top, and then the rest of, like Chinese, German, French, and Japanese would kind of like trade places for like week by week, which one is the most

93 00:18:17.980 --> 00:18:21.189 miishen: the most number of messages that week.

94 00:18:21.260 --> 00:18:45.639 miishen: But English is generally always on top. So this is part of what we were seeing. Where, like in terms of new audiences, you know. Are we only reaching North America and Europe in general, or are we reaching Africa Asian audiences quite as to the degree that we'd want to if we were hoping this is like a new means for other people that access Wikipedia information. And it looks like for the most part, mostly

95 00:18:45.650 --> 00:18:57.759 miishen: it's still North American Europe. There's some potential here and there. But it's very much mitigated by what open AI decides to do right like whether they have a pay wall where chatty pts, even accessible, etcetera.

96 00:18:59.620 --> 00:19:06.010 miishen: When we looked at self-reported gender and age, we saw that most people are from 26 to 45.

97 00:19:06.050 --> 00:19:10.610 miishen: Let's call that middle is aged people.

98 00:19:11.150 --> 00:19:22.749 miishen: so we don't see. II think we had one person reporting that was under 18. But II can't remember why we didn't use that, or maybe we don't report as much on under 18. But this was like

99 00:19:23.160 --> 00:19:31.710 miishen: it. It wasn't like we we threw out like 20 of self reports because they're all under 18. We just basically didn't see a lot of young people using the the plugin

100 00:19:32.020 --> 00:19:37.309 miishen: and then of the self reports we had predominantly, mostly men

101 00:19:37.560 --> 00:19:42.040 miishen: reporting usage of the Chachi team. Chat. Gt. Plugin.

102 00:19:43.680 --> 00:19:56.950 miishen: This one is just kind of interesting. I don't know what major conclusions I have from this. But we asked people what their primary purpose for using Wikipedia was, and most people use it for personal interest topics, some academic research.

103 00:19:57.200 --> 00:20:11.709 miishen: And you know there's a fair amount that said just trying out chat. Gbt, I suspect that's also why we see kind of them a decline after the peak, because, like people who are just like trying this cool new piece of technology and seeing how things work and trying out the plugin, seeing how that works.

104 00:20:12.090 --> 00:20:23.219 miishen: you know, if they did that for a few months, and then they're kind of like, okay, that was a cool little toy, like, you know, I'm gonna go back to the website, or something that we might expect to the drop off that we saw in unique users.

105 00:20:23.500 --> 00:20:29.110 miishen: And then, almost as expected as well, breaking news. Not a lot of people are looking at.

106 00:20:29.470 --> 00:20:33.749 miishen: I think, knowing that Chat Gp has a training window

107 00:20:33.800 --> 00:20:37.910 and then, knowing that Wikipedia is maybe always updated.

108 00:20:37.990 --> 00:20:50.930 miishen: you know, with the most breaking news to the very minute or something, and I think people were in patterns of like checking Twitter or checking like social media. I wouldn't expect breaking news to be like the highest use case.

109 00:20:52.090 --> 00:20:58.069 miishen: I see that there's some chats coming in, but I don't see the chats themselves. So if anybody wants to.

110 00:20:59.750 --> 00:21:05.900 miishen: that's okay, Mike, we'll tell you if there's something to respond to. That's what I was asking for.

111 00:21:06.460 --> 00:21:33.190 miishen: Okay, this is also interesting. So we asked people if they used to plug in more than the Wikipedia website. And by and large, we see that most people are still going to the website, probably at the same rate or more. Some people used to plug in. They said much more, but I think that's kind of minority. So this is part of the story of like, it's not like chatty Bt. Users, or like chat gpts like taking away like our lunch. Right? It's not like people are like abandoning Wikipedia to go find information on Chat Gbt.

112 00:21:33.440 --> 00:21:35.730 miishen: they're still good using the website.

113 00:21:37.190 --> 00:21:41.040 miishen: This is the click through rate.

114 00:21:41.350 --> 00:21:52.729 miishen: So, as we saw that each time chat our plugin would answer with it would provide a link if possible. Sometimes it's a little bad about it, but it would provide a link back to Wikipedia for people to click through and get more contacts.

115 00:21:52.870 --> 00:22:02.329 miishen: And we can see that it kind of starts up the average. Sorry. The click-through rate was similar to what we see on from Google, searches.

116 00:22:02.340 --> 00:22:04.920 miishen: And and then

117 00:22:05.430 --> 00:22:14.039 miishen: and then we have this queries per day, which is starts up kind of high and decline. So let's say, about 4 queries per day. So this is, this is kind of the

118 00:22:14.160 --> 00:22:35.290 miishen: the one that's a little bit less clear just by looking at what's on the slide. The story that I think that's coming out from this that we are seeing is that people are really using the plugin like search. They're using it as like a spe. A better natural language search to get information out of Wikipedia. And I think the 2 things that are kind of interesting here click through rate. It's similar to

119 00:22:35.500 --> 00:22:40.860 miishen: Google Search, which, of course, is kind of a not Apple's apples. But it's saying, like, you know.

120 00:22:40.910 --> 00:22:43.610 miishen: the numbers are the same. But, more importantly.

121 00:22:44.040 --> 00:23:13.710 miishen: they're not using people weren't really using it conversationally is what I'm kind of seeing here. Because if you can imagine, like the whole appeal of Chat Gbts, you have this conversational tool. So I ask it a question. I get an answer. If I want more context, or I wanna dig in. The the proposal is that I can always just ask more questions and follow up and have a conversation with this agent and get more information. If that's the case, I would expect click, click through rate to be quite low. Right? Because

122 00:23:14.220 --> 00:23:37.159 miishen: why would I click through and read the article? If I can just keep querying the system to get more information out of it. But if the quick click through rate is similar to Google than what it's telling me is that people are using the search, they get some information. And when they want more context, they're going to the website just as they would be on search rather than continuously query the system. And that's also why I'm including click our queries per day as well. Because

123 00:23:37.990 --> 00:23:43.360 miishen: you can imagine if you constantly are re querying the system, get more information you might see higher

124 00:23:44.920 --> 00:23:52.609 miishen: once again to get that from clicking. I can always send another message to get more information.

125 00:23:52.640 --> 00:24:08.520 miishen: So this isn't like, you know, the home run smoking gun. But I would say, when you kind of like, look at all these pieces, it does kind of look like people are using in the same paradigm which is search, and not this kind of new paradigm of like. Let me keep asking this robot questions over and over to get more information.

126 00:24:08.770 --> 00:24:13.120 Liam Wyatt: Mike, we have a question and a comment in the chat.

127 00:24:13.350 --> 00:24:16.080 Liam Wyatt: The the question is effectively.

128 00:24:16.240 --> 00:24:19.349 Liam Wyatt: why can't I still use it? Wouldn't it be

129 00:24:19.580 --> 00:24:24.490 Liam Wyatt: still interested to have it available. Anyway, I might be interested in

130 00:24:24.550 --> 00:24:28.759 Liam Wyatt: using it in the future so effectively. What's the harm of keeping it around

131 00:24:29.070 --> 00:24:36.999 Liam Wyatt: and the comment is that, excuse me. if we mediate on the who and how many

132 00:24:37.150 --> 00:24:47.919 Liam Wyatt: people? It's actually quite a small number of people compared to say Google with users and thinking about those people that they are mostly social media or creators.

133 00:24:47.960 --> 00:24:50.030 Liam Wyatt: anyway, which is even a smaller

134 00:24:50.080 --> 00:25:00.550 Liam Wyatt: percentage. So the implication of this comment is that it's quite a fraction of the users and their paid users, and so forth. Quite a minority of of

135 00:25:00.990 --> 00:25:04.759 Liam Wyatt: potential audience, either at the best of times. If I understand that

136 00:25:05.280 --> 00:25:07.050 Liam Wyatt: the scope of the comment.

137 00:25:08.610 --> 00:25:14.209 miishen: thanks. I'll try to address maybe both of them in a similar way, which is as Mariana said.

138 00:25:14.350 --> 00:25:27.420 miishen: the goal of this team is to run quick experiments and to learn as much as we can in a quick turnaround and that's going to mean that the data that we collect and can generalize from is going to be limited.

139 00:25:27.530 --> 00:25:29.369 miishen: And it's not complete.

140 00:25:29.460 --> 00:25:46.909 miishen: And you know, I have a lot of tension with our data scientists all the time, because they like to be very meticulous, since they like. Do we have enough data to support this. And my answer is typically going to be like, No, we don't. And II recognize that. But we're doing a little bit of reading the tea leaves here, because.

141 00:25:46.910 --> 00:26:07.040 miishen: you know, we ran an experiment in 2 quarters, which is actually quite good, and we have a lot of good information and things that are kind of pointing us in certain directions. And there's always going to be follow ups. And we can. We have better ideas of where we might want to follow up now, or whether it's worth it. But we're never gonna be able to conclusively say that this is like perfect information. So

142 00:26:07.050 --> 00:26:08.109 miishen: you know, like

143 00:26:08.590 --> 00:26:13.519 miishen: this is the best as we can know in the time that we're gonna run it. And

144 00:26:13.600 --> 00:26:18.059 miishen: this the speed at which we want to run these experiments is kind of the main priority here to get

145 00:26:18.120 --> 00:26:35.229 miishen: some indication. You know, we can get that 70% confidence level that, like, you know, we shouldn't invest like 50 million dollars into Chat Gbt, for example. Then that's like a pretty important thing for us to know as an organization rather than spending like 3 years getting like 85% confidence. That's like

146 00:26:35.460 --> 00:26:43.760 miishen: that's not going to help us in a world where things are moving very quickly. Technology and social kind of implications of that technology are moving very quickly.

147 00:26:43.860 --> 00:26:56.910 miishen: So I think that addresses kind of the second question of like, you know, limited sample size, etc. The first question of why we don't keep this running is is similar in a way, because we want to run these quick experiments, spend a lot of time

148 00:26:56.930 --> 00:27:00.370 miishen: diving in learning something, and then kind of

149 00:27:00.450 --> 00:27:02.409 moving on to the next thing, and

150 00:27:02.530 --> 00:27:21.270 miishen: there is always going to be a cost of keeping these things running right like our, we have to maintain the plugin. We have to. I mean even just to keep it running to improve it, even if we don't want to improve it. Chat Gbt changes their system. Now, everything is a Gbt, so that's some work to change the Plugin infrastructure to be a Gpt infrastructure and constantly having to follow that

151 00:27:21.630 --> 00:27:25.600 our data team has to be able to keep collecting data on it potentially

152 00:27:25.770 --> 00:27:35.780 miishen: because if we're not collecting data, then there's almost like no point in running. Continuing the experiment. The whole point of the experiment is, we're collecting data so that we can analyze and learn something from it.

153 00:27:35.830 --> 00:27:45.419 miishen: And I think this is kind of a key distinction from what Mariana was saying about like experiments versus products, right products is like, we're really trying to

154 00:27:45.460 --> 00:28:04.350 miishen: create something for users, improve it over time and have this product that is useful for people. And in an experiment. It's like we need people to use these things so that we can learn something from it. But the the ultimate point is to get these learnings. And it's not to put a product in the world that it's actually in its final form, useful.

155 00:28:04.440 --> 00:28:13.490 miishen: Which is not to say that this won't come back right like, if this was like a knockout we saw like amazing numbers, and that's a chance for us to go to leadership and say

156 00:28:13.570 --> 00:28:29.270 miishen: our recommendation is like people really love this Plugin. We should go build this. We should go find a team that actually can maintain this. We should actually go find the budget. For this. We should actually use the software that we want to use internally, that aligns with our values. That's open source. All of these things that we would normally consider for internal

157 00:28:30.160 --> 00:28:42.230 miishen: product development is that would be like starting down that path. And we can make a recommendation for that, but for us the experiment is the priority which is like, what can we learn from this? And at the point where, like.

158 00:28:42.500 --> 00:28:52.399 miishen: you know, the juice is not worth the squeeze, so to speak. Then we kind of know that this version, the the experiment it is like it's time for it to end. So I hope that it addresses the questions

159 00:28:54.220 --> 00:28:55.030 next bucket.

160 00:28:56.140 --> 00:29:03.979 Liam Wyatt: We have. We're past the half now. So we should get to the what's next. But I wanna make sure you have

161 00:29:04.700 --> 00:29:06.860 Liam Wyatt: this concluded

162 00:29:06.910 --> 00:29:12.809 Liam Wyatt: adequately. And people have an adequate time to respond to any immediate questions about Chat Gp, specifically.

163 00:29:13.400 --> 00:29:17.459 miishen: okay, cool. I'll try to fly through these slides a little bit.

164 00:29:17.690 --> 00:29:20.740 miishen: and then leave some time for questions, and we'll talk about citation needed.

165 00:29:20.780 --> 00:29:37.120 miishen: This is just Google trends, showing that the red line is searches for Chat. Gbt. It can. It's spiked, as everyone you know can see. But Wikipedia and Wiki searches. Just kind of stay level. So we didn't really see any once again, not really taking our pie taking our lunch.

166 00:29:37.180 --> 00:29:53.450 miishen: This is kind of similar. This is just the page view issue. Or this is, what is this page views you have per month, and effectively, if it looks like chaos, and it doesn't look like there's a pattern that's kind of the point, which is that chatty Pt. Launched, and it didn't really

167 00:29:53.580 --> 00:29:56.770 miishen: seem to move us in any significant way in terms of page views.

168 00:29:57.860 --> 00:30:06.889 miishen: So this is just returning to main questions. Lms are not like this new paradigm that we can see yet they're not displacing traffic to Wikipedia and

169 00:30:07.030 --> 00:30:09.679 miishen: in general, we're not really reaching new audiences.

170 00:30:09.990 --> 00:30:20.420 miishen: This is model quality. We did some internal annotation. The the punchline here is relevancy. We did pretty good. This is just whether or not the answer was relevant, whether we thought it was accurate or not.

171 00:30:20.590 --> 00:30:36.889 miishen: And it's it's pretty relevant most of the time. Accuracy is how accurate the information was to the Wikipedia source, also pretty accurate most of the time. The caveat in both of these is that there's some variation by language. But once again, there's not a lot of data. Sometimes there's only like a dozen

172 00:30:37.410 --> 00:30:42.329 miishen: you know, examples that we were able to annotate. And take a look at so

173 00:30:42.370 --> 00:30:46.279 some of these may look worse. It's kind of hard to tell in some cases.

174 00:30:46.290 --> 00:30:50.100 miishen: but overall. When we look at it overall, it looks like pretty okay.

175 00:30:51.050 --> 00:30:59.170 miishen: So in general, the the models are pretty accurate. They're pretty relevant. They could be better, but they're probably improving over time, and it's something we should keep an eye on.

176 00:30:59.330 --> 00:31:04.789 miishen: And once again, people generally trust the information more when they know it's coming from Wikipedia, which is pretty important.

177 00:31:06.410 --> 00:31:10.340 miishen: Okay, so I can kind of skip this part. This is looking forward.

178 00:31:10.430 --> 00:31:19.199 miishen: I will kind of stop it here for now and see if anybody has questions about the there is one more.

179 00:31:19.280 --> 00:31:36.999 Liam Wyatt: Okay. How confident are we about extrapolating from the Wikipedia chat Tpt user base as opposed to the general chat. Tpp user base, which is orders of magnitude larger especially considering that's a self selected sample upon a self selected sample.

180 00:31:38.120 --> 00:31:50.919 miishen: II think it depends on what we're trying to extrapolate right? Like here. We were really, really trying to learn who is using Chat Gbt, and you know what their purposes are and what their intentions are.

181 00:31:50.990 --> 00:31:57.899 miishen: So of course, there's going to be limitations about how much we can know about the people who don't use chat. Gbt.

182 00:31:58.050 --> 00:32:01.850 miishen: But in general I

183 00:32:02.080 --> 00:32:05.580 miishen: I don't see too much reason to not

184 00:32:06.840 --> 00:32:25.539 miishen: to be too like overly conservative and say, this doesn't apply at all to the general population. I would say that this should carry over to some comparable degree and this would be kind of like a good chance to do. Follow up research if we wanted to, you know, get a better idea. But I don't see any immediate reason why, you know this.

185 00:32:25.750 --> 00:32:28.430 miishen: II think we can kind of like make

186 00:32:28.540 --> 00:32:43.729 miishen: corrections in some cases, like, Okay, well, the self reported people that are using self or Chat Gbt are like mostly men, for example. So I can imagine the wider audience I would correct and say, there's probably more women using the website than like the chatty chat Gbt users.

187 00:32:43.760 --> 00:32:53.109 miishen: So I think in some cases you can treat this. What we're seeing here is either like the high end of the low end depending on you know how you think the data might skew towards self selection.

188 00:32:54.570 --> 00:32:56.569 and we had a further question,

189 00:32:57.320 --> 00:33:03.089 Liam Wyatt: that users are more interested or engaged in conversational format than reading long form articles

190 00:33:03.360 --> 00:33:10.959 Liam Wyatt: providing helpful and relevant information depends a lot on the question they're asking or you asking the right question?

191 00:33:11.180 --> 00:33:19.789 Liam Wyatt: Has there been thought, in case the previous queries that relate to the current one, and immediately suggesting for the information based on the most probable next question.

192 00:33:22.570 --> 00:33:45.260 miishen: The short answer is no, and I think that we started digging into the queries themselves. We didn't do too much with them, but we don't. I didn't show it here. But part of what the story that we're saying, that people are using it like search is that I would say at least half the the queries that we're seeing. I actually aren't even natural language questions. People are using it like search. They would just be like Istanbul Turkey.

193 00:33:45.710 --> 00:33:58.110 miishen: you know, like, that's something you might type into a Google search bar and then hope to like sort through the results. And you know the appeal of Chat Gt, is that you can actually ask a more specific question. You know, like, you know, what is

194 00:33:58.130 --> 00:34:05.150 miishen: that? That feral cat population of stable or something right? And and get the answer that you would want hopefully.

195 00:34:05.160 --> 00:34:10.890 miishen: And so we saw a mix of like all of these across the board, and I kind of bring it up, because

196 00:34:11.020 --> 00:34:31.469 miishen: that huge variation, I think, makes it a little bit more difficult to do. Some follow up behavior based on what the queries are going to be, because, especially for a search query, where you're just getting like a name. It's kind of really hard to know what to do with that afterwards, because the intention is a little bit hidden from the system itself. It's like in the users head

197 00:34:31.469 --> 00:34:49.429 miishen: that they're gonna follow up, and they know what they're looking for. But they're not really giving that information to us. So I think, as the trend of how people use these technologies changes, and maybe people are better trained to ask questions and follow ups. And it might be possible to kind of walk down that road. But it's not something that we explored in this stage.

198 00:34:50.310 --> 00:34:53.520 Liam Wyatt: Thanks, Mike. that's all the questions that have been submitted.

199 00:34:53.590 --> 00:34:55.810 People can continue obviously

200 00:34:55.840 --> 00:34:59.309 Liam Wyatt: asking questions in the chat about this section. But if you could

201 00:34:59.680 --> 00:35:02.460 Liam Wyatt: now revealed the what next?

202 00:35:02.590 --> 00:35:08.249 Liam Wyatt: Because there is one comment, that is, I think, leading basically to what you're talking about no.

203 00:35:08.560 --> 00:35:19.660 miishen: Okay, cool. I'll try to be pretty brief about this, too. I'll just show it first, and we can kind of talk a little bit about it. I think it's just easier to see it. So citation needed.

204 00:35:20.340 --> 00:35:24.630 miishen: which we are jokingly say, don't call it a fact checker. It's not a fact, checker.

205 00:35:24.920 --> 00:35:37.769 miishen: But II think there's something in this spirit that, like we kind of all recognize, which is that there's a lot of misinformation online. And there's a lot of claims online, dubious, not and

206 00:35:38.010 --> 00:36:02.579 miishen: a lot of it with AI as being, you know, AI is amplifying how much of this there could be. And so we're positioning this as an opportunity for allowing people to use Wikipedia as a way of verifying information. If it exists on Wikipedia, and seeing whether or not this these claims are supported or not, supported by Wikipedia, and, you know, kind of facilitating their journey towards like verifying information.

207 00:36:02.620 --> 00:36:06.860 miishen: you know, even teaching them, maybe problem, like information hygiene online.

208 00:36:06.980 --> 00:36:16.880 miishen: and so it looks very different behind the scenes. It's actually very similar to what we built before. We're still going through Chat Gbt just through through the Api

209 00:36:16.990 --> 00:36:20.019 miishen: and

210 00:36:20.100 --> 00:36:32.799 miishen: This is kind of like part of my belief that Chat Gbts like this great sandbox to see what these Lms can do and play with them. But ultimately we're gonna see a lot of this functionality exists behind the scenes. It's not like

211 00:36:32.800 --> 00:36:51.760 miishen: the chat interface or the chat assistant is really going to be the way we all you see technologies and interact with it. So this is kind of a more focused way of saying, like, you know, how can we actually implement these technologies in a way that's used for people that can bring information to where they are that can, like set us up as a you know, a source of

212 00:36:52.370 --> 00:37:04.690 miishen: credible, reliable information. So as you can see here. The way this is going to take shape is the experiment is going to be a browser extension for chrome, just because that's the easiest way for us to

213 00:37:05.030 --> 00:37:27.539 miishen: get in front of everybody's browsing experience when they're off the website. Because crucially, Mariana, touch on this like, how do we bring Wikipedia, content to people who are off the website and don't depend on you know you have to come to Wikipedia to get any content, you know, like, how can we bring it outwards, people? So as in the Browser extension, you know, it sits in the browser. You're browsing the Internet. You see this article on Taylor Swift.

214 00:37:27.540 --> 00:37:40.370 miishen: and you will select this claim about like this earthquake essentially, and then you will be able to search Wikipedia for information. It will return the relevant quote from the relevant article, if you can find one

215 00:37:40.530 --> 00:37:52.980 miishen: and then you can see that it returns some what's it called quality signals here. So, like the last edit date the number of references number of people who worked on this article, and then you can click through and continue reading more about this. If you'd like

216 00:37:53.130 --> 00:38:01.260 miishen: there is going to be another version we'll play with here. So this is just kind of like, search, right? This is like, we kind of give you a search. We give you a snippet that looks

217 00:38:01.700 --> 00:38:11.420 miishen: contextually relevant. And then we want to experiment with using a little bit more. AI, you know, firepower, so to speak, and having inference that says like.

218 00:38:11.660 --> 00:38:16.159 miishen: it won't be like as obvious as a green check mark or something, but it will be some kind of

219 00:38:16.210 --> 00:38:23.660 miishen: indicator, like a prime like, Hey, watch out like, maybe this is like not great information. You should like, maybe go read more about this, for example.

220 00:38:23.840 --> 00:38:31.470 miishen: or maybe we couldn't find anything on Wikipedia about this. You might want to like, go look someone else just to to fully verify this good thing.

221 00:38:31.610 --> 00:38:49.100 miishen: so II think the concept is pretty simple. There's a lot more we can kind of say about it, but I think people have a lot of questions. I can see the number in the chat, so I'll kind of open it up to the QA. Here. Well, first question in the chat here is, how does this work in different languages?

222 00:38:49.730 --> 00:39:00.600 miishen: The short answer is, it doesn't. Once again due to the scope of the experiment. We are restricting it just to English, for right now and I think that is just

223 00:39:00.750 --> 00:39:06.029 miishen: so that we can handle it. We don't have very many people working on this. We know that these models work best on English.

224 00:39:06.060 --> 00:39:14.360 miishen: and so we are restricting it to English for now and in the future. Once again, if this is like a huge smashing success, then we can

225 00:39:14.420 --> 00:39:21.839 miishen: do follow up studies or recommended to product development to do a more comprehensive kind of multilingual approach.

226 00:39:25.630 --> 00:39:37.579 Liam Wyatt: That's the only question. Explicit question in the chat. So far with regards to the to the citation needed tool. That might be because it's blown everyone's mind so much that they can.

227 00:39:37.920 --> 00:39:53.819 Maryana Pinchuk: Well, whether or not you have questions I'm just curious to get some like verbal reactions, either in chat or like. What do you think? If you think this is a terrible idea, please please speak up. We want to hear this, too. We are here for your transparent feedback.

228 00:40:02.420 --> 00:40:07.030 Maryana Pinchuk: Thank you, Andrew. All caps. I like it.

229 00:40:07.140 --> 00:40:13.219 Liam Wyatt: Could you say more about why you like it, Andrew, please, please talk.

230 00:40:13.610 --> 00:40:17.150 Liam Wyatt: You should be able to unmute yourself if if you can't.

231 00:40:18.010 --> 00:40:20.229 Andrew Lih: yeah. Can you hear me? Okay, yes, you can.

232 00:40:21.540 --> 00:40:34.689 Andrew Lih: Hey? There! Wow! I'm tilted II think any way of bridging the day to day whether you wanna call pedestrian surfing and the the flow of someone's day

233 00:40:34.720 --> 00:40:39.409 Andrew Lih: with Wikipedia in a virtuous way, is a big plus. So

234 00:40:39.440 --> 00:40:43.320 Andrew Lih: that's what II like seeing about stuff like this.

235 00:40:43.370 --> 00:40:45.259 Andrew Lih: and not only just for our core

236 00:40:45.600 --> 00:40:58.369 Andrew Lih: set of editors who tend to have this already in their mindset of like. you know, as I'm browsing the web like, oh, I wonder if that's Wikipedia? Oh, I wonder what I can improve in Wikipedia? Oh, that's already part of our DNA in many ways. But anyway, to

237 00:40:58.480 --> 00:41:00.740 Andrew Lih: you know. tap.

238 00:41:00.970 --> 00:41:14.459 Andrew Lih: Folks who have this kind of a vibe, but have never have been always puzzled by the interface and the difficulty of adding it. Any tools like this, that bridge that, and make it much lower hanging fruit I'm all for, so it's great to see

239 00:41:16.370 --> 00:41:23.989 Liam Wyatt: something. I, Anthony, have a hand hand up from Anton in the chat. Would you like to speak, or would you like to type? And I will be to that for you?

240 00:41:25.500 --> 00:41:26.260 Антон Обожин: Hmm!

241 00:41:26.400 --> 00:41:51.460 Антон Обожин: I have a small remarks that it is wonderful that you're doing that in English. But many things in Wikimedia are too much based on English language, too much biased towards English language, like Commons, is completely in English, completely in English, and the foundation gives most of it's

242 00:41:51.600 --> 00:41:56.969 Антон Обожин: attention to English Wikipedia. Exactly. And

243 00:41:57.090 --> 00:42:03.039 Антон Обожин: other language versions. Other languages like

244 00:42:03.090 --> 00:42:05.630 Антон Обожин: attention from develop

245 00:42:05.680 --> 00:42:18.070 Антон Обожин: first from foundation and the struggle a little bit. So it's important to scale this project to as many languages as possible.

246 00:42:18.100 --> 00:42:21.350 Антон Обожин: and not only focus completely on English.

247 00:42:25.460 --> 00:42:29.780 Liam Wyatt: Mike, that's 2 votes. So far for more than English, please.

248 00:42:29.780 --> 00:42:37.609 Maryana Pinchuk: Just a just a quick comment on that. Yes, I you're absolutely right. And to to go back to the very start of this presentation

249 00:42:37.610 --> 00:43:01.760 Maryana Pinchuk: the aim of any future audiences. Initiative is not to build a finished product what we're doing is trying to understand whether there is interest in demand for this experience. And if so, we will stop the experiment. And we will hand this off to a team of engineers and designers and developers and interns a internationalization localization specialists that we that we have on staff.

250 00:43:01.830 --> 00:43:10.859 Maryana Pinchuk: who would make this available in all languages? So we are. We are not the team that would do that. But we are the team that would make that recommendation.

251 00:43:14.530 --> 00:43:16.840 Liam Wyatt: It's also important to get the

252 00:43:17.370 --> 00:43:19.120 Liam Wyatt: largest visibility

253 00:43:19.240 --> 00:43:24.649 Liam Wyatt: testing platform, because if it doesn't work in English. It's not going to work in other languages.

254 00:43:26.410 --> 00:43:29.810 Liam Wyatt: the is it? Is it correct to say, Mike, that

255 00:43:29.890 --> 00:43:35.549 Liam Wyatt: this is also the back end of this system is also based on the Wikimedia enterprise. Api.

256 00:43:36.640 --> 00:43:39.099 miishen: I'm I'm confusing something else.

257 00:43:39.580 --> 00:43:41.719 I'm not fully

258 00:43:42.030 --> 00:43:52.770 miishen: cognizant of the current state of the you know architecture. II know that we use the search Api, like we did with the Wikimedia or the Chachi Bt. Plugin.

259 00:43:52.810 --> 00:44:08.609 miishen: And I know that we've been working closely with enterprise. There's some discussion right now whether it's going to be hosted on tool forage or use the enterprise, you know, back in. So that's why, that's like as of this week. So that's why I don't have like full details on exactly which format it will take.

260 00:44:08.780 --> 00:44:24.230 miishen: I mean, currently our our sole engineer working on this part time. It's kind of sitting on this machine locally, basically. And so that's why, you know, we don't really know where it's going to be hosted yet. We're kind of working out these decisions as he's building it.

261 00:44:24.380 --> 00:44:28.919 miishen: but I believe it is going to be on tool forge.

262 00:44:30.140 --> 00:44:37.990 Liam Wyatt: and could you speak unless anyone else had their hands up? Please please come up, but I will add comments at Mike to elaborate further things.

263 00:44:38.320 --> 00:44:49.680 Liam Wyatt: Could you speak a bit about the name? Choice specifically what is not named. and the significance of that in terms of

264 00:44:51.170 --> 00:44:53.730 Liam Wyatt: the expectation management

265 00:44:53.870 --> 00:44:58.360 Liam Wyatt: of both Wikimedians and the wider Internet.

266 00:44:59.420 --> 00:45:08.059 miishen: Yeah, of course, thanks for bringing this up. I think we had a lot of deliberation on the name and most notably it's not called a fact checker like I said

267 00:45:08.260 --> 00:45:16.900 miishen: and it easily could be called that, because that's seemingly what the function is. But I think there's a lot of kind of

268 00:45:17.360 --> 00:45:34.560 miishen: technical to philosophical debates about. You know what fact and truth is, and what our position in asserting it or not asserting it is, and what I really wanted to avoid is a world in which we're saying, this is the facts of the world. You know, this is the truth in the world. We are the kind of decision makers about

269 00:45:34.580 --> 00:45:37.810 miishen: what truthfulness truthiness is, or something.

270 00:45:38.250 --> 00:45:50.179 miishen: And I and I really wanted to. If you stick, take a step back and look at the chat. Utt Plugin. One of the things we've learned is people trust our brand right? And so I'm kind of trying to thread this needle of saying, like

271 00:45:50.500 --> 00:46:09.310 miishen: Wikipedia, is for most people pretty reliable good resource. And so, you know, you should look at Wikipedia as a way of like verifying or checking information. It's not going to tell you the truth. People know. Sometimes there's mistakes. People know. Sometimes things are missing, and it doesn't mean it's truthful, or it's not truthful, but

272 00:46:09.640 --> 00:46:17.249 miishen: it does lend a good resource, and it does provide a good at least first step of like credibility.

273 00:46:17.300 --> 00:46:41.229 miishen: And so part of this experiment is also kind of seeing like what the power of that, you know Brand is. You know, like how powerful is Wikipedia as a resource for people, as a trustworthy kind of source of information. And then we provide like, here's something that looks relevant to what you were searching for. And then the kind of like needle to thread it's kind of like that final mile is like if people really do trust us as a source of information. Then

274 00:46:41.580 --> 00:46:51.400 miishen: we should see that the that last gap is kind of like, okay, well, it's on Wikipedia. Then, like, maybe my trust in this information goes up, and I can do more research to kind of like.

275 00:46:52.510 --> 00:46:54.839 miishen: Continue verifying this, for example.

276 00:46:54.990 --> 00:46:59.130 miishen: but we can add a call to action of

277 00:46:59.160 --> 00:47:00.639 Liam Wyatt: come over and edit.

278 00:47:01.000 --> 00:47:05.929 Liam Wyatt: especially the content which produces a a null result, perhaps.

279 00:47:06.190 --> 00:47:11.180 miishen: Exactly. Yeah. There. There is a question. Can you please go back to slide. 9, by the way.

280 00:47:12.160 --> 00:47:16.749 Liam Wyatt: So yes, what happens if there is a null result? Or if the fact that is returned

281 00:47:16.900 --> 00:47:20.400 Liam Wyatt: is a refutation, or

282 00:47:21.120 --> 00:47:25.189 Liam Wyatt: perhaps even vandalism. What's what's the negative result?

283 00:47:26.230 --> 00:47:37.020 miishen: So I think the most neutral result is to say that this is not support. Right. If we can't find anything, then it doesn't mean it's not true. We'll just say like, Hey, we couldn't find anything on Wikipedia to support this, you know, and

284 00:47:37.120 --> 00:47:40.420 miishen: that's once again not saying it's true or not true. So that's kind of

285 00:47:40.650 --> 00:47:49.050 miishen: perhaps an easier case. Right? There's a lot of information that is not on Wikipedia that should not be on Wikipedia, and it doesn't mean that it's not real

286 00:47:49.310 --> 00:47:59.570 miishen: with the AI in some cases will be able to tell whether there's conflicting information. So one of our mockups.

287 00:48:00.040 --> 00:48:24.089 miishen: there's, you know, a claim about like, I think, like aside varies being like super food or something, and like some dubious claims, health claims and then the Wikipedia article that we have in that example says, like, you know, maybe head your bets a little bit like maybe some of these claims aren't true. And so in that case, there's clear conflict of information, and we can label it as such and say, like.

288 00:48:24.170 --> 00:48:39.119 miishen: you know, this looks wrong, basically. Or these things don't match up right. Whatever this claim is, whatever is on Wikipedia, they don't match up, and we will indicate that in some way. I think at first we had very conspicuous things like check green checks, Red X's.

289 00:48:39.350 --> 00:48:46.199 miishen: and we might wanna we're considering, maybe backing off on that like, there's some signal that's important, right? Like, there's something important about like.

290 00:48:46.270 --> 00:49:06.960 miishen: I'm being prime to read the information in a in a useful way that, like something read means like, Oh, hey! I should be careful. I should like, look into this more. Something is off. So we want to keep that feeling. But without doing it in such a way that, like X, this is wrong. We've made it like a very strong judgment that this is like false etc.

291 00:49:06.960 --> 00:49:23.640 miishen: And so we don't know what final form is going to take out or take. Yet we're working with our designer to kind of keep that, like, you know, alert mode of that prime mode, but without being too strong, where somebody just sees the x, and then like never follows up, and just like thinks that you know the claim is wrong or something.

292 00:49:24.480 --> 00:49:27.359 Liam Wyatt: Can we see the video again as a request in the chat?

293 00:49:28.200 --> 00:49:40.260 miishen: Ii didn't show the video. It's 2 min long. This is a demo. Video. This is a screenshot from it. I'm I'm happy to show it. I don't know how good the quality will be, but I'm also wanting to be aware of the time.

294 00:49:40.690 --> 00:49:46.419 Liam Wyatt: We have 5 min, and there is not a queue of questions so I think it might be

295 00:49:47.000 --> 00:49:53.050 miishen: useful. Right? You could make it full screen. Yeah, give me a second.

296 00:49:55.440 --> 00:50:15.540 miishen: Hello, folks, this is Amin from the design team. And today I'm gonna give you a workthrough extension that we are working on with future audiences team. It's called citation needed. And it's a Wikipedia and Gpt power extension that you can use to quickly verify content online as you runs the web.

297 00:50:15.790 --> 00:50:18.730 miishen: So here, imagine you're reading an article

298 00:50:18.900 --> 00:50:24.290 miishen: on an entertainment website which says that Taylor student concert caused earthquake.

299 00:50:24.520 --> 00:50:34.499 miishen: So you have to browser extension install, and you can see here your browser chrome. And if you trigger the extension tells you that you have to highlight one sentence to start.

300 00:50:34.640 --> 00:50:47.809 miishen: So we're gonna do that. And now that we have highlighted a sentence, we can trigger the session again, and we see that the sentence is here extension, and you have a big verified button. So we're gonna trigger that.

301 00:50:48.240 --> 00:50:51.300 miishen: And now the new verification process is going to start

302 00:50:51.430 --> 00:51:12.320 miishen: here. We're going to be transparent about what's happening behind the scenes. And so we're gonna say, extension users that we are reading the sentence. Now we are disrupting relevant keywords from the sentence. In order, then, to trigger a search on Wikipedia and through the search, we're gonna then find and generate ourselves

303 00:51:12.410 --> 00:51:14.840 miishen: for papers. There's to be reviewing.

304 00:51:14.970 --> 00:51:25.270 miishen: And so once the result is ready, we're gonna load the article that we found we're gonna provide rather than quotes from the article. This case is about

305 00:51:25.280 --> 00:51:35.459 miishen: relevant information on these system activity. And we're gonna also give additional information that will be full signals of quality

306 00:51:35.470 --> 00:52:05.150 miishen: regarding the latest, adding a number of references and a number of people who work article. And beside the citation result, we're all gonna give quick links or quick direct links to visiting the article page. And we also have here. Gonna click! Here it's gonna bring us to the Wikipedia article with the citation highlighted. And people can continue reading. And yeah

307 00:52:05.200 --> 00:52:08.649 miishen: came to their own conclusions after years of accession.

308 00:52:08.810 --> 00:52:12.910 miishen: And that should be all. Thank you very much, and thank you soon. Bye, bye.

309 00:52:19.570 --> 00:52:25.439 Liam Wyatt: video. There's a question about whether this could or should be a booklet, a book market.

310 00:52:25.490 --> 00:52:39.749 Liam Wyatt: or an extension, or rather than necessarily a plug, into a browser. Sorry! An extension to present. What is the other ways of doing it? I'm assuming the answer is because this is the quick and easy way to start. And in the future, if it's very successful.

311 00:52:39.760 --> 00:52:42.930 Liam Wyatt: who knows how it might become more solid?

312 00:52:43.500 --> 00:52:47.230 miishen: Yeah, I didn't really paint a lot of the

313 00:52:47.590 --> 00:52:55.950 miishen: super idealistic feature state of this once again. That that is the answer. This is the quickest, easiest way for us to just get something in front of people to get results.

314 00:52:56.250 --> 00:52:59.450 miishen: if you ask me more like

315 00:52:59.930 --> 00:53:05.880 miishen: my end goal here, if this becomes a successful product is not to build like the best browser extension on earth or something.

316 00:53:05.890 --> 00:53:12.770 miishen: What I would really like to do is to prove the value of Wikipedia content as information verification

317 00:53:12.860 --> 00:53:38.160 miishen: especially in places where they're needed, such as like social media platforms and incentivize them to build it natively. So that like on Facebook, you could like, like comment or just verify something. And they just build it themselves. And they see the value. And then we provide maybe an Api or something. Instead of having to maintain a suite of like dozens and dozens of different plugins into different ecosystems, they would want to come to us and and have some

318 00:53:38.200 --> 00:53:44.729 miishen: and have something native that's going to work better and just be able to connect it and our information in a way that makes sense.

319 00:53:45.280 --> 00:54:02.350 miishen: And then also, I make a comparison with with with ad blockers. Right where, 15 years ago, on the Internet, people installed all sorts of extensions to manage their privacy and etc. And nowadays it's kind of just a built in way of using the Internet where most browsers have some rudimentary fund functions to

320 00:54:02.420 --> 00:54:20.710 miishen: block all the garbage online. And we're just seeing that there's different types of garbage online these days. And you know, is there a world in which you know Firefox or Chrome is gonna just build this in natively, and then once again we work with them directly. But we're not on the hook for building out and maintaining like dozens and dozens of like these different

321 00:54:21.020 --> 00:54:24.400 miishen: a plugins or extensions, or what have you?

322 00:54:26.750 --> 00:54:36.470 Liam Wyatt: Alright? Thanks, Mike. We are at time, and I know that Susana would like to have a moment to talk about an event that's coming up in Helsinki talent.

323 00:54:36.950 --> 00:54:54.389 Susanna Ånäs: I can just briefly say that II pasted 2 links in the chat because I could, of course, run through the, you know, mood board slides, but I think I might save time by just pasting the links. I can do that if you wish to stay on the call. I don't know.

324 00:54:56.450 --> 00:54:58.859 Susanna Ånäs: Yeah. Can I share.

325 00:54:59.320 --> 00:55:00.869 Liam Wyatt: I think, can you?

326 00:55:01.810 --> 00:55:02.940 Susanna Ånäs: Maybe

327 00:55:03.890 --> 00:55:09.549 Susanna Ånäs: it seems so. Yes, it seems many people are here already. So

328 00:55:09.710 --> 00:55:20.750 Susanna Ånäs: It's happening in Helsinki after the Tallinn Hackathon, and this is a 2 day event starting on Monday, the sixth, and lasting until the seventh.

329 00:55:20.790 --> 00:55:47.760 Susanna Ånäs: So who are joining this is sound like finished. AI finished glams. Research projects Wikimedians from the from the hackathon, and hopefully also very much from the foundation and the the chapters and affiliates staff, and then companies as well as then other open knowledge advocates.

330 00:55:47.970 --> 00:56:00.659 Susanna Ånäs: And here's the first day we'll have inspire talks in the morning. In the afternoon. We'll be having a like a ideation and group forming exercises for the whole afternoon, and then have sound

331 00:56:00.840 --> 00:56:15.920 Susanna Ånäs: and then start, you know, like working towards the next day, which is a practical collection of different practical activities for different groups of people, some some of them learning how

332 00:56:16.520 --> 00:56:28.579 Susanna Ånäs: learning like nuts and bolts about AI, just dipping their toe in the in the topic that could be also serious hacking with serious tools.

333 00:56:28.580 --> 00:56:50.870 Susanna Ånäs: And then it could be a strategic discussion between different advocates of open knowledge like policy issues that we we discuss with different types of methods and and practices. So it's a like a big playground for activities of this type. So so this is the timeline, like from

334 00:56:50.870 --> 00:57:08.950 Susanna Ånäs: empathizing to ideating, prototyping, even testing, you know, just trying out what's there, and maybe that, you know, like the the Plugin would have been a fun fun thing to test in that. That's and if you know that you have the link to the presentation, you also have links to

335 00:57:08.970 --> 00:57:22.209 Susanna Ånäs: planning document and project page, which is actually updated. We just shortened the name from glam plus commons plus AI sound into AI sauna. So it's just more.

336 00:57:22.220 --> 00:57:31.050 Susanna Ånäs: It's it like it will look better in a link. So III would really be happy to see many of you there.

337 00:57:33.190 --> 00:57:51.600 Liam Wyatt: That was lovely, and and I appreciate the quality, the visual quality of those slides. That is the there is one more further question. Child, people can certainly leave. We are over time. There's a question in the chat which deserves

338 00:57:51.780 --> 00:57:59.860 Liam Wyatt: mentioning. Since I have been asking for questions specifically on interactive content and the

339 00:57:59.950 --> 00:58:03.689 Liam Wyatt: capacity of the Wikimedia Foundation to support development, for that

340 00:58:03.970 --> 00:58:25.990 Maryana Pinchuk: was addressed it. Marshall, who is no longer in the call, since we're over time. But, Mariana, perhaps you can speak to that. I can speak to it. Yeah. So thank you, Sandy. It's a really great question. And I think the the main takeaway from what we're trying to do here is that before we can even talk about headcount we need to understand what it is we need to be building

341 00:58:25.990 --> 00:58:49.229 Maryana Pinchuk: whether that's an an amazing AI driven AI powered new reading experience, or a video platform, or a place where people can make interactive graphics and come together and and and collaborate over those, those, each of those things would be a completely different direction. Product and tech wise, and would require completely different resourcing.

342 00:58:49.230 --> 00:59:05.939 Maryana Pinchuk: So before we can even get into understanding, you know how many people we need, or even who we need, we need to understand what is needed for us. As a movement to evolve into. And and we still don't know. There are so many changes happening so rapidly.

343 00:59:05.970 --> 00:59:09.270 So that is the thing that we are trying to do first

344 00:59:09.410 --> 00:59:11.260 Maryana Pinchuk: before anything else.

345 00:59:15.170 --> 00:59:26.590 Liam Wyatt: Sorry, thank you, Mariana. Hope that answers the question. I wanna add in that this being the future audiences team. This isn't a team per se. It's a rag tag bunch of individuals rather than

346 00:59:26.840 --> 00:59:34.970 Liam Wyatt: other parts of the Wikimedia foundation technology and engineering community who are standing

347 00:59:35.040 --> 00:59:42.190 Liam Wyatt: groups who focus on apps or search, etc. This is experimenting in a variety of ways in general.

348 00:59:42.850 --> 00:59:46.050 I will now turn off the recording and upload this

349 00:59:46.240 --> 00:59:49.330 Liam Wyatt: two-week media commons shortly afterwards.

350 00:59:50.280 --> 00:59:54.319 Liam Wyatt: and people are welcome to stay here. Thank you for your time.

351 00:59:56.080 --> 00:59:57.219 Maryana Pinchuk: Thank you.