Wiki Education Foundation/Quarterly reviews/2014-Q3 Programs Strategy
The following are notes from the Quarterly Review Meeting for the Wiki Education Foundation's Director of Programs on Februrary 2, 2015.
Present: Jami Mathewson, LiAnna Davis, Samantha Erickson, Renée LeVesque, Bill Gong, Eryk Salvaggio, Frank Schulenburg
Participating remotely: Sage Ross, Ryan McGrady, Helaine Blumenthal, Ian Ramjohn, Adam Hyland
Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases, and other official material.
Introduction[edit]
Frank: Quarterly reviews are a good start for people within our organization for getting an overview of what others are doing, for looking at what we have achieved, and for reflecting on what we are seeing across departments. Some questions to answer during these sessions: what do we need to change in order to better achieve our mission, and what are your roadblocks?
Also, quarterly reviews provide transparency. We want to achieve a high level transparency towards the Wikipedia community, our board, our funders, Wikimedia chapters, etc. That’s why it’s important that the discussion we have internally is visible to the people outside of our organization.
Our monthly reports already provide useful information about what happened in the past. The quarterly reviews are for big picture and they also look toward the future. Also, they provide us with an opportunity to have more in-depth discussions.
This marks the first overall Programs Strategy conversation to determine directions and opportunities for growth. It will also be more discussion-heavy than quarterly reviews in the past.
LiAnna: Thanks Frank.
Some context for our work (slide 2, Oct–January). Supporting programs, but also representing programs to the board, our funders and other organizations – places where a senior representative is needed.
Ways of growing our organization’s impact in the future[edit]
Strategic planning process: Frank and Bill, LiAnna, board members, and external experts. Large component of LiAnna’s time as she focuses on this effort.
Annual planning: Programs is the largest chunk of the annual plan. As part of the process, LiAnna will be doing a 2-day planning sprint with Sage and WINTR in Seattle, collecting feedback from the rest of the staff, setting numeric goals for annual plan.
LiAnna: I want to think about this work to create an open discussion rather than presenting work to the group.
- Preliminary mission statement (mission slide)
- What’s keeping us from fulfilling our mission?
- Brainstorming session
Ryan: Seems like we’re trying to recruit subject matter experts?
LiAnna: The term “subject matter experts” includes students.
Ryan: It sounds like professors.
LiAnna: We hadn’t thought of that, we always refer to students as subject matter experts.
Jami: We do engage instructors to bring students to Wikipedia.
Frank: Maybe we need to think about this in a broader way. We started this organization with focusing on the existing classroom program, but what about other areas of academia that can help with improving Wikipedia’s content? Think, e.g. of university libraries, university archives, university museums etc. Should we also work with these groups on campus?
LiAnna: The question I want to focus on is the spirit of the mission, we can address the language later, but we’re looking at the idea of the mission statement as being able to broaden our mission work. We don’t want to narrowly define it to university staff, or structured university systems.
Frank: For example, lets say there’s a list of 1,000 potential names of women writers. What keeps us from filling that list with 1,000 feature articles? And how would we get there?
Sage: One thing keeping us from fulfilling our mission is that its hard for academic subject matter experts to develop the skills they need to write that feature article – the whole gamut of skills needed to write that level of content is not easy to pick up.
LiAnna: What can we do about that?
Sage: One of the things we can do is provide better rails for leading people along the steps that they need to take to contribute good content.
LiAnna: Like what?
Sage: Specific instructions for specific tasks, managing the volume and complexity of those steps so instructions come up when they need them, to scale down the level of reading required before people get started.
Helaine: Instructors may go ahead with Wiki assignments without using our resources or following best practices.
Jami: One benefit of broadening this – once you convince an instructor, they bring the students along. But instructors want a lot of autonomy, so they may not react well to heavy-handed instructions such as “edit these 1000 pages on women.” Whereas partners may help strategize targets and goals and decide on what articles make good targets.
Helaine: Many find article selection daunting, so being a bit heavy handed on recommendations can be helpful. Many instructors welcome our suggestions on articles, but its not easily accessible
Jami: We don’t really have it at all.
Helaine: No, its done on a course by course basis.
Bill: How much work is that for Adam and Ian?
Adam: That’s what we’re going to find out this quarter. This summer we’re going to tackle this and get a sense of time.
Frank: Jami, what I hear from you maybe is that we need to do things a bit differently? Do we need to be more involved in topic selection? And, on a higher level of thinking: should we increase our impact by fine-tuning our program mechanics, or should we have more impact by focusing on scaling up the number of participants in our programs? Would we need to hire 40 more people to have more impact, or just improve our approaches to how we do things?
Jami: It depends on the fit with the program; my suggestion would be up to instructors to opt in. Many instructors may not need us to do that for them; but there’s potential for us to give guidance.
LiAnna: So, what I hear is, we have one core program that isn’t designed to take a list of 1,000 articles and cross those 1,000 articles off of a list. We have a program that splits this up across a variety of interests, but we’re targeting broadly when individual courses are more narrowly focused… A society or partnership, perhaps, would be better at broader…
Jami: For example, one organization has a goal of promoting, specifically, historical women, and working with them may be more fruitful…
Frank: I have a second question: what’s our sense of urgency? Or, in other words, does time matter? Would it be ok for us to stay a relatively small organization that only supports a comparably small number of participants in our programs, meaning we would slowly grind our way through the existing content gaps, even if it takes the next 100 years to fill those content gaps?
Adam: 100 years is out of scope for even a 5-year-plan. But think about utility – how useful is scope and time for conveying what we’re doing, for recruiting people. When we talk about scaling up, it’s students and professors who will think about topics… even if we double in size, we’re better off thinking about targeting and allowing other people to build on what we do. How do we pick a discipline and say, for example, “Women in history is now 100% better?”
Frank: I don’t know, I am not convinced, really, I think there is a certain need for readers to get certain information and it’s not there. Does it matter when we provide them with that information?
Adam: But that isn’t our mission really. Our mission is to make that information richer, our actions should keep in mind that we are trying to direct students toward writing about what they are excited about, not covering the breadth of human knowledge.
Frank: Well, let’s say we found out that we had huge content gaps in the field of medicine. What if the articles about the ten most important causes of death were incomplete? And Wiki Ed has a way of systematically filling content gaps. What a super weapon that is! It's like the diamond sword that you want in a video game and that would enable you to easily win the game! In our game, the content gap is the enemy, and our method is potentially the most powerful way to fill those content gaps. We have that diamond sword, now how are we going to use it?
Adam: We have to think about what the audience says, though – students see a list of 100, and want to add 100 more. We tell instructors how to find content gaps and fill them. If we hire me or Ian 10 more times, we may never match that scale. We want people to be conversant in what we do.
Frank: That’s a legitimate position, but if we were still grinding through that list of articles about the most notable 100 women scientists in ten years, would that really be ok?
Samantha: Don’t we feel like we are already doing that on a small scale that could grow?
Sage: Another answer to this question is that academic content experts often find the Wikipedia community to be hostile to their contributions. The solution is finding a way forward through that; can we change that dynamic, or help the academic content experts dive into it in a way that doesn’t provoke reactions. When we bring in academic content experts, and they get invested in Wikipedia, they do start to bring their voices in and participate in the community. But broader trends show there are fewer and fewer people contributing to daily processes; many Wikipedians feel that the community is kind of breaking down; administrative concerns don’t get dealt with quickly enough, etc. We need to think about how to turn this around, and really, to address Wikipedia as a social place that can be a challenge.
Frank: So we have to make academia understand why the Wikipedia community reacts a certain way; but also to work with existing community to be less hostile towards new users? Is that what you mean?
Sage: Not just instructors, but students. We’ve acknowledged student retention isn’t something we’re going after. That would be great, to create students who continue editing and rejuvenate the spirit of the community and get into the internal world of Wikipedia and move the community forward.
LiAnna: That is interesting for deviating a bit from just content gaps; it’s recommending that people get involved in the daily tasks that keep Wikipedia running.
Sage: Yes, and the more they dive into the community and Wikipedia's processes, the more others will receive them in a collaborative rather than confrontational way. They may be more inclined to do things like get a Featured Article.
Frank: But it can happen that people slowly get involved in the Wikipedia community and then stop writing articles, start talking about discussions and deletions, etc. That’s perhaps why we don’t want academics getting engaged with daily tasks.
Sage: If we do scale up and half of the contributors to Wikipedia are now student editors, what happens to the daily practices of Wikipedia when that happens? Maybe the way you do that is by giving people enough instructions to make sure they don’t instigate a negative reaction to their work.
LiAnna: For context, there are some in community who are hostile to academic activity on Wikipedia; Wikipedia is ground up collaboration on knowledge and free knowledge and they can view academics as top down, etc. But cultural change to make things more open to academics is, certainly, to get more people invested in the community. And to get people who believe in and understand Wikipedia as it is, but share a vision for the future of Wikipedia that is more open.
Sage: It’s critical to have both kinds of people. There are spots where hostility can be triggered. But on the other hand, you're a long-time Wikipedian focused on articles, and you think you’re doing good work... and then a peripheral anon comes along and griefs your work. You can try to use Wikipedia processes to solve this problem, but what if you can’t find a good community processes to resolve it? What if there’s nobody around who can help navigate the system? You want allies who will help resolve issues before they disappear into a black hole without a resolution. When that happens can end up with a burnt out editor, frustrated by internal problems.
Jami: So are we saying that we need to grow not only content editors, but also the people who run the machinery of Wikipedia, or to create well-rounded editors? Because I see Wikipedia has people who do one thing, like clean up, but not well-rounded, per se. And I don’t think students are going to become that kind of all-around editor or invest in single tasks. But can we target, say, librarians, to recruit say page patrollers, etc, maybe we say “1 per 100 editors” or something?
Frank: Maybe we should invest in training librarians or other staffers on campus to perform certain tasks that our students are not good at. I’m thinking about librarians as a perfect group for checking references and improving those parts of articles that student editors have contributed to Wikipedia.
Jami: We have always said that we’re trying to find parallels between Wikipedia and academia. If we are expanding the scope of the mission beyond academia or universities, let’s identify the different roles and traits that improve Wikipedia, and find where they align with academia and students… if we don’t do this, we’re overwhelming content without supporting the rest of Wikipedia. We were moving there with the task-based tagging system.
Adam: I don’t think we need to worry about the experience level of editors, but maybe we could implement good ramp ups, and maybe 2 of those 100 will stick around, and if we scale up that means many more…
LiAnna: That’s been tried, it doesn’t work.
Adam: Then would a targeted excursion work, then? We’re not very good at retention, maybe 2 out of 300 or something?
LiAnna: Well, we looked at retention and the goal of our organization is content. In my experience in 9 countries, I think people either come to Wikipedia for fun in their free time or they don’t. If you don’t like it, the best intentions and training won’t make people keep editing. 60% of students say they want to continue editing, but maybe 1% actually do. It’s not about retention anymore, but about getting students in universities, or in Sam’s program for example, to find potential editors.
Frank: But we’re assuming that if we train editors, they will keep writing. But they don’t always keep writing articles. If our goal is to fill content gaps, do we need more people doing page patrols? Remember, our goal is to fill content gaps…
Sage: The thing that keeps new contributors from continuing to improve Wikipedia content is the hostility of the community and the breakdown that comes up from behind the scenes. We need to find ways to allow students to help the system keep up.
LiAnna: We find that those who keep editing might keep working, but not always in the same subject area/content gap they were studying in class. I've seen students continue editing, only to stop having their work accepted because of an aggressive dispute over the color of the tables they were building.
Jami: I just don't think students going to stay; they aren’t going to edit Wikipedia on a large scale. We know retention is hard. Undergraduate student groups, maybe if they are interested in the idea of information access as advocacy, but its a tremendous uphill battle to get undergrads to keep editing.
Helaine: Have we ever just tried getting academics? Forget about students for a minute, have we asked just academics?
LiAnna: We’ve tried, it doesn’t work; until Wikipedia content is recognized for tenure, it’s not likely to happen...
Sage: Or if people get published with articles written as Wikipedia articles [like the PLOS Computational Biology "Topic Pages" project] . This is something we could try, if we want to follow this and solve that, it’s possible.
Jami: But then there are instructors struggling with Wikipedia, which looks at what you are referencing, not who you are.
Samantha: We’re looking at fellowship programs that are interdisciplinary, where people are being asked to contribute to a program rather than a class.
What’s the right growth trajectory for our programs and our organization?[edit]
LiAnna: So the next question is, what’s the right amount of growth? We just brainstormed a bunch of ideas, ie, grad students, librarians, scaling classroom program, honing processes, etc. So, I’d like to ask about the feeling of proper amounts of growth – should we have 40 staff members next term, do we say we want 1 person for all programs, or for all programs… what do we want to do? Obviously the answer is somewhere between those extremes…
Sage: Growth of our programs and growth of our organization?
LiAnna: Yes.
Frank: One option is to grow our organization from, say, 5 people in classrooms program to 20 in order to scale the impact of our work. A different option would be to develop a couple of new programmatic approaches that complement our existing classroom program.
LiAnna: Pretend all existing contracts and limitations are off the table. Pretend we have infinite resources.
Bill: But, you know, first, are you satisfied with the improvement of the current programs? If we can grow in this program why put growth into new programs?
Frank: Maybe we’re missing an opportunity?
Bill: Sure, but growth includes growth in the current program.
Jami: I think saying we won't experiment with other programs until we've maxed out on the current one is a tough road because there are thousands of universities we could be working with in the US/CA.
Bill: But maybe within classroom program, are there areas that are lacking or content gaps that could be tackled, would these kind of things improve participation… just don’t leave program growth aside in the hunt for new programs.
LiAnna: Given the constraints we have, we’re doing pretty good. But for example, I’d love to see growth in our tech budget, which would help make things significantly easier for program participants. We want to make the classroom program scale better, of course, and we want to develop that, but do we want to scale more with staff while we refine that, or diversify what we’re doing?
Samantha: It seems like the right amount of growth is the question, Wiki Ed seems like it always wants to try new things.
Frank: Like the car factory, you have one model that sells really well. Would that keep you from developing a new car line? Then: how many new car lines do we want? Also, look at it from this angle: what information is missing for us to make decisions like this?
Sage: We’re looking at what we can build upon to do better at what we do.
Helaine: Measuring the quality of the “car” as it is, I think we do a bit haphazardly. It’s important to know what is working, and improving the quality.
Frank: So a dedicated evaluation person?
Helaine: Yes.
Jami: Yes. And we’ve had a lot of conversations, but one thing we should consider is: do we want to find the born Wikipedians, who fit perfectly into Wikipedia as it currently exists? Or are we starting programs that specifically target the kind of people who will diversify Wikipedia?
Frank: We have always been talking about growing the amount of content that our student editors produce. Now, what about further improving the quality of the content we get per individual student editor? Is that another viable option of thinking about what we should focus on when we’re talking about our future strategy?
Wrap up[edit]
LiAnna: Any final thoughts? Three minutes left. This has been extremely useful for thinking about the different directions we could take as an organization.
Samantha: It would be interesting to have a formal discussion of ideas where people propose ideas.
Frank: If we as an organization could do one additional thing that would help us to reach our mission, what would it be?
Ian: Just on the idea of Wikipedians being born… there are good candidates in community college faculty. Is there something we can do to make editing Wikipedia valuable to them?
Adam: I want to answer that by suggesting… doubling Sage’s budget and targeting interventions to the point where we know students will create good work in response to the tools, guide students as an intervention toward articles and knowing they would improve. You could scale the same thing out after that.