Grants:APG/Proposals/2017-2018 round 1/Wiki Education Foundation/Impact report form

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search


Purpose of the report[edit]

This form is for organizations receiving Annual Plan Grants to report on their results to date. For progress reports, the time period for this report will the first 6 months of each grant (e.g. 1 January - 30 June of the current year). For impact reports, the time period for this report will be the full 12 months of this grant, including the period already reported on in the progress report (e.g. 1 January - 31 December of the current year). This form includes four sections, addressing global metrics, program stories, financial information, and compliance. Please contact APG/FDC staff if you have questions about this form, or concerns submitting it by the deadline. After submitting the form, organizations will also meet with APG staff to discuss their progress.

Global metrics overview - all programs[edit]

We are trying to understand the overall outcomes of the work being funded across our grantees' programs. Please use the table below to let us know how your programs contributed to the Global Metrics. We understand not all Global Metrics will be relevant for all programs, so feel free to put "0" where necessary. For each program include the following table and

  1. Next to each required metric, list the outcome achieved for all of your programs included in your proposal.
  2. Where necessary, explain the context behind your outcome.
  3. In addition to the Global Metrics as measures of success for your programs, there is another table format in which you may report on any OTHER relevant measures of your programs success

For more information and a sample, see Global Metrics.

Overall[edit]

Metric Achieved outcome Explanation
1. # of total participants 16,887 We exceeded our goal of 16,035.
2. # of newly registered users 15,699 We exceeded our goal of 15,520.
3. # of content pages created or improved, across all Wikimedia projects 18,028 Due to the Student Program change noted below, we fell short of our goal of 19,960.
4. # of words added to the article namespace 14.2 million We exceeded our goal of 12 million words added.
5. # number of articles that have at least a 10-point improvement ORES-based quality prediction score 4,935 We fell about 500 short of our goal of 5,475. See below for an explanation of our learnings about this metric.


Telling your program stories - all programs[edit]

Introduction[edit]

Calendar year 2018 was one of big change for Wiki Education — and our most impactful year to date. Through our strategy, fundraising, and programmatic work, we made great strides and have set ourselves up for an even more successful 2019 — with lots of learnings along the way.

Strategy[edit]

Our board met in January to confirm our strategic direction for 2018–21, with a focus on three pillars: Equity, Quality, and Reach. In June, the board approved the strategic plan emerging from that direction. Our new three-year strategy, includes the following goals: (1) Increase knowledge equity by focusing on content and communities that are under-represented on Wikipedia and Wikidata; (2) Provide people who seek knowledge online with accurate information in topic areas that are underdeveloped; and (3) Reach large audiences with free knowledge by making Wikipedia and Wikidata more complete.

Wiki Education Strategy for 2018-21
Program & Events Dashboard usage over time.

This strategy complements our existing strengths, while more directly aligning our work with the overall Wikimedia Movement strategic direction in both the "knowledge as service" and "knowledge equity" pillars. Our support of the Program & Events Dashboard is key to both pillars. As an important piece of infrastructure used by 2,006 program leaders globally to support programs on 101 different wikis, the P&E Dashboard is a tool that serves the global Wikimedia community as we interact with GLAM institutions, universities, and other knowledge repositories to collect and use our knowledge. P&E Dashboard usage more than doubled in 2018, from 1,418 to 3,240 courses or events tracked. Because it works across all languages and all projects, global Wikimedians working on knowledge equity issues have found it to be a key piece of software; it's been used by program leaders doing projects on the Acehnese Wikipedia (ace.wikipedia.org) to the Taiwan Chinese Wikipedia (zh-tw.wikipeida.org) and everything in between, including the Arabic Wikibooks, the Hebrew Wiktionary, the Indonesian Wikisource, the Italian Wikinews, the Polish Wikivoyage, the Portuguese Wikiversity, and the Spanish Wikiquote. Many of the courses and events tracked in the P&E Dashboard are run by program leaders specifically interested in improving knowledge equity.

Our own work in knowledge equity has been confirmed as a priority in our new strategy, in terms of both content and contributors. While we are unaware of any Wikipedia editing community research that has identified race of contributors, our survey results from fall 2018 showed 37% of our student editors identify as non-white; we believe this is significantly higher than the general editing population. Through our targeted outreach work to faculty who teach in equity content areas, we've also improved thousands of articles related to race, gender, and sexuality.

Fundraising[edit]

We recruited for paying customers at the American Geophysical Union's 2018 annual meeting.

In 2018, we hit a major milestone: We brought in our first money from our new fee-for-service model. Thanks in part to a push from the FDC and an organizational development grant from the Hewlett Foundation, we worked with a business development consultant to explore possible business models for our work. The investigation revealed we'd have a hard time making a fee-for-service model work with our Classroom Program. However, we discovered demand for a version of our Wikipedia Fellows program aimed at providing professional development credits — and institutions have professional development budgets to pay for this. As such, we re-organized our staffing to create a new Advancement Department tasked with enacting the new fee-for-service model, rebranded our programs into the Student Program (formerly Classroom) and Wiki Scholars & Scientists (formerly Wikipedia Fellows), and set off to bring in money in this new model. In our first two courses, we partnered with the National Archives and Records Administration (NARA) to encourage experts to improve articles related to the 100th anniversary of the 19th Amendment to the U.S. Constitution, which granted women the right to vote. NARA is planning an exhibit for 2019 about women's suffrage, so we were able to sell two courses of professional development work to improve articles related to women's suffrage using NARA resources.

In addition to our FDC grant, the match of the FDC grant from the Stanton Foundation, and our January donation from the Pineapple Fund, we also received major support for our Communicating Science initiative from the Heising-Simons Foundation and the Moore Foundation. We also received a renewal and increase on our grant from the Hewlett Foundation.

Programs[edit]

2018 was our most successful year ever in terms of impact to Wikipedia. Our programs added 14.2 million words to Wikipedia — 2 million more words than we added in 2017. In early 2018, we also reached a major organizational milestone: Contributions to English Wikipedia by participants in Wiki Education's programs have now eclipsed the final print edition of Encyclopædia Britannica. The entire last print edition of Britannica contained 44 million words in 32 volumes; from 2010 to 2018, student editors in our Student Program have added more than 53 million words to Wikipedia.

The Wikimedia Foundation Audiences Metrics & Insights Q1 2018-19 report called out Wiki Education's impact on global new editor numbers on the English Wikipedia.

With nearly 17,000 participants in our programs, we're successfully supporting large numbers of new Wikipedians at scale. And we're making a difference in the global numbers; according to the Wikimedia Foundation's Audiences team, we bring in 19% of all new editors on the English Wikipedia — that's 9% of all new editors in all projects globally each year joining the Wikimedia movement from Wiki Education's programs.

As the slide from the WMF Audiences team's deck pictured at right demonstrates, we're making such a big impact on the new editors on the English Wikipedia that in early 2018, we actually brought in more new editors than every other source of new editors to the English Wikipedia combined. For years, a major focus of the Wikimedia Foundation and its affiliates has been on bringing in new editors; Wiki Education takes particular pride in our many-years effort to demonstrate impact on the new editor numbers at scale, which several staff members have been steadily working toward since 2010. We look forward to continuing this trend in 2019.

Student Program[edit]

Note: In late 2018, as part of the organizational reorganization mentioned in the Fundraising section above, we renamed our programs to better distinguish the target audiences. In our proposal, this program was named the "Classroom Program" but is now known as the "Student Program"; we are using its current name throughout this section.

Quantitative targets[edit]

Measure of success Type Goal Actual Explanation
New discipline-specific handouts developed Input 4 4 We published new discipline-specific handouts on science communication, LGBT+ studies, cultural anthropology, and art history.
New academic association partnership agreements signed Input 2 0 We did sign a partnership with the Deep Carbon Observatory focused on another of our programs.
Academic association conferences attended to recruit instructors Input 15 17 We attended two more than anticipated.
Number of students Output (participants) 16,000 16,696 We anticipated supporting 16,000 students, and came in right above target.
Number of newly-registered students Output (participants) 15,500 15,555 We set 15,500 new participants as our goal, and met it.
Percentage of students who identify as women Output (participants) 60% 57% The slight drop in the percentage of women compared with 2017 is likely a result of our shift away from supporting large lecture courses. In 2017 there were several large course with very high percentages of women students.
Number of articles edited Output (direct product) 19,200 16,394 Our goal was 19,200, which we missed by about 2,800 articles. We attribute this to a change in programmatic strategy where we stopped supporting new "small projects" as we determined these did not add high enough quality content to Wikipedia to justify our staff time. That meant we pushed participants into spending more time working on one article in depth instead of small contributions to multiple articles.
Amount of content added Impact (short term) 11.2 million words 13.27 million Due to the shift away from small projects, our contributors added more content to fewer articles, meaning we significantly exceeded our 11.2 million goal for words added.
Number of articles edited with at least 10 ORES point improvement Impact (short term) 5,000 4,497 We missed our goal of 5,000 by about 500. See below for a discussion of how we adjusted this metric for next year.
Instructor satisfaction rate (percentage who indicate interest in teaching with Wikipedia again) Impact (short term) 98% 97% While we missed our goal by 1%, we still feel this survey result speaks well of our work.
New student learning outcome study completed Impact (long term) 1 0 We eliminated this goal due to the decrease in funding from the FDC.

What we did[edit]

Ten faculty members who have taught in our program attended Wiki Conference North America, enabling us to lead a focus group to gather learnings about the program.

In 2018, we developed a strong focus on the quality of courses we are bringing in to the Student Program. Several terms ago, we developed a formula to assign a numerical "course rating" to each course that participates in our program. The formula takes both quantitative as well as qualitative factors into account. How much content students contribute as well as the quality of that content and the amount of staff time spent on the course all factor into a course's end of term rating. We separate our returning instructors into different buckets, and focused our biggest retention efforts on the courses with the highest ratings. We no longer do active outreach to instructors with low-rated courses. If an instructor with a previously low-rated course wishes to run another Wikipedia assignment, staff discuss whether the course is viable and then facilitate a discussion with the instructor to improve the course's contributions to Wikipedia. This year, we further refined our metrics around course ratings, and we are beginning to see positive results. As part of this quality effort, we also discontinued supporting new courses that made only minor edits to Wikipedia, and we discouraged large multi-section courses from participating at all.

To enable us to more easily make updates to content delivered to instructors and students, we undertook a complex revamp of our course page and training content on our Dashboard software. The outcome is that if we need to make a text change (for example, if we realize some way we've described how to do something is confusing to students), we can make one edit and have it populate to all students now. We also added four new discipline-specific handouts to provide better support to student editors who are editing in topic areas related to science communication, LGBT+ studies, cultural anthropology, and art history.

Wiki Education Article Finder tool

To better support students and instructors, we also debuted a new Article Finder tool that enables people to locate existing articles within topic areas that might be ripe for improvement, using a combination of Wikipedia's category system, ORES rating, and page views. The tool was developed by Google Summer of Code intern Pratyush Singhal, an indication of how beneficial our open source software interns have been to our programmatic efforts.

As part of our ongoing efforts to create a better program, we also conducted a focus group of instructors at WikiConference North America. Ten of our instructors were attending WikiConference North America, and the organizers were kind enough to find space for us to engage with these faculty and learn from their experiences. Feedback from them and from our end-of-term instructor surveys has helped guide our plans for changes we'd like to make in 2019 to better support our instructors.

Chief Programs Officer LiAnna Davis led a Wikidata Workshop at Wayne State University in December 2018.

In 2018, we also started laying the groundwork for a forthcoming Wikidata program we plan to launch in 2019. As part of our new strategy, we recognize that people are increasingly getting knowledge from personal digital assistants like Alexa or Siri and not from reading long-form encyclopedia articles. Thus, to ensure that our mission of getting representative, accurate, and complete information to people, we are also pursuing programmatic efforts to improve Wikidata content as well. We are exploring a partnership with Wayne State University's School of Information Science to incorporate Wikidata assignments into their library students' curriculum; as part of this work, we led Wiki Education's first Wikidata workshop at Wayne State's campus in December.

What worked well[edit]

In 2018, we had our largest impact to date: We supported 811 courses at 366 different universities in the United States and Canada. All told, more than 16,000 student editors added more than 13 million words to Wikipedia — up from our baseline of 11 million words in 2017. The sheer scale that we're operating on is unparalleled in the Wikimedia movement: The fact that students in this program represent 9% of all new editors to all Wikimedia projects in all languages speaks to how important this program is to our movement. We are unaware of any other Wikimedia movement program that brings in the same numbers of new contributors.

Georgetown University's Jeanine Turner, a longtime instructor in our program, stopped by our booth at the National Communication Association conference to talk her colleagues into teaching with Wikipedia through our program.

This achievement has been many years in the making. We've built up our visibility over time, and in 2018 we reached the milestone of having brought in more new classes from our general visibility work than we did through our targeted outreach. While our targeted outreach is still an important part of our strategy — it enables us to specifically target content areas related to our Communicating Science and Equity initiatives — we don't need to do as much general recruitment in topic areas that aren't of particular strategic importance. That means we were able to reduce the number of academic conferences we attended this year, while still achieving our goals. We count visibility as people who join our program from a recommendation of a colleague, media or blog posts that inspire them, or Googling and finding us. We anticipate attending even fewer conferences in 2019 as the impact from our ongoing visibility work grows.

A key to scalability has been building up our Dashboard software system to provide participants everything they need without interacting with our staff unless they run into a problem. We continued making progress on this in 2018, with an increasing number of instructors finding our website, creating a course page, assigning their students to edit Wikipedia, and empowering the students to add high quality contributions to Wikipedia, without ever interacting one-on-one with our staff. We are scaling the impact of the program without significantly adding staffing resources to it, and succeeding in adding high-quality content to Wikipedia.

What didn't work well[edit]

As mentioned in our midterm report, we had to make two programmatic cuts based on lower-than-anticipated revenue: (1) the student learning outcomes research study, and (2) the Guided Editing project. While we still believe both are important projects and we hope to get them funded at some point in the future, we've been left to conclude now is not the right timing to pitch them to funders, who seem to have other funding priorities.

Additionally, we fell short of our goal for the number of articles improved by 10 points on the ORES scale. As we've had a full year of using this metric now, we still believe it has value in capturing, in a rough way, the improvement people make to articles. However, one challenge is that different edits are rated differently. Simply creating a new stub article only big enough to pass through the New Page Patrol gauntlet provides about 20 points; for an example, see the article on Ethiopian poet Nebiy Mikonnen, written by one of our student editors in fall 2018, which was a 21-point improvement. Another student editor in the same course added the same number of words by making multiple minor improvements to the article about Ugandan entertainer and politician Robert Kyagulanyi Ssentamu, but despite the similarities in amount of content added, the second student's work didn't even merit an increase in ORES score on the article. Thus, it's an imperfect measurement — but we still think it's better than any other currently existing measure of article quality in the Wikimedia world, so we are planning to continue using it in 2019.

Connecting it to our strategy[edit]

Our strategy has three main pillars: Equity, Quality, and Reach. Our programmatic work in the Student Program connects to each pillar in our strategy.

Equity

Our strategic goal for Equity states: "Increase knowledge equity by focusing on content and communities that are underrepresented on Wikipedia and Wikidata".

In our Student Program, we've addressed the content axis by specifically targeting higher education faculty that teach in underrepresented topic areas, such as race, gender, and sexuality. Articles in underrepresented content areas improved by our students during this year include:

  • Karankawa people, an expansion of the article on an extinct Native American group written by a student editor in a University of Southern California course on "Borderlands in a Global Context"
  • Police use of deadly force in the United States, an expansion of an issue of timely interest in the context of racial equity in the United States, written by a student in an Emory University course on "Gender, Race, Inequality - Evidence"
  • Chanda Prescod-Weinstein, an article on a woman scientist expanded and shepherded through the Good Article process by a student editor from Mount Allison University's "Gender and Science" course
  • Ann Brewster, a new article on a female cartoonist and illustrator written by a student in a University of Florida course on "American Women in Comics"

All told, student editors improved nearly 2,000 articles in equity content areas in 2018, adding an impressive 1.6 million words to those articles.

Survey data suggests 37% of our student editors identify as non-white.

Student editors in our program are also more diverse than the general Wikipedia editing population in areas we have demographic data for. While the 2018 Community Engagement Insights Report found 90% of contributors to Wikimedia projects are male, and the median age range of all contributors is 35–44, among our student editor population:

  • 57% identify as women
  • 37% identify as non-white
  • For 13%, English is their second language
  • 5% are adult non-traditional students; the rest are of typical college age (late teens/early 20s)

Quality

Our strategic goal for Quality states: "Provide people who seek knowledge online with accurate information in topic areas that are underdeveloped".

Quality is one of the hardest elements to measure in the Wikimedia universe. While we are eager to pursue a better method for assessing Wikipedia content quality on a regular basis, we are currently limited in terms of our metrics for quality of articles. Nevertheless, as part of our course close-out process, our Wikipedia Experts on staff review student work that comes from this program and provide a general assessment of quality of content that emerged from each course. We prioritize encouraging high-quality content-producing courses to teach with Wikipedia again. We also use the ORES measure of structural completeness as a stand-in for quality, with the caveats mentioned above. In past years, we have undertaken manual quality assessments and found the article improvements made by the students be significant; while we did not undertake these studies in 2018, we have anecdotal evidence that quality has either stayed the same or increased from past years. We are confident that as our onboarding and monitoring systems continue to get better, we are getting better at bringing high quality content to Wikipedia through this program at scale.

Reach

Our strategic goal for Reach states: "Reach large audiences with free knowledge by making Wikipedia and Wikidata more complete".

In terms of Reach, there are three elements this program tackles. The first is developing our infrastructure through tools like the Article Finder mentioned above, that enable us to scale the impact of our programmatic work. The second is page views; the sheer size this program is operating on means articles improved by Student Program participants get millions of page views each term. While the page views each participant brings on average is only 32,000 (compared to 130,000 per participant in our Wiki Scholars & Scientists Program), the sheer volume of participants makes this program the biggest driver of page views in our programmatic stable. Finally, we have laid the groundwork this year to start a Wikidata program in 2019, with the goal of improving Wikidata content used by personal digital assistants like Alexa or Siri.

Visiting Scholars Program[edit]

Quantitative targets[edit]

Measure of success What it measures Goal Actual Explanation
Number of Scholars Output (participants) 15 15 We did not grow the program significantly this year, and met our goal of 15.
Number of blog posts about Scholar work Output (activity) 15 28 We encouraged Visiting Scholars to write guest posts for our blog, causing us to exceed this number.
Find sponsors for qualified Scholar applicants within 6 months Output (activity) 60% 67% We met this goal.
Number of articles edited Output (direct product) 680 1,002 We exceeded this goal mostly thanks to two very productive editors.
Amount of content added Impact (short term) 810,000 words 621,460 words We fell short of this goal, likely due to the fact that we did not devote much time to encouraging existing participants beyond just providing them access.
Number of articles edited with at least 10 ORES point improvement Impact (short term) 435 293 Because we had less activity, we also fell short of this goal.

What we did[edit]

Rosie Stephenson-Goodknight, shown here during a visit to Wiki Education's offices with staffers Helaine Blumenthal and Cassidy Villeneuve and fellow Wikimedian Camelia Boban, was our most productive Visiting Scholar in 2018.

We placed two new Visiting Scholars in early 2018. However, taking FDC feedback into account as discussed in our mid-term progress report, we decided to decrease emphasis on this program in the second half of 2018. We no longer proactively pushed this program, instead moving it into reactive mode. Our existing Visiting Scholars have continued to edit, but as their tenures have come to an end, we haven't pushed to find a new candidate for the position, or a new opportunity for that Scholar unless they specifically ask for one. Instead, we focused the staff attention that had previously been devoted to this program to the Wiki Scholars & Scientists program, where we're able to make up the numbers we expected to get from the Visiting Scholars Program in a more scalable fashion.

What worked well[edit]

Our existing Visiting Scholars continued to produce excellent content on Wikipedia, which is why we think this program is worth continuing in reactive mode for the near future. Northeastern University Visiting Scholar Rosie Stephenson-Goodknight (User:Rosistep) was our most productive Visiting Scholar; she was responsible for more than half of all the words added by Visiting Scholars in 2018, and 2/3rd of all the ORES+10 articles. George Mason University's User:Wehwalt was our major quality juggernaut, bringing an impressive total of 10 articles to Featured Article status; the rest of the Visiting Scholars brought an additional 5 articles to Good Article status.

What didn't work well[edit]

This program simply didn't scale; while the rockstars like Rosiestep and Wehwalt make extensive contributions, others make more minor contributions, and the outcomes simply didn't justify the expense of staff time devoted to the program. As such, we've moved the program into reactive mode; we expect the numbers to dwindle in the coming years.

Connecting it to our strategy[edit]

Equity

Our strategic goal for Equity states: "Increase knowledge equity by focusing on content and communities that are underrepresented on Wikipedia and Wikidata". While not all of our Visiting Scholars are working on equity, three of our active Visiting Scholars are: Rosiestep (discussed above), who focuses on women writers; User:Jackiekoerner, who focused on disability studies; and User:Owlsmcgee, who focused on ethnic studies.

In 2018, the topic of algorithmic bias was in the news a lot, especially in relation to the European Union's General Data Protection Regulation, but until Owlsmcgee created the article as a stub in 2017 as part of his Visiting Scholar position, Wikipedia didn't have coverage of it at all. In 2018, he put a significant amount of work into the article, bringing it up to Good Article status. Highlighting the social implications of technology, especially in relation to systemic bias issues, is an important dimension to equity.

Jackiekoerner tackled the article on ugly laws in 2018, which used to be used to ban "unsightly" people from appearing in public. Jackie expanded the article from a stub that only talked about ugly laws in the United States into a solid article, adding historical information from several countries and documenting criticism, representation, and the impact of them. Coverage of disability topics on Wikipedia can be sparse, and improvements like this improve underrepresented disability-related topics.

In 2018, Rosiestep edited 388 and created 136 articles as part of her Visiting Scholar role at Northeastern University, with the majority of them focused on women writers. Improving coverage of women on Wikipedia is obviously an important element of equity, and one that Rosie has been passionate about for years; access to sources through Northeastern allows her to better source her contributions.

Quality

Our strategic goal for Quality states: "Provide people who seek knowledge online with accurate information in topic areas that are underdeveloped".

The Visiting Scholars Program excels at adding quality content, with 15 Good and Featured Articles achieved thanks to the access to sources provided by this program. 10 Featured Articles, all by Wehwalt, Visiting Scholar at George Mason University:

Good Articles:

Reach

Our strategic goal for Reach states: "Reach large audiences with free knowledge by making Wikipedia and Wikidata more complete".

Many of our Visiting Scholars contributed to high-traffic articles, making substantive improvements to information people access regularly. In 2018, Deep Carbon Observatory Visiting Scholar User:RockMagnetist significantly expanded the scientific information in the article on diamonds, which averages about 3,000 views each day. He also added sections about modes of operation to the fermentation article, which gets about 1,500 views each day. Many of Wehwalt's high-quality articles also get a lot of traffic; for example, his Featured Article on James K. Polk averages 2,300 page views each day. And User:Rosiestep not only improves coverage of women writers on Wikipedia; she also improves their Wikidata entries and adds images of them to Wikimedia Commons. Through their contributions to these well-read topics, Visiting Scholars help improve reach.

Wiki Scholars & Scientists Program[edit]

Note: In late 2018, as part of the organizational reorganization mentioned in the Fundraising section above, we renamed our programs to better distinguish the target audiences. In our proposal, this program was named the "Wikipedia Fellows Program" but is now known as the "Wiki Scholars & Scientists Program"; we are using its current name throughout this section.

Quantitative targets[edit]

Measure of success What it measures Goal Actual Explanation
Number of academic associations signed on to participate Output (participants) 5 9 Associations added: American Anthropological Association, American Chemical Society, American Sociological Association, Association for Psychological Science, Association for Women in Mathematics, Linguistic Society of America, Midwest Political Science Association, National Communication Association, and National Women's Studies Association.
Number of Scholars & Scientists Output (participants) 20 183 Since we put significantly more effort into this program than anticipated, we far exceeded our goal of 20.
Number of articles edited Output (direct product) 80 632 Similarly, we exceeded our goal of 80.
Amount of content added Impact (short term) 24,000 words 316,012 words Similarly, we exceeded our goal of 24,000.
Number of articles edited with at least 10 ORES point improvement Impact (short term) 40 145 Similarly, we exceeded our goal of 40.

What we did[edit]

Wiki Scholars participants in our fall 2018 humanities course, which took place using Zoom.

At the beginning of 2018, we piloted the Wiki Scholars & Scientists program, in which subject matter experts learned to contribute content to Wikipedia through a weekly hour-long synchronous three-month course. Three academic associations joined our efforts for the pilot phase, and it was a wild success: All nine Wiki Scholars finished the course, making meaningful contributions to Wikipedia. This led us to the conclusion that the program had potential to have impact, was more likely to scale than the Visiting Scholars program, and was worth investing more staff time in.

In summer 2018, we hosted six courses, raising our number of participants from 9 in the first course to a total of 99 new editors. In fall, we hosted another set of seven courses, with a total of 75 participants. We used these cohorts to test variables, including group size, course length, and scheduling. Our learnings from these courses are documented in an extensive evaluation on Meta.

In fall 2018, as part of our fee-for-service model development, we rebranded this program as Wiki Scholars and Scientists. We hosted our first course with NARA, focused on improving articles related to women's suffrage in anticipation of a forthcoming NARA Museum exhibit about the 100th anniversary of the 19th Amendment to the U.S. Constitution, which granted women the right to vote, and the ongoing suffrage challenges women of color suffered. An anonymous donor agreed to offer partial scholarships for the course, enabling us to offer the course for a smaller fee to individual participants. When our final Wiki Scholars & Scientists course wrapped up in December 2018, we stopped offering this program for free, and instead offered it for a fee, with the option for an individual or foundation donor to offer a scholarship for participants.

What worked well[edit]

Zoe Brigley Thompson and Amy Dye-Reeves, two Wiki Scholars & Scientists participants, joined Program Managers Will Kent and Ryan McGrady for a presentation at Wiki Conference North America 2018.

A full documentation of what worked well is part of our full evaluation report on Meta. Overall, we were very pleased with the outcomes from courses we led in this program. While our Student Program works at scale to make a huge impact across many disciplines on Wikipedia, the Scholars & Scientists program enables us to engage subject matter experts to make updates to high-profile articles that are otherwise difficult to edit unless one has both a deep grounding in the subject matter and a deep grounding in Wikipedia. All of the participants in this program have the deep subject matter expertise, but they are lacking the Wikipedia portion; we provide this to them through the program.

One surprising finding for us is that 21% of the people who took our courses had previously registered a Wikipedia account. We had anticipated this program would be exclusively for new editors, and thus they would be registering their account during our course. But 39 of our participants had previously registered an account and attempted but failed to make some edits that stuck; none were currently active editors. Editing the English Wikipedia as a newbie isn't easy, and our participants found that they were able to actually effectively contribute after taking our course, in a way they had not been able to in just trying to figure it out on their own. We find this metric very telling and a success for our program: We captured people who had tried in the past and given up, until they joined our program, when they finally were able to successfully contribute.

What didn't work well[edit]

We tested three variables: size of course, course length, and scheduling.

Size of course:

  • We tested a large course of 50 participants. While we had enough applicants and invited 50 people to participate, only 36 even made it to the point of creating Wikipedia user accounts, and 10 of those never touched the article namespace. We received feedback that the group was too diverse to have expert-level conversations and to have everyone contribute during our meetings in meaningful ways. We determined smaller cohorts are better.
  • On the other hand, we also tried a small cohort of 6 people. That quickly became a challenge if a few people had to miss one week's session, leaving the group without enough critical mass to build community. We determined larger cohorts are better.
  • Given these two findings, we recommend cohorts of 10–15 people.

Course length:

  • Our initial pilot was 12 weeks long, modeled after the best practices from our Student Program. Pilot participants suggested another month would be helpful, so we tested a 16-week course. But we found that gave participants too much time to forget about the course, leading participants to stratify in terms of progress. At the end of the course, they had a similar number of contributions to our 12-week test group, meaning the additional staff time we devoted to the course did not result in more impact to Wikipedia.
  • We also tested an 8 week course. Since many of our participants are academics, various academic conference and committee demands meant everyone was likely to miss a week or two. These absences were particularly detrimental for a course of only 8 weeks.
  • Given these two findings, we recommend the original course length of 12 weeks.

Scheduling:

  • We had a few courses' schedules straddle both summer months and the start of the fall academic term. Since many of our participants are academics, their schedule availability changed midway through the course, meaning there was significant dropoff in September when the term started. Holidays also proved challenging.
  • We recommend trying to schedule with the academic calendar in mind.

These and additional learnings are all documented in our full evaluation report on Meta.

Connecting it to our strategy[edit]

Equity

Our strategic goal for Equity states: "Increase knowledge equity by focusing on content and communities that are underrepresented on Wikipedia and Wikidata".

Similar to the Student Program, our Wiki Scholars & Scientists program addresses equity from both the content and the contributors perspective. The Wiki Scholars & Scientists Program enables us to specifically create courses aimed at improving equity topic areas on Wikipedia. In 2018, we had several program participants who improved topics related to race, gender, and sexuality, including:

  • Two courses in collaboration with NARA to improve coverage of women's suffrage on Wikipedia, which led to improvement of 60 articles including articles on suffragists like Mary McHenry Keith and Ida B. Wells and suffrage topics like Prison Special and Expatriation Act of 1907.
  • Two courses specifically devoted to creating biographies of women scientists, which led to creation of 45 new biographies of women in science and improvement of 99 total biographies of women in science, including biographies of Letitia Obeng, Tanja Bosak, and Darlene Lim
  • Scholars in other courses improved articles related to equity issues, such as the article on Hometown association, an important immigration-related topic.

Contributors to the Wiki Scholars & Scientists Program are also more diverse than the general English Wikipedia editing population. According to survey data from 2018:

  • 78% of participants identify as non-male
  • 31% of participants identify as non-white
Unsurprisingly, women were motivated to participate in our cohorts aimed at improving coverage of women in science on Wikipedia.

Quality

Our strategic goal for Quality states: "Provide people who seek knowledge online with accurate information in topic areas that are underdeveloped".

So far, quality has been a big driver of this program. Engaging subject matter experts to improve topics that would otherwise be challenging to improve means Wiki Scholars & Scientists participants can make a real difference to articles that have languished in mediocrity for many years. For example, participants have improved articles such as authority, protest vote, Religious Liberty Accommodations Act, Media transparency, and membrane curvature. Sometimes they create new important articles, such as the article on feminist poetry. And Wikipedians notice their work; the participant who expanded the collective memory article had this message left on their talk page: "I tried to thank you for all the good work you are doing, but the wiki thank feature applies only to single edits." We expect as we grow this program, we will be able to maintain the high quality output; our screening process for program participants brings in subject-matter experts who make high-quality contributions.

Reach

Our strategic goal for Reach states: "Reach large audiences with free knowledge by making Wikipedia and Wikidata more complete".

Unsurprisingly, this program melds very well with the Reach goal. Because we work individually with participants in a course setting, we are able to guide them through the process of editing Wikipedia articles that get high traffic but are lacking in quality. The examples listed above in the quality section illustrate both quality and reach — many are high-traffic articles that were significantly improved by our participants. In the Wiki Scholars & Scientists Program so far, participants each improved articles that get an average of 130,000 page views during the course — about four times the views compared with the typical Student Program article.

Revenues received during this period (6 month for progress report, 12 months for impact report)[edit]

Please use the exchange rate in your APG proposal.

Table 2 Please report all spending in the currency of your grant unless US$ is requested.

  • Please also include any in-kind contributions or resources that you have received in this revenues table. This might include donated office space, services, prizes, food, etc. If you are to provide a monetary equivalent (e.g. $500 for food from Organization X for service Y), please include it in this table. Otherwise, please highlight the contribution, as well as the name of the partner, in the notes section.
Revenue source Currency Anticipated Q1 Q2 Q3 Q4 Cumulative Anticipated ($US)* Cumulative ($US)* Explanation of variances from plan
Individuals USD $30,000 $554,000 $7,000 $5,958 $9,896 $576,854 $30,000 $576,854 We received a large unanticipated donation from the Pineapple Fund in January.
Foundations: FDC USD $750,000 $233,333 $0 $164,641 $2,026 $400,000 $750,000 $400,000 The FDC recommended $350,000 less than our request.
Foundations: Stanton USD $400,000 $240,833 $0 $164,641 $0 $405,474 $400,000 $405,474 As expected.
Foundations: Other USD $1,945,000 $0 $100,000 $477,000 $497,500[1] $1,074,500 $1,945,000 $1,074,500 $500K of this difference was that the Pineapple Fund gift was classified as an individual donation; we also reduced spending when it was clear we were going to miss this target.
Total USD $3,125,000 $1,027,000 $107,000 $812,240 $509,422 $2,455,662 $3,125,000 $2,455,662 We reduced spending to accommodate for our reduced revenue.

* Provide estimates in US Dollars


[1] $485,000 of this is a restricted grant that has a grant period of November 2018 to October 2019, so the majority of the grant will be spent in 2019.

Spending during this period (6 month for progress report, 12 months for impact report)[edit]

Please use the exchange rate in your APG proposal.

Table 3 Please report all spending in the currency of your grant unless US$ is requested.

(The "budgeted" amount is the total planned for the year as submitted in your proposal form or your revised plan, and the "cumulative" column refers to the total spent to date this year. The "percentage spent to date" is the ratio of the cumulative amount spent over the budgeted amount.)
Expense Currency Budgeted Q1 Q2 Q3 Q4 Cumulative Budgeted ($US)* Cumulative ($US)* Percentage spent to date Explanation of variances from plan
Student Program USD $828,183 $199,476 $166,342 $147,564 $184,826 $698,207 $828,183 $698,207 84% We spent less on this program since our revenue was less than anticipated.
Visiting Scholars Program USD $225,920 $52,742 $54,002 $4,474 $6,288 $117,505 $225,920 $117,505 52% Due to FDC feedback, we shifted focus from this program to Wiki Scholars & Scientists.
Wiki Scholars & Scientists Program USD $223,382 $55,590 $60,708 $92,578 $137,369 $346,245 $223,382 $346,245 155% Since our pilot was promising, we invested more in this program.
Guided Editing USD $583,050 $0 $0 $0 $0 $0 $583,050 $0 0% We did not receive the Guided Editing grant as anticipated, so we did not spend anything on the project.
Operations, Fundraising, and Governance USD $906,472 $196,312 $194,622 $193,019 $207,307 $791,260 $906,472 $791,260 87% We spent less since our revenue was less than anticipated.
TOTAL USD $2,767,007 $504,119 $475,674 $437,635 $535,790 $1,953,218 $2,767,007 $1,953,218 71% Without Guided Editing, our budget was $2.2 million, meaning we spent 89% of our budget in 2018.

* Provide estimates in US Dollars


Compliance[edit]

Is your organization compliant with the terms outlined in the grant agreement?[edit]

As required in the grant agreement, please report any deviations from your grant proposal here. Note that, among other things, any changes must be consistent with our WMF mission, must be for charitable purposes as defined in the grant agreement, and must otherwise comply with the grant agreement.

  • All changes were described above in Programs section.

Are you in compliance with all applicable laws and regulations as outlined in the grant agreement? Please answer "Yes" or "No".

  • Yes

Are you in compliance with provisions of the United States Internal Revenue Code (“Code”), and with relevant tax laws and regulations restricting the use of the Grant funds as outlined in the grant agreement? Please answer "Yes" or "No".

  • Yes

Signature[edit]

Once complete, please sign below with the usual four tildes.

Resources[edit]

Resources to plan for measurement[edit]

Resources for storytelling[edit]