Grants:Programs/Wikimedia Community Fund/Rapid Fund/Examining how Professional Academic Organizations engage with Wikimedia (ID: 22278145)/Final Report

Examining how Professional Academic Organizations engage with Wikimedia

Rapid Fund Final Report

Report Status: Accepted

Due date: 07 October 2024

Funding program: Rapid Fund

Report type: Final

This is an automatically generated Meta-Wiki page. The page was copied from Fluxx, the web service of Wikimedia Foundation Funds where the user has submitted their midpoint report. Please do not make any changes to this page because all changes will be removed after the next update. Use the discussion page for your feedback. The page was created by CR-FluxxBot.

General information

Applicant username: BrettButtliere
Organization name: N/A
Amount awarded: 4947.61
Amount spent: 4954.48 USD, 20784.22

Part 1: Project and impact

1. Describe the implemented activities and results achieved. Additionally, share which approaches were most effective in supporting you to achieve the results. (required)

The rapid grant involved assessing how Academic professional organizations are or were engaging with Wikimedia, especially in terms of e.g., organizing edit-a-thons, hosting wikimedians in residence, and running any other sorts of trainings or activities. Indeed the point of the project was to identify the best and most common activities and how we might help. Method In order to do this we identified a large scale database of professional organizations and societies, in particular the Encyclopedia of Associations (Gale, 2024), both U.S. and International editions, which provides a list of professional organizations, and which is the premier list containing over 32,000 organizations internationally and 23,800 in the U.S. In order to do a worldwide search, while also keeping the focus on the US, we sampled 100 US based professional organizations and 100 non US based organizations. We decided to limit our organizational search to those with more than 10,000 members and a budget of more than 1 million dollars, to ensure that the organizations we studied were both sizable and significant. Sample We identified 100 US based and 100 non-US based organizations associated with science and technology (compared to e.g., trade organizations) in the database and randomly selected 100 of each by assigning numbers between 0 and 1 to each, ordering them by this number, and then selecting the smallest, closest to 0, 100 organizations. Thus, this should be considered only a small sample of the overall landscape. Procedure Having the list of organizations to examine, we went to each of the websites of the organzation (identified through a search engine search) and searched for Wikimedia and Wikimedia in the search bars for the website. Thus, we were able to identify all activities that these organizations posted about within the domain of their website. Importantly, we did not check other social media sites, except as they were linked from the based web page of the website. As the project progressed, we noticed a lack of engagement and so also decided to examine the Wikipedia pages of the organizations to see whether they were well represented or not. The data were collected in Summer 2024, with a discussion after the first ~20 to discuss the low participation rate and ensure that we were collecting all the variables that we might want. Results Our datasets can be found here, and include many variables, for instance the address of the professional organization, its phone number, the organization’s email, homepage, facebook, instagram, youtube, and any other links the organization provided. These data at least potentially could be used in the future to actively recruited from and then compared to another random 100 organizations The most relevant variables for the analysis are the Comments, where we summarized what we found and put any additional links or interesting things that could be usefully looked at. This was done because the Probably the headline result is that the vast majority of professional organizations do not mention Wikipedia/ Wikimedia in their domain at all. Approximately ~15% have some mention, which could be considered a lot, but most of them are simply citations, and we estimate that less than 10% of professional organizations have mentioned or actually taken part in any Wikimedia related event. Most websites did not contain any mentions. An example of a relatively well integrated source was the International Council of Museums (ICOM) is a standout in having 212 articles, some of which are related. Here we found mentions of, International Museum Day (IMD) across multiple years from 2020 to 2022, and many citations to Wikipedia as a data source.

Results for U.S. Organizations Our analysis of U.S. organizations demonstrates that a majority of these are represented in English Wikipedia (84%), despite the fact that only 3% indicate some active engagement with Wikimedia projects. Those that show engagement include: Association for Computing Machinery (ACM), the American Institute of Physics (AIP), and the American Speech-Language-Hearing Association (ASHA). Upon further examination of the Association for Computing Machinery (ACM), we found mention of a Wikipedia project to increase public understanding of logic in ACM in 2010, but could not identify any evidence of sustainability. A seemingly separate initiative is mentioned in a 2023 report by ACM Europe council which mentions a Wikipedia Working Group led by Araz Yusubov" - "We are currently building a new team to improve the visibility of European women in computing on Wikipedia. Araz Yusubov, (see: https://www.acm.org/binaries/content/assets/about/annual-reports/acm-europe-fy23.pdf). The American Institute of Physics (AIP) website also showed evidence of active engagement of membership, most notably in the organization and hosting of an Edit-a-thon to raise awareness about black physicists (see: https://ww2.aip.org/news/american-institute-physics-host-wikipedia-edit-thon-raise-awareness-black-physicists) ASHA, or the American Speech Language Hearing Association, returned 64 results in a search for Wikipedia. Much of these results related to research that used Wikipedia as a source or a subject rather than contributor initiatives. However, one result did encourage members to engage in another association’s Wikimedia campaign: NIOSH (National Institute for Occupational Safety and Health) dedicated to sound health (See: bit.ly/wiki-sound-2020). Two additional findings our analysis highlighted were 1) the prevalence of organizations using images from Wikimedia Commons and/or content from Wikipedia articles (using Wikipedia articles as sources) and 2) that a small number of the organizations sampled had either linked to or created their own open platforms for sharing knowledge related to their organization, which also appeared in search results due to the fact that they were named “wikis” or used MediaWiki software. These organizations include the Society of Petroleum Engineers (SPE), whose “PetroWiki” seems to borrow some norms from Wikipedia (See: https://petrowiki.spe.org/PetroWiki), and the American Academy of Ophthalmology (AAO) which has founded EyeWiki (https://eyewiki.org/Main_Page). Also among these results were also organizations that provided members information about other wikis they themselves did not create, but which were similar in topic. The Emergency Medicine Residents' Association (EMRA), for example, links to the WikiEM - The Global Emergency Medicine Wiki, (though not created by this organization, See: https://wikem.org/wiki/Main_Page). In summary, it is significant that the majority of these organizations enjoy coverage and representation on English wikipedia, and that many of them also utilize content (in the form of written or visual information), while very few engage their members in any contribution initiatives. Furthermore, organizations that have founded their own open knowledge platforms are notable in that they have made contributions to the wider digital commons. Results for International Organizations Looking at the results for the international organizations, 92% of them have Wikipedia websites. Conversely, only 42% of the organizations have any mentions of Wikimedia/ Wikipedia on their websites. Looking at those international organizations that do mansion Wikipedia/ Wikimedia, 15% of the organizations mention it in a substantive way beyond as a citation. Only 3 organizations of the 100 mention Wikimedia more than 100 times on their website, and these are mostly as citations for e.g., pictures and links to get more information. 48% of the organizations that contain wikimedia do so within the context of citations only. Refers to Wikipedia as a source of information and pictures – 20% of all IO cite, reference wiki articles or pictures. 48% out of the IO mentioning Wiki (42% group). Only 10% of the International Organizations overall mention wikimedia in a way that is substantial and beyond only citations for pictures. Approximately 20% of the International Organizations do use Wikimedia as a source of information, which can be seen as quite good but obviously still in need of improvement. Notably, only one International Organization has a Wikipedia mention in Qs1&2 2024. Mentions of Specific Wikimedia related events in International Organizations We found specific mentions of ‘International Women’s Day Wikipediathon’ at Royal College of Surgeons in Ireland (RCSI), Science and Technology Australia (STA), and European Physical Society (EPS). This is the most popular activity with 3 of the 100 organizations mentioning it, which is not that bad given that most of the organizations had 0 relevant mentions. This speaks to the success of the International Women’s day campaign. Probably this will be an organization to interview and learn from in terms of best practices. Conseil International des Monuments et des Sites (ICOMOS) is an example of an organization which has engaged in several Wikimedia events, Other mentions we encountered include, ‘International Day of Monuments and Sites’ at the Conseil International des Monuments et des Sites (ICOMOS) ‘Physiological Society Wikipedia Editathon’ at Royal Society of Biology (RSB), and …. Overall, the impression is that very few of the organizations in our dataset had actually engaged with or at least posted about Wikipedia on their websites. There is of course the potential that they engage but do not post it, but we consider logging the activity as some minimum threshold in terms of the importance that the group gives to the activity. Seven percent of the international organizations mentioned Wikipedia or Wikimedia, as mentioned above, with three of the 7 being for the International Women’s day.

Conclusion: Nearly all organizations have a Wikipedia page, but most do not engage. While we did not focus on the organization or quality of the Wikipedia article, we were able to confirm presence and collect links for 92% of the International Organizations and 83% of the U.S. Organizations represented on English Wikipedia. We were surprised to find that there would be more International Organizations represented on English Wikipedia, but hypothesize that either 1) this is likely a result of the sample limits or 2) the notability of International Organizations compared to some of the U.S. counterparts. Discussion The major take away for Wikimedia will be what it wants to do to get better results on such a test. There are obviously more events than this, but it is not that effectively translating into especially self motivated activity on their professional webpages. International Women’s day should be looked at as a model of success so far. The main result that we found was that essentially every organization had a page on Wikipedia, but that very few of them featured Wikimedia activities or engagement on their website. Less then 10% (of the international organizations) had engaged, with most being citations especially for pictures. Limitations Examining professional organization’s coverage and involvement in Wikimedia projects in a systematic way does present certain challenges. Our results were limited due to particular limitations with our search methodology. For example, many professional organizations have both a public-facing, accessible website, as well as members-only content. Because the researchers were not members of any of these organizations, we were not able to assess this type of content for mentions of WIkipedia. Additionally, our sample selection relied on membership size and financial stability. While these are useful indices for choosing influential professional organizations, they don’t account for how some organizations might be more disposed to contribute to Wikipedia based on their mission or agenda. However, we are confident in the general findings - especially as it relates to the overall lack of involvement of professional organizations in systematic engagement with Wikimedia projects.

2. Documentation of your impact. Please use space below to share links that help tell your story, impact, and evaluation. (required)

Share links to:

Project page on Meta-Wiki or any other Wikimedia project
Dashboards and tools that you used to track contributions
Some photos or videos from your event. Remember to share access.

You can also share links to:

Important social media posts
Surveys and their results
Infographics and sound files
Examples of content edited on Wikimedia projects

This Rapid Grant paved the way both for the Research grant that we recently received this year, as well as the EU level COST Action grant that we are preparing for strengthening the Academic Wikimedia bond. This application is going to be successfully submitted, and it has 17 authors from 10 different nations supporting it and more science on Wikimedia. These projects come directly out of the networking, motivation, and data that came out of this Rapid grant. A major promise of the Rapid grant that we can say we successfully completed was obtaining additional and larger grant money to sustain the project, as well as significant buy-in from the community and even on top of this hopefully longer term funding. In June 2024 we received a Research Grant to support further research in this area, especially in relation to the creation of Wikimedia Impact Metrics and the Wikimedia Impact Tracker. This grant was for 42,000$ and also has Sage Ross on helping with the tool creation. https://www.iup.edu/languageliteraturewriting/news/2024/05/vetter-awarded-research-grant-from-wikimedia-foundation.html https://meta.wikimedia.org/wiki/Grants:Programs/Wikimedia_Research_Fund/Developing_Wikimedia_Impact_Metrics_as_a_Sociotechnical_Solution_for_Encouraging_Funder/_Academic_Engagement This rapid grant also led to a large network of researchers working in the Wikimedia space to come together around the idea of doing a COST Action, which is an European Commission grant for establishing networks and developing research and policy priorities at the EU level. Once again, the initial success with the rapid grant can and should be considered as directly feeding into to these actions. https://docs.google.com/document/d/19nqb-5WDQYSpvBf1620BuWHQ0746fGdRK48Q2mGwAUM/edit This work was a result of the many meetings and discussions at Wikimania 2024, and it is leading to some real formation of networks and working groups taking place now. We basically killed ourselves meeting people and getting this network going to push for larger grants. Here you see a picture of Matt and I at the concluding ceremony looking properly tired after an excellent week of meeting people working toward these same goals as us. https://www.facebook.com/photo/?fbid=761483780022&set=a.512169532772 Another result of this initial Rapid grant was the development of the survey of researchers that is currently running and has approximately 65 participants at this moment, concerning how we can build this wikimedia-academia link. This is a part of the Research grant that we got but it also in part relies on the findings from the rapid grant. An additional potential outcome of the grant was the development of some initial version of the Wikimedia Impact Vizualizer, which was developed by our teammate Sage Ross and the WikiEducation as a part of the research grant mentioned above. https://wikimania.eventyay.com/2024/talk/U7EMQW/

Additionally, share the materials and resources that you used in the implementation of your project. (required)

For example:

Training materials and guides
Presentations and slides
Work processes and plans
Any other materials your team has created or adapted and can be shared with others

The dataset that we generated can be found here. https://docs.google.com/spreadsheets/d/1Rve4eJyjLlf1s69wPKVJWuu4AKCxzltTyzqcaHn4j7M/edit?gid=549882096#gid=549882096

Our presentation slides where we presented, at Wikicon North America, can be found here. https://docs.google.com/presentation/d/1v6ZkWklFkdSd8Pocslg8uFx724AXkG8gw0IeeHk8X3g/edit#slide=id.g3042fd80c4b_1_82

The COST Action proposal can be seen here. https://docs.google.com/document/d/19nqb-5WDQYSpvBf1620BuWHQ0746fGdRK48Q2mGwAUM/edit?tab=t.0

3. To what extent do you agree with the following statements regarding the work carried out with this Rapid Fund? You can choose “not applicable” if your work does not relate to these goals. Required. Select one option per question. (required)

Our efforts during the Fund period have helped to...
A. Bring in participants from underrepresented groups	Strongly agree
B. Create a more inclusive and connected culture in our community	Agree
C. Develop content about underrepresented topics/groups	Agree
D. Develop content from underrepresented perspectives	Agree
E. Encourage the retention of editors	Agree
F. Encourage the retention of organizers	Agree
G. Increased participants' feelings of belonging and connection to the movement	Agree
F. Other (optional)

Part 2: Learning

4. In your application, you outlined some learning questions. What did you learn from these learning questions when you implemented your project? How do you hope to use this learnings in the future? You can recall these learning questions below. (required)

You can recall these learning questions below: The project is aimed at creating a database not only of activities but the organizers of those activities. The information in this database will give us a good idea of Wikimedia engagement by professional organizations, but hopefully also some ideas about how these activities can be taken to the next level. Specifically, the idea is to focus on identifying the people within these organizations who can be surveyed, activated, and usefully brought together to bring about the further adoption of and engagement with Wikimedia projects by professional organizations. The core learning questions include: 1. What activities have academic professional organizations implemented? 2. How can these activities be amplified and taken to the next level? 3. Who are the people that manage these activities within the organizations? These questions will be answered with a systematic search of professional organization’s websites, looking for the keywords “Wikipedia” and “Wikimedia.” Our team will collect activity level and organization level data concerning what activities are done and how often they have been done. Activities will be logged with links and a categorization schema, and the name of the organization and how many blog posts relate to the keywords will be logged. Additionally, we expect to collect champion level data, that is data about those people who are organizing activities for the organization. Here we expect to log the type of activity and the email address of the individual who organized the activity. These data can be expected to form the basis of a paper on the engagement of professional organizations with Wikipedia, and in the longer term a survey and the engagement of the champions that we find.

Certainly we learned throughout our project, not only about the low level of systematic engagement of professional organizations, but also about how we might improve the situation. In the course of the project we found that there is very much initiative and interest in this work, but also that there is a lack of organization around the issue. This is evidenced by the lack of overall organizational structures which focus on science and getting high quality science onto Wikipedia. We are now in discussions to start an organization like WikiSci, which will be internationally focused and dedicated to getting scientific contributions to Wikimedia. Regarding our three research questions which we raised at the beginning of our project. 1. What activities have academic professional organizations implemented?

If the goal was to identify activities that professional organizations search in through the database search, then in this regard we should say that the project was not that successful. We found only a few references to Wikimedia projects and events, mostly related to Wikimedia Women’s day, and the International day of Museums. This is most likely due to the random selection or organizations from a database that another organization put together. This is certainly not representative of the many initiatives which do exist, and one of the major questions we are coming out of the project with is how can we improve these measures. There are in fact many initiatives, Wikimedians know, but apparently they are either not engaging professional organizations or they are not being reported about. We suspect professional organizations are not being engaged as effectively as possible.

2. How can these activities be amplified and taken to the next level?

In this regard we can suggest the project has been a success, especially with our actions at Wikimedia and the building of this network of researchers who will be looking to develop grants together. I believe this will be important for bringing the resources needed to free e.g., professors and other highly sought after experts from their courses or other obligations that pay them.

In particular, Wikimedia Women is a clear example of success, being the top organization being mentioned on 3 professional organization’s websites. Looking at this organization as a model would be a good move for Wikimedia in general.

3. Who are the people that manage these activities within the organizations? Within the project, the effort was not that successful because we found so little activity on the part of professional organizations. On the other hand, in terms of network building I believe the project was highly successful and created a strong basis of people with which we will build projects into the future. These individuals, who we spoke to, or are academics working in this space, and that are at least on the list to be invited to the COST Action that we are developing can be seen at this link. In this sense we believe the project was a great success, because we have been able to bring together quite a number of researchers, museum workers etc who are interested and this sets a good basis for future work. The database of interested researchers can be seen below. https://docs.google.com/spreadsheets/d/1Hu_8kaCp9rGSYQ6kGyIA_cVvC8IP6362igUnfvPvElE/edit?gid=840667423#gid=840667423

5. Did anything unexpected or surprising happen when implementing your activities? This can include both positive and negative situations. What did you learn from those experiences? (required)

The low engagement of professional organizations was a real surprise to us. We shifted the point of the paper from how many organizations engage and what do they do toward something saying: look how many organizations have never engaged even though nearly all of them have wikipedia pages (see results above). On the other side we have also been quite surprised by how much interest there is in the movement already for something like this, and how quickly the community has taken up the call. Our team size has grown from 2 when we initially submitted this grant to now over 16+ people who are in some way committed to taking part in the COST action that we are putting together.

6. What is your plan to share your project learnings and results with other community members? If you have already done it, describe how. (required)

The project was presented informally at Wikimania (click here to see our handout). The project was also presented at WikiCon North America during a presentation on our broader mission. We intend to send around the project report, as well as to distribute the results informally as a part of the argument for why we need more professional engagement. We are also looking for a short format publication where we can publish this, since it is still quite small, such that we and others can cite it when calling for greater engagement from organizations. One of the major outcomes of our project was the development and submission of the research grant, and this has also resulted in the development of this network and group COST Action. These are direct learnings from this Rapid grant.

Part 3: Metrics

7. Wikimedia Metrics results. (required)

In your application, you set some Wikimedia targets in numbers (Wikimedia metrics). In this section, you will describe the achieved results and provide links to the tools used.

	Target	Results	Comments and tools used
Number of participants	200	200
Number of editors	0	0
Number of organizers	4	16	We did not hire a student for Matt, but throughout the project we brought on many collaborators, we now have 16 authors on the COST Action.

Wikimedia project	Target	Result - Number of created pages	Result - Number of improved pages
Wikipedia
Wikimedia Commons
Wikidata
Wiktionary
Wikisource
Wikimedia Incubator
Translatewiki
MediaWiki
Wikiquote
Wikivoyage
Wikibooks
Wikiversity
Wikinews
Wikispecies
Wikifunctions or Abstract Wikipedia

8. Other Metrics results.

In your proposal, you could also set Other Metrics targets. Please describe the achieved results and provide links to the tools used if you set Other Metrics in your application.

Other Metrics name	Metrics Description	Target
Activities identified	We will search 200 academic association’s websites, but we are unsure of how many activities we will identify and be able to categorize. We expect to find that some large organizations have dozens or hundreds of activities, and while we will log the number of activities, we will cap our activities at 10. Thus we can expect to log up to 2,000 activities.	100
Champions and Emails identified:	Associated with each activity, we hope to identify the organizer of the activity, and their email, whom we hope to engage and organize in further actions. We would like to identify one unique person per organization, but also recognize that certain individuals are especially likely to host multiple events, and we hope that the data can also point us in particularly passionate people whom we might support in the future.	50

9. Did you have any difficulties collecting data to measure your results? (required)

No

9.1. Please state what difficulties you had. How do you hope to overcome these challenges in the future? Do you have any recommendations for the Foundation to support you in addressing these challenges? (required)

Part 4: Financial reporting

10. Please state the total amount spent in your local currency. (required)

20784.22

11. Please state the total amount spent in US dollars. (required)

4954.48

12. Report the funds spent in the currency of your fund. (required)

Provide the link to the financial report https://docs.google.com/spreadsheets/d/1uf20V-Vzmd3LELw25e5tOoffjBACR5tfGcuvH9kkP4M/edit?gid=928458698#gid=928458698

12.2. If you have not already done so in your financial spending report, please provide information on changes in the budget in relation to your original proposal. (optional)

The one change was that Matt did not end up hiring a student, and instead decided to do the work himself. After reaching out to check with our program officer, we decided to give Matt the portion of money that would have gone to his student.

Aside from this there were some minor number differences of a dollar or two related to splitting the money and the currency exchange rate.

13. Do you have any unspent funds from the Fund?

No

13.1. Please list the amount and currency you did not use and explain why.

N/A

13.2. What are you planning to do with the underspent funds?

N/A

13.3. Please provide details of hope to spend these funds.

N/A

14.1. Are you in compliance with the terms outlined in the fund agreement?

Yes

14.2. Are you in compliance with all applicable laws and regulations as outlined in the grant agreement?

Yes

14.3. Are you in compliance with provisions of the United States Internal Revenue Code (“Code”), and with relevant tax laws and regulations restricting the use of the Funds as outlined in the grant agreement? In summary, this is to confirm that the funds were used in alignment with the WMF mission and for charitable/nonprofit/educational purposes.

Yes

15. If you have additional recommendations or reflections that don’t fit into the above sections, please write them here. (optional)

The project was a great success in our eyes, though we were disappointed by the low number of hits we found. This could be due to the random sampling and the huge number of organizations out there that Wikimedia simply cannot engage them all. Or can they? We are working on building expert engagement, and we think it would be a very interesting next rapid or research grant to try to actively engage these professional organizations, or otherwise track how this impact might be changed through some strategic WikiActions.

Review notes

Review notes from Program Officer:

N/A

Applicant's response to the review feedback.

N/A