Research:Beyond the Individual: Community-Engaged Design and Implementation of a Framework for Ethical Online Communities Research
Most people participate in multiple online communities, including activities like commenting on a friend’s blog, editing a Wikipedia page, or asking a question on Reddit. Researchers from both industry and academic institutions may seek to improve these online communities by gathering some of this data or engaging in experiments in these online spaces. However, there are currently no common sets of rules to govern this kind of work and the guidelines that do exist for researchers may be significantly different from the expectations of members of these communities.[1] Evidence of this mismatch can be seen in high profile cases where researchers have violated a community’s standards, values, or expectations.[2][3][4]
Several researcher-driven investigations have generated sets of rules and guidelines for online research. Two prominent ones include Communication Technology Research by The Menlo Report and Ethical Guidelines for Internet Research by AoIR.[5][6] While these provide general guidelines, they were not shaped through engagement with online communities and they may fail to account for the specific practices and considerations of those communities. Many mature online communities have established their own standards of research pages and policies out of prior experiences with research (see Research:Committee), but not all communities have the capacity to invest time into developing these practices. The investigators of this project will work with members and organizers of four representative online communities (Wikipedia, CaringBridge, InTheRooms, and r/AskHistorians) to create and disseminate an ethical framework to guide future research in online spaces. Notably, we aim to learn from the successful policies developed in specific communities and generalize those practices, where applicable, to the broader online and research community.
Benefits for the Wikimedia community
[edit]Because the Wikimedia editor and research communities have a much more mature relationship than many other online communities, the benefits of this project to the Wikipedia community will likely be much less than those received by other participating online communities. With that in mind, we believe continually reviewing these practices is important. The research practices and policies identified in other online communities will be shared and disseminated with Wikipedia members and Wikimedia employees during Workshop 3 and may be used to inform future research policies and practices on Wikipedia.
Methods
[edit]Our proposed methods would involve conducting a series of three 2-hour participatory design workshops with various members of the Wikipedia editor community. Each workshop will involve a pre-workshop recruiting/scheduling survey and a post-workshop exit survey. Pre-workshop surveys will give participants the opportunity to individually reflect on the topic questions before discussing them as a group; themes from these asynchronous contributions will be used to prepare synchronous workshop materials. Participants will be compensated for each workshop they participate in. The details of recruitment for each workshop are outlined below.
We are currently seeking participants interested in Workshop 1. Please see the recruitment page for participation instructions.
Workshop 1
[edit]We aim to recruit 6-12 Wikipedia editors. We will follow best practice recommendations from our partners to recruit this small sample size of participants. The workshop's goals involve understanding Wikipedia's community values, as defined by its members, discussing ways research can benefit and harm the Wikipedia community, and discussing practices for engaging in research in the community. Results from Workshop 1 will seed Workshop 2.
Workshop 2
[edit]We aim to recruit 2-6 Wikipedia editors from Workshop 1, 2-4 Wikimedia administrators, and 2 Wikipedia researchers. We'll again follow best practice recommendations and utilize snowball sampling techniques to recruit this small sample size of participants. The goals of this workshop mirror Workshop 1 with the added dynamic of including other stakeholder groups in the conversations. Versions of Workshop 1 & 2 will be concurrently conducted with 3 other distinct online communities and their results will be synthesized.
Timeline
[edit]- 09.2023: Create project page and finish Workshop protocol [done]
- 09.2023: Pilot workshop protocol [done]
- 11.2023: Recruit and schedule Phase 1 Workshop with 6-12 Non-US Wikipedia editors [done]
- 04.2024: Recruit and schedule additional Phase 1 Workshop targeting US-Wikipedia editors [done]
- Recruitment was targeted at US editors, but we still received attention from an international audience.
- 04.2024: Recruit and Schedule Phase 2 Workshop with Wikipedia editors, WMF employees, and researchers [done]
- 08.2024: Synthesize results across all community Phase 1 & 2 Workshops and develop multiple alternative frameworks for guiding research engagement with online communities [done]
09.2024: Recruit and schedule Phase 3 Workshop with any interested prior participants- 10.2024: Analyze results and write CSCW paper [submitted to Oct. 2024 Cycle]
- 4.2025: Paper accepted to CSCW 2025
- 9.2025: Paper receives Best Paper award at CSCW 2025
- 10.2025: Documented the results on this research page [Done]
Policy, Ethics and Human Subjects Research
[edit]The methods and protocols proposed in the present study have been reviewed and approved by the University of Minnesota’s IRB on 8/29/2023 as human subjects research. Details of this approval are filed under STUDY00019610. Finally, the purpose of this document is to consult with and gain the community's consent before beginning research activity.
Results
[edit]The project’s findings are described in detail in our open access, Best Paper Award–winning publication at CSCW 2025.
Executive overview: For the most part, Wikipedia is already putting many of these ideas into practice as a role model for other online communities. Here is a short subset of findings relevant to Wikipedia members and organizers.
(1) Clarify expectations. Prioritize guides, bright-line rules, and documentation for others to make research decisions that align with Wikipedia's research expectations.
(2) Have conversations about community health and values. Members are curious about Wikipedia's values and how they relate to their values. Create spaces in the community to have discussions about community health.
(3) Highlight how research benefits the Wikipedia (when it does). Think about what success looks like for Wikipedia — and how research or external collaborations can support those goals.
(4) Help members make informed choices about data sharing. Be clear about what Wikipedia policies do (and don’t) protect, and help people understand the risks of public data use. What could enable members to have more control over their data?
FACTORS Framework: To translate our lessons from community workshops into practical guidance, we developed FACTORS, a framework that outlines four key functions important in ethical online community research. FACTORS stands for Functions for Action with Communities: Teaching, Overseeing, Reciprocating, and Sustaining.
Teaching Community Norms Researchers should learn and demonstrate understanding of communities' unique values. By socializing researchers to community norms, they can preserve community’s core goals.
Overseeing Bright Lines Some communities have boundaries that should never be crossed. When those boundaries are unclear, researchers should seek review from community members or organizers before conducting research.
Reciprocating Value Research should equalize who benefits from research. Communities want to see tangible benefits, so research findings should be made accessible and valuable to the wider audience.
Sustaining Resources Research can create compounding paper-cuts between independent projects over time. Researchers should prioritize minimizing and offsetting their footprint.
For information, consider reading my blog post overview.
The full, open-access CSCW paper is available here: Beyond the Individual: Community-Engaged Design and Implementation of a Framework for Ethical Online Communities Research.
Resources
[edit]Workshop materials and additional resources for conducting community-engaged research workshops are available in the publication’s supplementary materials section.
Researchers and community organizers can also access the FACTORS checklist—a set of critical prompts for identifying FACTORS in online communities. It is included in the blog post or the paper's appendix.
Dissemination
[edit]This work was presented at the ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2025), where it received a Best Paper Award. The talk attracted a full audience.
Additionally, we co-organized the Context Matters Workshop—inspired by this research—to connect with scholars interested in online community ethics.
Finally, we connected with the Wikimedia Foundation Research team following this work to provide an actionable set of recommendations for promoting FACTORS within the Wikimedia/Wikipedia research community. Recommendations where aimed at low-stakes, high-impact improvements to existing initiatives—such as including stronger expectations for proposing research in the Research:Index for wiki workshop attendance/acceptance.
References
[edit]- ↑ Fiesler, Casey; Proferes, Nicholas (2018). "“Participant” Perceptions of Twitter Research Ethics". Social Media + Society 4 (1). doi:10.1177/2056305118763366. Retrieved 2023-09-19.
- ↑ Clark, Mitchell (2021). "University of Minnesota banned from contributing to Linux kernel". The Verge. Vox Media. Retrieved 2023-09-19.
- ↑ Hallinan, Blake; Brubaker, Jed R; Fiesler, Casey (2019). "Unexpected expectations: Public reaction to the Facebook emotional contagion study.". New Media & Society 22 (6). doi:10.1177/1461444819876944. Retrieved 2023-09-19.
- ↑ Mayer, Jonathan (2021). "Princeton-Radboud Study on Privacy Law Implementation". Princeton. Retrieved 2023-09-19.
- ↑ "The Menlo Report: Ethical Principles Guiding Information and Communication Technology Research" (PDF). Department of Homeland Security. 2012. Retrieved 2023-09-19.
- ↑ franzke, aline shakti; Bechmann, Anja; Zimmer, Michael; Ess, Charles (2020). "Internet Research: Ethical Guidelines 3.0" (PDF). Association of Internet Researchers. Retrieved 2023-09-19.