User:Halfak (WMF)/Trip reports/CSCW 2018

This year, I attended CSCW 2018 as well as the Participation and Algorithms workshop beforehand. In this short essay, I plan to highlight some of the things I think are relevant to Wikimedia.

Participation and Algorithms

From https://algorithmsworkshop.wixsite.com/mysite

Contributing to a growing attention to algorithms and algorithmic interaction in the CHI and CSCW communities, this workshop aims to deal centrally with the topic of human “participation” and its changing role to data-driven, algorithmic ecosystems. Such a focus includes projects that involve users in the design of algorithms and “human-in-the-loop” systems, broader investigations into the ways in which “participation” is situated in data-driven activities, as well as conceptual concerns about participation’s changing contours in contemporary social computing landscapes. This one-day workshop led by academic and industry researchers and sets out to achieve three goals: identify cases and ongoing projects on the topic of participation in algorithmic ecosystems; create a tactical toolkit of key challenges and strategies in this space; and set a forward-facing agenda to provoke further attention to the changing role of participation in contemporary sociotechnical systems.

In the workshop, we split into working groups early and used the time to answer some key questions about participation. My group worked describing the points in an algorithm's development process where participation is relevant. We came up with a messy diagram.

A poster sheet is presented with a diagram of the points in an algorithm's development where participation would be relevant. Shared value identification, decision process design, algorithmic UI design, labeled data gathering, algorithm design, and algorithm development.

This gist is that there are 5 places where broader participation is interesting/desirable:

value identification

This needs to be done by some community. Values involve tradeoffs and discussion. E.g. is it OK to have a slightly biased model if it is more accurate? How much bias is too much?

decision process design

Usually AIs/algorithms are used to support some decision process. E.g. should this edit be reverted? or should this person receive a low rate loan offer? The design of the process goes well beyond the scope of any algorithm. The decision to include an algorithm as support is part of the decision process design.

algorithmic UI design

How will people interact with the algorithm. How will a model's predictions be presented?

algorithm design

How will we train/evaluate a model? How should we gather training data? What types of modeling strategies should be used.

labeled data gathering

This is the process of gathering training/test data to build/evaluate the model. Participation in this process means that your judgment will be directly used to evaluate success/failure and to train the model how to behave.

algorithm development

This is the code that makes the model work. Code is necessary for training and evaluating the model. Code is necessary for integrating the modeling into any UI.

Notable works

• Operationalizing conflict and cooperation between automated software agents in Wikipedia: A replication and expansion of ‘Even Good Bots Fight,’ R. Stuart Geiger: UC-Berkeley; Aaron Lee Halfaker: Wikimedia Foundation
  • This paper replicates, extends, and refutes conclusions made in a study published in PLoS ONE ("Even Good Bots Fight"), which claimed to identify substantial levels of conflict between automated software agents (or bots) in Wikipedia using purely quantitative methods. By applying an integrative mixed-methods approach drawing on trace ethnography, w...
• Reciprocity and Donation: How Article Topic, Quality and Dwell Time Predict Banner Donation on Wikipedia, Rafal Kocielnik: University of Washington; Os Keyes: University of Washington; Jonathan T Morgan: Wikimedia Foundation; Dario Taraborelli: Wikimedia Foundation; David W. McDonald: University of Washington; Gary Hsieh: University of Washington
  • Donation-based support for open, peer production projects such as Wikipedia is an important mechanism for preserving their integrity and independence. For this reason, understanding donation behavior and incentives is crucial in this context. In this work, using a dataset of aggregated donation information from Wikimedia's 2015 fund-raising campa...
• Mind Your POV: Convergence of Articles and Editors Towards Wikipedia's Neutrality Norm, Umashanthi Pavalanathan: Georgia Tech; Xiaochuang Han: Georgia Tech; Jacob Eisenstein: Georgia Tech
  • Wikipedia has a strong norm of writing in a ``neutral point of view (NPOV). Articles that violate this norm are tagged, and editors are encouraged to make corrections. But the impact of this tagging system has not been quantitatively measured. Does NPOV tagging help articles to converge to the desired style? Do NPOV corrections encourage editors ...
• Bot Detection in Wikidata Using Behavioral and Other Informal Cues, Andrew Hall: University of Minnesota; Loren Terveen: University of Minnesota; Aaron Halfaker: Wikimedia Foundation
  • Bots have been important to peer production’s success. Wikipedia, OpenStreetMap, and Wikidata all have taken advantage of automation to perform work at a rate and scale exceeding that of human contributors. Understanding the ways in which humans and bots behave in these communities is an important topic, and one that relies on accurate bot recogn...
• With Few Eyes, All Hoaxes are Deep, Sumit Asthana: Indian Institute of Technology, Patna; Aaron Halfaker: Wikimedia Foundation
  • Quality control is critical to open production communities like Wikipedia. Editors enact quality control on the borders of Wikipedia to review edits (counter-vandalism) and new article creations (new page patrolling) shortly after they are saved. In this paper, we describe a long-standing set of inefficiencies that have plagued new page patrollin...
• Welcome' Changes? Descriptive and Injunctive Norms in a Wikipedia Sub-Community, Jonathan T Morgan: Wikimedia Foundation ; Anna Filippova: GitHub
  • Open online communities rely on social norms for behavior regulation, group cohesion, and sustainability. Research on the role of social norms online has mainly focused on one source of influence at a time, making it difficult to separate different normative influences and understand their interactions. In this study, we use the Focus Theory to exa...
• It was fun, but did it last? The dynamic interplay between fun motives and contributors’ activity in peer production, Martina Balestra: New York University; Lior Zalmanson: New York University; Coye Cheshire: UC Berkeley; Ofer Arazy: University of Haifa; Oded Nov: New York University
  • Peer production communities often struggle to retain contributors beyond initial engagement. This may be a result of contributors' level of motivation, as it is deeply intertwined with activity. Existing studies on participation focus on activity dynamics but overlook the accompanied changes in motivation. To fill this gap, this study examines the ...

Notes

On the position of Wikimedia

When it comes to participation in algorithmic development, we're decades ahead of everywhere else. I was able to rattle off a few places where we do participatory work with our various communities and give examples off the top of my head. It took me a while to get everyone at my table up to date with where my thinking was. I think this is not evidence of some sort of deep insight I have but rather how long Wikipedians have been developing their own governance strategies. We need to write more about this. I think it will be very productive and well received.

Martina Balestra's recent work in motivation of newcomers

I've been asleep at the wheel and I need to catch up on what Martina has been up to. Just check out her list of recent publications: https://scholar.google.com/citations?hl=en&user=awKoSOQAAAAJ&view_op=list_works&sortby=pubdate

• Individual and Collective Determinants of Activity in Collaborative Production
• It was fun, but did it last? The dynamic interplay between fun motives and contributors’ activity in peer-production
• Investigating the motivational paths of peer production newcomers
• Motivational Determinants of Participation Trajectories in Wikipedia.
• etc.

I've invited her to come to the Wikimedia Research Showcase to present some of her work. The relevance is overwhelming.

Descriptive and Injunctive Norms

This is mostly in reference to what I learned from User:Jtmorgan's presentation at CSCW. I thought the idea of demonstrating norms is fascinating. I've always thought about the teahouse as a great demonstration of norms. And I think the idea of a sort of first-mover effect in norms was fascinating. But I really liked how the framework of "descriptive" vs. "injunctive" norms gives one the ability to talk about how apparent a norm is. By making certain norms apparent, we can help people know what the norms *are*. One example that comes to me is the foot prints in the Hong Kong subway that show where people should line up to wait for the train. It's really helpful for a tourist like me who may not realize that I'm cutting an entire line or standing in the way like an idiot.