User:EpochFail/CSCW 2016 report

Hey folks. I'll use this space to provide a limited report on what I saw at CSCW 2016. I was only able to attend at most 1 talk/panel for every 5 talks/panels because of parallel tracks, so this is going to be a limited view of the conference. Still, I hope it is useful.

The Workshop

This CSCW, I spend a lot of energy organizing a workshop for exploring technologies intended to make it easier to get started working on Wikimedia datasets. We asked participants to replicate a past CSCW paper (this one: ). We introduced participants to R:Quarry and one of Yuvipanda's new projects -- PAWS (PywikibotAs[a]WebShell) -- which is a Jupyter hub instance running in mw:Wikimedia Labs. We also provided information about available datasets using http://census.datafactories.org. Regretfully, the methods protocol was not ready, so I made sure that I was available to answer any methods questions that workshop participants had.

I think we learned a lot from this workshop. I kept hearing from others that it went really well, so that's re-assuring. Here are my own personal take-aways:

1. People wanted to use *both* quarry and PAWS to work with the data. Quarry was useful for exploring the structure and contents of a table. PAWS was useful for both performing more substantial analysis (that are hard to do in SQL) and forming a report.
2. It would be great if the methods for the paper provided deep links into my field notes while writing the paper. We ran into some weird results that it took me a long time to figure out. It turns out that some decisions I made while performing the analysis colored the results quite substantially in some rare cases.
3. "Replication" is a kind of loose term -- more loose than I expected. Some people literally replicated the measurements taken in the study while others just performed minor verifications and then applied the measure to new data. One group spent most of the day trying to identify unusual actors (usually bots) in the data by looking for substantial irregularities. They found an anomaly from the early days of Wikipedia -- which I'm pretty sure was the original conversion from UseMod wiki to MediaWiki.

Notable papers

There were a few papers that stood out to me as particularly relevant to my work -- or just fun to think about.

Analysing volunteer engagement in humanitarian mapping

Dittus, M., Quattrone, G., & Capra, L. Analysing volunteer engagement in humanitarian mapping: building contributor communities at large scale. PDF

In this study the researchers looked at the activities of the Humanitarian OpenStreetMap Team (HOT) and tried to infer how recruiting lead to different engagement dynamics. Of course, these types of recruitment/engagement studies are relevant to Wikimedia given our interest in recruiting and retaining newcomers, but the reason that I want to highlight this work is really because of a call to action raised by Martin Dittus while presenting. He called on researchers to come and meet the OpenStreetMap community rather than just doing "drive-by" research projects. This is something that resonates with me as I see myself as more able to contribute back to Wikimedian things that matter because I have chosen to become a part of the community. I'll be looking for Martin at future CSCW's and related conferences to share notes on being an actively involved community researcher.

The Crowd is a Collaborative Network

Gray, M. L., Suri, S., Ali, S. S., & Kulkarni, D. (2016). The Crowd is a Collaborative Network. Proceedings of Computer-Supported Cooperative Work. PDF

I'll admit that I was distracted for most of this presentation, so I can't give a fair summary, so I'll just call attention to a bit that resonated with me as a collaborative system designer. Your "crowd workers" will organize into a collaborative network whether you build support for them to do that or not. E.g. mechanical turk does not provide any means for workers to organize around good employers and work conditions. So they set up external/parallel technical infrastructures to do this. They communicate on web forums, mailing lists and private messages about good jobs and bad employers. Other platforms see similar behavior. We should embrace this collaborative aspect of crowds and design for it rather than trying to minimize it.

Early activity diversity

Karumur, R. P., Nguyen, T. T., & Konstan, J. A. (2016, February). Early Activity Diversity: Assessing Newcomer Retention from First-Session Activity. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 595-608). ACM.

I love this study. It's good work, but most importantly, it builds off of a lot of my work around the early activities of newcomers to Wikipedia and using sessions to think about human activity better. Raghav described a study where they looked at the first session activities of users in en:MovieLens and used them to predict the long-term retention of these users. They found that they could make strong predictions about how long an editor would use the recommender system by looking at how much activity the user did in their first session (replicating my work in another system), but they also found that the diversity of the user's activities in this first session was even more predictive. I'm not sure I buy the argument about why users stuck around longer (diversified value from the site), but I think that the result is fascinating and something that would be worth experimenting with in Wikimedia stuffs.

One and done

McInnis, B. J., Murnane, E. L., Epstein, D., Cosley, D., & Leshed, G. (2016, February). One and Done: Factors affecting one-time contributors to ad-hoc online communities. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 609-623). ACM.

I love the motivation for this study, but I'm actually quite surprised that it was accepted at all. The authors argue that many contributors in online spaces will only make one contribution ever and that we should understand what types of contributions these one-off editors do well and design our systems to account for that. Man, am I right there with that argument! But then they went on to fake one-off contributors by designing a mechanical turk study. Arg! Just come to Wikipedia and study our one-off contributors!? Or Reddit! Or Zooniverse! Or stack overflow! Or OpenStreetMap! You know -- real systems with real context in the field. Oh well. Fun idea. Maybe I'll challenge their results with a study of one-off Wikipedia contributors for CSCW next year.

A case for sequential analysis in CSCW

Keegan, B. C., Lev, S., & Arazy, O. (2015). Analyzing Organizational Routines in Online Knowledge Collaborations: A Case for Sequence Analysis in CSCW. arXiv preprint arXiv:1508.04819. PDF

This is a weird paper. It isn't actually a study. It's more like a methods position paper. Keegan et al. argue for the analysis of activity sequences in spaces like Wikipedia. E.g. he uses the example of edits to articles where you might see something like <newcomer edit>, <oldtimer edit>, <newcomer edit>. The commonality (or rarity) of this sequence might imply something about the editing dynamics of the page in question. Regretfully, the authors did not see fit to include an actual example of this measurement strategy being useful, but I do think that they are right. I just wish they would have demonstrated the measure somewhere and showed that you could learn something from it. As it stands, it seems like this was a sort of least publishable unit and I expect Keegan et al. to publish a real analysis somewhere else.

Considering Time in Designing Large-Scale Systems for Scientific Computing

Chen, N. C., Poon, S. S., Ramakrishnan, L., & Aragon, C. R. (2015). Considering Time in Designing Large-Scale Systems for Scientific Computing. arXiv preprint arXiv:1510.05069. PDF

This paper reports on a study of scientists using a high performance computing cluster. The key result happens to be something I have personally experienced and this (for some reason) makes it very interesting to me. The researchers found that scientific developers (researchers developing against the computing cluster) were substantially more concerned about the time it takes to engineer and debug a chunk of code than the actual runtime. This means that changing a process that generally takes 3 days to only take one day is not really worth it if it consumes more of the scientist's time. These considerations are generally not accounted for by those who engineer high-performance computing infra, and it should.

Parting Crowds

Kairam, S., & Heer, J. Parting Crowds: Characterizing Divergent Interpretations in Crowdsourced Annotation Tasks. PDF

This paper reports on a method for analyzing the labels provided by a crowd of workers. By using a simple clustering strategy on the provided labels, the researchers were able to identify clusters of labelers who consistently disagreed with other clusters. The separation of these clusters can be used to identify ambiguities in the labeling instructions or real differences in the subjectivities of the labelers. I hope to apply this method to the data we have been getting out of Wiki labels. I imagine that we can find substantial differences between editors' interpretations of good-faith and the types of intentions that they associate with edits. This could potentially help us adjust our labeling taxonomies and documentation.

Conclusion

I'm tired and I don't want to think about CSCW for at least a week!