Research talk:Measuring editor time commitment and workflow

From Meta, a Wikimedia project coordination wiki

comments[edit]

This is an interesting topic, but I think it will be really hard to get solid data. Not only are individual edits very variable, but the editing process can be extremely variable per editor. There are a lot of different kinds of work to do around here. Some people use semi-automated tools to make hundreds of edits per hour, some people can invest a lot of time on edits. Your target participant pool seems on the small side to reliably capture that variability.

Are you sampling all edits, including Talk pages and Wikipedia space? Sometimes it takes significant discussion with many participants to sort out article issues, and of course the encyclopedia couldn't function without behind the scenes work deleting junk-articles, blocking vandals, and other maintenance.

You'll have a hard time sampling casual editors. They generally don't even know Village Pump exists, and only a portion of serious editors follow the Pump. You might get multiple admins, where even one admin would be a statistical anomaly in a truly random sample.

In the most common case the time for an edit will equal time since last edit. You may want to track that and allow a 1-click accept for that value.

I don't plan to participate, but I imagine it would be fugly trying to do so. I have a habit of keeping a large number of browser tabs open, often for days, and I jump between them. Sometimes I do reading or research to make a rapid fire series of edits, or even to decide not to make an edit. The only way to count that is as part of the interval since last edit.

You'd probably be better off trying to track online time and asking what offline time needs to be added in. That would give a much more extensive sample of all of the edits during the test period, and it solves the sometimes messy issues of how to attribute time spent on individual edits. Alsee (talk) 09:01, 18 April 2016 (UTC)[reply]

Hey Alsee. I've been advising Another Article on the design of this study (so there's my COI). It is intended to fill gaps left by my own methodologies for delineating editing sessions and measuring labor hours. Despite a strong theoretical and empirical justification, we really don't know what editors are doing while they are working on an edit -- and whether the assumptions I have been making hold at all. So, I see this study as a general exploration into the unknowns of editors' work.
With that said, I agree with your critique. This study will not be able to "reliably capture [editing pattern] variability" in a way that would let us quantify the Wikipedia population with exactness. However, I still think that this methodology is well designed for finding out: "Do people run to the library? How long does that take? What kind of edits require that kind of investment? How long does it generally take? Does it make sense to include that trip to the library in labor hour measurements?" Right now, I think it's OK that we won't be able to know "What proportion of all edits require a trip to the library?" and "What's the average time to visit the library across all Wikipedians?". IMO it's OK that such quantitatively specific questions are left to a follow-up study. I'd even go so far as to argue that the proposed study is necessary before any randomized, quantitative study could take place. How do we even know that measuring the time taken to travel to the library is important?
Your notes about how ugly and complicated editing really is is exactly the kind of notes that I think Another Article is looking for. Right now, the edit-to-edit work practices of Wikipedians are really not known well. If this phenomena were better understood, it could be beneficial in a lot of ways -- e.g. WMF Product could do a better job of designing technologies that actually work for the reality that is editing Wikipedia -- rather than some idealized, simplistic assumption or solely based some minor amount of personal experience. --Halfak (WMF) (talk) 14:44, 25 April 2016 (UTC)[reply]