Research talk:Editor milestones

Comments[edit]

Fascinating stuff E3 team! I'm really excited to see where this work winds up and hope we'll get to talk about your plans soon. Also, here are some comments I'd love to discuss with any of you:

It would be great to perform other, more sensitive analyses on this data. For example, this kind of data would be totally amenable to randomization inference (lets you test against a "sharp null" of no treatment effects - no confidence intervals!).
Also, I would love to test not only for additive, multiplicative, and/or tobit effects (these would allow us to think about how the intervention might have produced different kinds of effects beyond raising the mean number of edits). Sadly, I can't quickly locate articles on these topics in WP, but I do have some books/articles that are useful resources.
In future studies, you might want to think about blocking on certain attributes of editors that we already know correlate with differential behaviors/effects. For example, in this study I'd love to see what happens in sub-populations of the "long-slow-growth" editors vs. the "fast and furious" editors and blocking could help you address this.

Aaronshaw (talk) 20:23, 13 July 2012 (UTC)[reply]

Also, two more comments about adjustments to the data/variables that might enhance the precision of the analysis:

One of the things Mako and I struggled with in our study was finding some way to confirm that the people who received barnstars but didn't move them to their user pages actually saw the award. Our solution (totally not ideal) was to only analyze users in our data set who made at least one edit to their user page or user talk page after the barnstar arrived. Could you employ some similar (or maybe even better?) mechanism to verify that the barnstar recipients have seen their awards? Given that this is an experiment, you won't be able to just remove the people who didn't see the award from the analysis, but it would still be useful to consider this potential source of variation in your results.
Relatedly, did you confirm that all of the editors included in this analysis continued to edit the encylopedia after receiving their barnstar?
Any particular reason why you chose to analyze +/- three days of activity around the distribution of the award? If not, how about analyzing +/- one week or +/- one month? I am inclined to believe that the activity of many editors is likely to follow at least some sort of weekly pattern (given that many people have jobs, families, etc. that keep the rest of their lives on a schedule with a weekly regularities), so I would suspect that in looking only at three days of activity you would not find any results from a barnstar.

Aaronshaw (talk) 15:08, 14 July 2012 (UTC)[reply]

We are currently in the process of analyzing the activity of editors in the 30 days following receipt of the barnstar. We published this report shortly after the last barnstaree received their barnstar; it simply hadn't been long enough for us to include the 30 day metric. We should be able to report on that 1 month metric next week.-Kgladstone (WMF) (talk) 21:05, 19 July 2012 (UTC)[reply]