Research talk:HHVM newcomer engagement experiment/Work log/2014-10-24
Friday, October 24, 2014
Now for the figures. First, lets look at proportion measures.
While we see significant differences for the proportion of editors who save one edit (new editors), HHVM doesn't seem to have affected the proportion of newly registered users who fully activate in the first week since registration. Interestingly, it appears that HHVM has reduced the probability of coming back for a second activity session for desktop newcomers. This is surprising since we should expect that improved performance would result in in higher likelihood to return to editing. --Halfak (WMF) (talk) 23:22, 24 October 2014 (UTC)
Next, lets look at comparisons of scalar metrics.
Here, again, we see edit rate measures dominated by the probability of making at least one edit. There's a marginally significant difference between edit rates for desktop where HHVM may be decreasing total edits while for mobile, there's a clear, significant gain. When we filter article edits for those that are productive (not deleted or reverted) the negative effect for desktop disappears in the noise while the benefit for mobile users stays consistent. This could suggest that there really isn't a meaningful effect for valuable edits on desktop, but that there is a substantial benefit for mobile users.
The measurements of time spent editing editing are harder to reason about. Again we see the familiar pattern that we saw with edit rate, but this time we're talking about the time a user spends working. We should expect that, when saving pages is faster the user will spend less time working, but that's clearly not the case for mobile and I'm skeptical that we should read into the effect for desktop. --Halfak (WMF) (talk) 23:39, 24 October 2014 (UTC)