Research talk:HHVM newcomer engagement experiment/Work log/2014-10-24

Friday, October 24, 2014

Today, my goal is to explore the basic patterns of time spent editing. First things first, I need to gather a week's worth of stats. --Halfak (WMF) (talk) 21:05, 24 October 2014 (UTC)

Stats:

bucket	ui_type	n	editing.k	second_session.k	active.k	week_revisions.geo.mean	week_main_revisions.geo.mean	week_session_seconds.geo.mean
hhvm	mobile	10243	3436	669	458	0.4491373	0.4101124	6.298709
php5	mobile	10222	3308	681	434	0.4277993	0.3913162	5.794002
hhvm	desktop	29678	9978	2817	1992	0.5193337	0.3566733	7.30325
php5	desktop	29356	10054	2932	2019	0.5316955	0.3645053	7.606171

Now for the figures. First, lets look at proportion measures.

Proportion measures

New editor proportion. The proportion of new editors per newly registered user is plotted by experimental condition and registration interface.

Active editor proportion. The proportion of active editors per newly registered user is plotted by experimental condition and registration interface.

Returning editor proportion. The proportion of newly registered users who came back for a second activity session is plotted by experimental condition and registration interface.

While we see significant differences for the proportion of editors who save one edit (new editors), HHVM doesn't seem to have affected the proportion of newly registered users who fully activate in the first week since registration. Interestingly, it appears that HHVM has reduced the probability of coming back for a second activity session for desktop newcomers. This is surprising since we should expect that improved performance would result in in higher likelihood to return to editing. --Halfak (WMF) (talk) 23:22, 24 October 2014 (UTC)

Next, lets look at comparisons of scalar metrics.

Scalar measures

Edit rate. The geometric mean revisions saved per user is plotted by experimental condition and registration interface.

Article edit rate. The geometric mean article revisions saved per user is plotted by experimental condition and registration interface.

Productive edit rate. The geometric mean productive edits saved per user is plotted by experimental condition and registration interface.

Time spent editing. The geometric mean total session duration per user is plotted by experimental condition and registration interface.

Here, again, we see edit rate measures dominated by the probability of making at least one edit. There's a marginally significant difference between edit rates for desktop where HHVM may be decreasing total edits while for mobile, there's a clear, significant gain. When we filter article edits for those that are productive (not deleted or reverted) the negative effect for desktop disappears in the noise while the benefit for mobile users stays consistent. This could suggest that there really isn't a meaningful effect for valuable edits on desktop, but that there is a substantial benefit for mobile users.

The measurements of time spent editing editing are harder to reason about. Again we see the familiar pattern that we saw with edit rate, but this time we're talking about the time a user spends working. We should expect that, when saving pages is faster the user will spend less time working, but that's clearly not the case for mobile and I'm skeptical that we should read into the effect for desktop. --Halfak (WMF) (talk) 23:39, 24 October 2014 (UTC)