Research talk:HHVM newcomer engagement experiment/Work log/2014-10-24

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search

Friday, October 24, 2014[edit]

Today, my goal is to explore the basic patterns of time spent editing. First things first, I need to gather a week's worth of stats. --Halfak (WMF) (talk) 21:05, 24 October 2014 (UTC)

Stats:

bucket ui_type n editing.k second_session.k active.k week_revisions.geo.mean week_main_revisions.geo.mean week_session_seconds.geo.mean
hhvm mobile 10243 3436 669 458 0.4491373 0.4101124 6.298709
php5 mobile 10222 3308 681 434 0.4277993 0.3913162 5.794002
hhvm desktop 29678 9978 2817 1992 0.5193337 0.3566733 7.30325
php5 desktop 29356 10054 2932 2019 0.5316955 0.3645053 7.606171

Now for the figures. First, lets look at proportion measures.

Proportion measures[edit]

The proportion of new editors per newly registered user is plotted by experimental condition and registration interface.
New editor proportion. The proportion of new editors per newly registered user is plotted by experimental condition and registration interface.
The proportion of active editors per newly registered user is plotted by experimental condition and registration interface.
Active editor proportion. The proportion of active editors per newly registered user is plotted by experimental condition and registration interface.
The proportion of newly registered users who came back for a second activity session is plotted by experimental condition and registration interface.
Returning editor proportion. The proportion of newly registered users who came back for a second activity session is plotted by experimental condition and registration interface.


While we see significant differences for the proportion of editors who save one edit (new editors), HHVM doesn't seem to have affected the proportion of newly registered users who fully activate in the first week since registration. Interestingly, it appears that HHVM has reduced the probability of coming back for a second activity session for desktop newcomers. This is surprising since we should expect that improved performance would result in in higher likelihood to return to editing. --Halfak (WMF) (talk) 23:22, 24 October 2014 (UTC)

Next, lets look at comparisons of scalar metrics.

Scalar measures[edit]

The geometric mean revisions saved per user is plotted by experimental condition and registration interface.
Edit rate. The geometric mean revisions saved per user is plotted by experimental condition and registration interface.
The geometric mean article revisions saved per user is plotted by experimental condition and registration interface.
Article edit rate. The geometric mean article revisions saved per user is plotted by experimental condition and registration interface.
The geometric mean productive edits saved per user is plotted by experimental condition and registration interface.
Productive edit rate. The geometric mean productive edits saved per user is plotted by experimental condition and registration interface.
The geometric mean total session duration per user is plotted by experimental condition and registration interface.
Time spent editing. The geometric mean total session duration per user is plotted by experimental condition and registration interface.

Here, again, we see edit rate measures dominated by the probability of making at least one edit. There's a marginally significant difference between edit rates for desktop where HHVM may be decreasing total edits while for mobile, there's a clear, significant gain. When we filter article edits for those that are productive (not deleted or reverted) the negative effect for desktop disappears in the noise while the benefit for mobile users stays consistent. This could suggest that there really isn't a meaningful effect for valuable edits on desktop, but that there is a substantial benefit for mobile users.

The measurements of time spent editing editing are harder to reason about. Again we see the familiar pattern that we saw with edit rate, but this time we're talking about the time a user spends working. We should expect that, when saving pages is faster the user will spend less time working, but that's clearly not the case for mobile and I'm skeptical that we should read into the effect for desktop. --Halfak (WMF) (talk) 23:39, 24 October 2014 (UTC)