Research talk:HHVM newcomer engagement experiment/Work log/2014-10-20
Monday, October 20, 2014
Today, I get to try and figure out how I got conflicting results for desktop and mobile. First things first, let's check to see if there's something weird going on with the measurement by trying difference measurement strategies. First, I want to plot the proportion of editors who make it to their second edit.
- What proportion of users make at least two edits?
- What proportion of users who make at least one edit will also make at least two?
Here, we can see the effects fall away when I look at the proportion of newcomers who both made one edit and also made two. I suspect that we'll see the effect them if I merely draw the proportion of newcomers who make at least one edit. --Halfak (WMF) (talk) 20:34, 20 October 2014 (UTC)
- What proportion of users will make at least one edit?
It looks like our effect is there. If HHVM is causing anything here, it's encouraging mobile editors to make at least one edit and discouraging desktop editors from making their first edit. Weird. I think that we'll likely see much more interesting bits in the full week-long analysis that will allow me to observe session length. --Halfak (WMF) (talk) 20:41, 20 October 2014 (UTC)
Showed Ori and he wanted to know about the proportion of users who make it to 5 edits or more. So...
- What proportion of user who make at least one edit will make at least five?
R:Newly registered users who make it to 5 edits in 24h.
New editors who make it to 5 edits in 24h.
So, while we didn't see a difference in the proportion of new editors who make it to at least 2 edits, we do see a difference in the proportion of newcomers who make it to 5 edits for mobile. Here, HHVM seems to be making it easier to save more edits in the first 24h.
This suggest that, for mobile, HHVM makes it easier to save the first edit and from there, save the fifth. However for Desktop, HHVM makes it hard to save the first edit, but otherwise, we see no significant effect. --Halfak (WMF) (talk) 21:08, 20 October 2014 (UTC)