Research talk:VisualEditor's effect on newly registered editors/May 2015 study/Work log/2015-05-27
Wednesday, May 27, 2015[edit]
Today I am performing a preliminary analysis of the pilot study. The bucketing for this study started and ended three days ago, so that means I won't have a full week with which to review newcomers contributions, but we should be able to looks for anomalies in their first 24-48 hours just fine.
Bucketed users[edit]
So, first things first, I need to gather the set of bucketed users and label them by condition.
> SELECT event_userId AS user_id, timestamp AS registration, event_displayMobile AS via_mobile FROM ServerSideAccountCreation_5487345 WHERE wiki = "enwiki" AND event_isSelfMade AND timestamp BETWEEN "2015052115" and "2015052215" LIMIT 10; +----------+----------------+------------+ | user_id | registration | via_mobile | +----------+----------------+------------+ | 25251662 | 20150521150038 | 0 | | 25251673 | 20150521150153 | 0 | | 25251675 | 20150521150208 | 1 | | 25251680 | 20150521150245 | 0 | | 25251684 | 20150521150309 | 0 | | 25251687 | 20150521150343 | 0 | | 25251689 | 20150521150427 | 0 | | 25251690 | 20150521150430 | 0 | | 25251696 | 20150521150500 | 0 | | 25251700 | 20150521150502 | 1 | +----------+----------------+------------+ 10 rows in set (0.00 sec)
OK. Let's make sure I have the time bounds right before we go any farther.
> SELECT LEFT(user_registration, 10) AS hour, SUM(ve.up_user IS NOT NULL)/COUNT(*) AS ve_prop FROM user LEFT JOIN user_properties AS ve ON up_user = user_id AND up_property = 'visualeditor-enable' WHERE user_registration BETWEEN "2015052114" AND "2015052217" AND user_id > 25241662 GROUP BY 1; +------------+---------+ | hour | ve_prop | +------------+---------+ | 2015052114 | 0.0073 | | 2015052115 | 0.3188 | | 2015052116 | 0.3147 | | 2015052117 | 0.3268 | | 2015052118 | 0.3298 | | 2015052119 | 0.3091 | | 2015052120 | 0.2739 | | 2015052121 | 0.3302 | | 2015052122 | 0.3030 | | 2015052123 | 0.3299 | | 2015052200 | 0.3322 | | 2015052201 | 0.3681 | | 2015052202 | 0.3713 | | 2015052203 | 0.3535 | | 2015052204 | 0.3401 | | 2015052205 | 0.3728 | | 2015052206 | 0.3371 | | 2015052207 | 0.3651 | | 2015052208 | 0.3291 | | 2015052209 | 0.3144 | | 2015052210 | 0.3326 | | 2015052211 | 0.3450 | | 2015052212 | 0.3348 | | 2015052213 | 0.3154 | | 2015052214 | 0.3300 | | 2015052215 | 0.0249 | | 2015052216 | 0.0079 | +------------+---------+ 27 rows in set (0.10 sec)
Looks like the deployments are tight to the hour. Good. Now to confirm which users were bucketed.
> SELECT user_id % 2 = 0 AS even_id, LEFT(user_registration, 10) AS hour, SUM(ve.up_user IS NOT NULL)/COUNT(*) AS ve_prop FROM user LEFT JOIN user_properties AS ve ON up_user = user_id AND up_property = 'visualeditor-enable' WHERE user_registration BETWEEN "2015052115" AND "2015052118" AND user_id > 25241662 GROUP BY 1, 2; +---------+------------+---------+ | even_id | hour | ve_prop | +---------+------------+---------+ | 0 | 2015052115 | 0.0109 | | 0 | 2015052116 | 0.0000 | | 0 | 2015052117 | 0.0036 | | 1 | 2015052115 | 0.6268 | | 1 | 2015052116 | 0.6295 | | 1 | 2015052117 | 0.6489 | +---------+------------+---------+ 6 rows in set (0.04 sec)
OK. it is clearly newcomers with an even user_id. Now to add that to my user extraction query.
> SELECT event_userId AS user_id, IF(event_userId % 2 = 0, "experimental", "control") AS bucket, timestamp AS registration, event_displayMobile AS via_mobile FROM ServerSideAccountCreation_5487345 WHERE wiki = "enwiki" AND event_isSelfMade AND timestamp BETWEEN "2015052115" and "2015052215" LIMIT 10; +----------+--------------+----------------+------------+ | user_id | bucket | registration | via_mobile | +----------+--------------+----------------+------------+ | 25251662 | experimental | 20150521150038 | 0 | | 25251673 | control | 20150521150153 | 0 | | 25251675 | control | 20150521150208 | 1 | | 25251680 | experimental | 20150521150245 | 0 | | 25251684 | experimental | 20150521150309 | 0 | | 25251687 | control | 20150521150343 | 0 | | 25251689 | control | 20150521150427 | 0 | | 25251690 | experimental | 20150521150430 | 0 | | 25251696 | experimental | 20150521150500 | 0 | | 25251700 | experimental | 20150521150502 | 1 | +----------+--------------+----------------+------------+
Cool. Now to include whether ve_enabled property.
SELECT event_userId AS user_id, IF(event_userId % 2 = 0, "experimental", "control") AS bucket, timestamp AS registration, event_displayMobile AS via_mobile, ve.up_user IS NOT NULL AS ve_enabled FROM ServerSideAccountCreation_5487345 LEFT JOIN enwiki.user_properties ve ON event_userId = up_user AND up_property = 'visualeditor-enable' WHERE wiki = "enwiki" AND event_isSelfMade AND timestamp BETWEEN "2015052115" and "2015052215" LIMIT 10; +----------+--------------+----------------+------------+------------+ | user_id | bucket | registration | via_mobile | ve_enabled | +----------+--------------+----------------+------------+------------+ | 25251662 | experimental | 20150521150038 | 0 | 1 | | 25251673 | control | 20150521150153 | 0 | 0 | | 25251675 | control | 20150521150208 | 1 | 0 | | 25251680 | experimental | 20150521150245 | 0 | 1 | | 25251684 | experimental | 20150521150309 | 0 | 1 | | 25251687 | control | 20150521150343 | 0 | 0 | | 25251689 | control | 20150521150427 | 0 | 0 | | 25251690 | experimental | 20150521150430 | 0 | 1 | | 25251696 | experimental | 20150521150500 | 0 | 1 | | 25251700 | experimental | 20150521150502 | 1 | 1 | +----------+--------------+----------------+------------+------------+ 10 rows in set (0.00 sec)
Cool. It looks like we're enabling VE for mobile registrations. That'll be noise if they don't eventually edit from the desktop (no VE on mobile yet), so I'll keep 'em flagged so we can filter them out in the analysis later.
SELECT
event_userId AS user_id,
IF(event_userId % 2 = 0, "experimental", "control") AS bucket,
timestamp AS registration,
event_displayMobile AS via_mobile,
ve.up_user IS NOT NULL AS ve_enabled
FROM log.ServerSideAccountCreation_5487345
LEFT JOIN enwiki.user_properties ve ON
event_userId = up_user AND
up_property = 'visualeditor-enable'
WHERE
wiki = "enwiki" AND
event_isSelfMade AND
timestamp BETWEEN "2015052115" and "2015052215";
$ wc datasets/pilot_users.tsv 4190 20950 161399 datasets/pilot_users.tsv
--Halfak (WMF) (talk) 12:43, 27 May 2015 (UTC)
Metrics![edit]
OK. Time to extract some metrics.
I used this library https://github.com/halfak/mwmetrics, to run the following bit:
datasets/pilot_user_metrics.tsv: datasets/pilot_users.tsv cat datasets/pilot_users.tsv | \ mwmetrics new_users enwiki $(dbstore) > \ datasets/pilot_user_metrics.tsv
It didn't take long at all. Now, I've created a repo for my own code[1] and loaded up my work in R.
> summary(user_metrics)
user_id bucket registration via_mobile
Min. :25251662 control :2080 Min. :2.015e+13 Min. :0.0000
1st Qu.:25254181 experimental:2109 1st Qu.:2.015e+13 1st Qu.:0.0000
Median :25256579 Median :2.015e+13 Median :0.0000
Mean :25256593 Mean :2.015e+13 Mean :0.2234
3rd Qu.:25259016 3rd Qu.:2.015e+13 3rd Qu.:0.0000
Max. :25261540 Max. :2.015e+13 Max. :1.0000
ve_enabled user_registration day_revisions
Min. :0.0000 Min. :2.015e+13 Min. : 0.000
1st Qu.:0.0000 1st Qu.:2.015e+13 1st Qu.: 0.000
Median :1.0000 Median :2.015e+13 Median : 0.000
Mean :0.5058 Mean :2.015e+13 Mean : 0.925
3rd Qu.:1.0000 3rd Qu.:2.015e+13 3rd Qu.: 1.000
Max. :1.0000 Max. :2.015e+13 Max. :149.000
day_reverted_main_revisions day_main_revisions day_wp_revisions
Min. : 0.0000 Min. : 0.0000 Min. :0.00000
1st Qu.: 0.0000 1st Qu.: 0.0000 1st Qu.:0.00000
Median : 0.0000 Median : 0.0000 Median :0.00000
Mean : 0.2344 Mean : 0.6469 Mean :0.01146
3rd Qu.: 0.0000 3rd Qu.: 0.0000 3rd Qu.:0.00000
Max. :28.0000 Max. :75.0000 Max. :5.00000
day_talk_revisions day_user_revisions week_revisions
Min. : 0.0000 Min. : 0.0000 Min. : 0.000
1st Qu.: 0.0000 1st Qu.: 0.0000 1st Qu.: 0.000
Median : 0.0000 Median : 0.0000 Median : 0.000
Mean : 0.1015 Mean : 0.2153 Mean : 1.179
3rd Qu.: 0.0000 3rd Qu.: 0.0000 3rd Qu.: 1.000
Max. :148.0000 Max. :149.0000 Max. :149.000
week_reverted_main_revisions week_main_revisions week_wp_revisions
Min. : 0.0000 Min. : 0.0000 Min. : 0.00000
1st Qu.: 0.0000 1st Qu.: 0.0000 1st Qu.: 0.00000
Median : 0.0000 Median : 0.0000 Median : 0.00000
Mean : 0.2616 Mean : 0.8083 Mean : 0.01743
3rd Qu.: 0.0000 3rd Qu.: 0.0000 3rd Qu.: 0.00000
Max. :28.0000 Max. :90.0000 Max. :13.00000
week_talk_revisions week_user_revisions surviving sessions
Min. : 0.0000 Min. : 0.0000 False:4094 Min. : 0.0000
1st Qu.: 0.0000 1st Qu.: 0.0000 True : 95 1st Qu.: 0.0000
Median : 0.0000 Median : 0.0000 Median : 0.0000
Mean : 0.1115 Mean : 0.2614 Mean : 0.4249
3rd Qu.: 0.0000 3rd Qu.: 0.0000 3rd Qu.: 1.0000
Max. :148.0000 Max. :149.0000 Max. :18.0000
time_spent_editing
Min. : 0.0
1st Qu.: 0.0
Median : 0.0
Mean : 469.3
3rd Qu.: 410.0
Max. :40092.0
Looks like our users are relatively balanced. 22% of our users registered via mobile and and *probably* be discarded. All of the week measures and "surviving" are invalid due to the fact that we haven't been able to observe them for a week yet. ...and our timestamp is misinterpreted, so I'm going to have to clean that up. --Halfak (WMF) (talk) 17:05, 27 May 2015 (UTC)
Drumroll please:
bucket via_mobile editing.k productive.k n 1: experimental 0 534 272 1627 2: control 0 544 271 1626 3: control 1 117 59 454 4: experimental 1 115 51 482
So, no big deviations there. I can eyeball them as non-significant. That's an expected outcome at this point. Tomorrow I'm going to be looking at burden and log events. Until then. --Halfak (WMF) (talk) 17:31, 27 May 2015 (UTC)