Research talk:VisualEditor's effect on newly registered editors/May 2015 study/Work log/2015-05-27

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search

Wednesday, May 27, 2015[edit]

Today I am performing a preliminary analysis of the pilot study. The bucketing for this study started and ended three days ago, so that means I won't have a full week with which to review newcomers contributions, but we should be able to looks for anomalies in their first 24-48 hours just fine.

Bucketed users[edit]

So, first things first, I need to gather the set of bucketed users and label them by condition.

> SELECT event_userId AS user_id, timestamp AS registration, event_displayMobile AS via_mobile FROM ServerSideAccountCreation_5487345 WHERE wiki = "enwiki" AND event_isSelfMade AND timestamp BETWEEN "2015052115" and "2015052215" LIMIT 10;
+----------+----------------+------------+
| user_id  | registration   | via_mobile |
+----------+----------------+------------+
| 25251662 | 20150521150038 |          0 |
| 25251673 | 20150521150153 |          0 |
| 25251675 | 20150521150208 |          1 |
| 25251680 | 20150521150245 |          0 |
| 25251684 | 20150521150309 |          0 |
| 25251687 | 20150521150343 |          0 |
| 25251689 | 20150521150427 |          0 |
| 25251690 | 20150521150430 |          0 |
| 25251696 | 20150521150500 |          0 |
| 25251700 | 20150521150502 |          1 |
+----------+----------------+------------+
10 rows in set (0.00 sec)

OK. Let's make sure I have the time bounds right before we go any farther.

> SELECT LEFT(user_registration, 10) AS hour, SUM(ve.up_user IS NOT NULL)/COUNT(*) AS ve_prop FROM user LEFT JOIN user_properties AS ve ON up_user = user_id AND up_property = 'visualeditor-enable' WHERE user_registration BETWEEN "2015052114" AND "2015052217" AND user_id > 25241662 GROUP BY 1;
+------------+---------+
| hour       | ve_prop |
+------------+---------+
| 2015052114 |  0.0073 |
| 2015052115 |  0.3188 |
| 2015052116 |  0.3147 |
| 2015052117 |  0.3268 |
| 2015052118 |  0.3298 |
| 2015052119 |  0.3091 |
| 2015052120 |  0.2739 |
| 2015052121 |  0.3302 |
| 2015052122 |  0.3030 |
| 2015052123 |  0.3299 |
| 2015052200 |  0.3322 |
| 2015052201 |  0.3681 |
| 2015052202 |  0.3713 |
| 2015052203 |  0.3535 |
| 2015052204 |  0.3401 |
| 2015052205 |  0.3728 |
| 2015052206 |  0.3371 |
| 2015052207 |  0.3651 |
| 2015052208 |  0.3291 |
| 2015052209 |  0.3144 |
| 2015052210 |  0.3326 |
| 2015052211 |  0.3450 |
| 2015052212 |  0.3348 |
| 2015052213 |  0.3154 |
| 2015052214 |  0.3300 |
| 2015052215 |  0.0249 |
| 2015052216 |  0.0079 |
+------------+---------+
27 rows in set (0.10 sec)

Looks like the deployments are tight to the hour. Good. Now to confirm which users were bucketed.

> SELECT user_id % 2 = 0 AS even_id, LEFT(user_registration, 10) AS hour, SUM(ve.up_user IS NOT NULL)/COUNT(*) AS ve_prop FROM user LEFT JOIN user_properties AS ve ON up_user = user_id AND up_property = 'visualeditor-enable' WHERE user_registration BETWEEN "2015052115" AND "2015052118" AND user_id > 25241662 GROUP BY 1, 2;
+---------+------------+---------+
| even_id | hour       | ve_prop |
+---------+------------+---------+
|       0 | 2015052115 |  0.0109 |
|       0 | 2015052116 |  0.0000 |
|       0 | 2015052117 |  0.0036 |
|       1 | 2015052115 |  0.6268 |
|       1 | 2015052116 |  0.6295 |
|       1 | 2015052117 |  0.6489 |
+---------+------------+---------+
6 rows in set (0.04 sec)

OK. it is clearly newcomers with an even user_id. Now to add that to my user extraction query.

> SELECT event_userId AS user_id, IF(event_userId % 2 = 0, "experimental", "control") AS bucket, timestamp AS registration, event_displayMobile AS via_mobile FROM ServerSideAccountCreation_5487345 WHERE wiki = "enwiki" AND event_isSelfMade AND timestamp BETWEEN "2015052115" and "2015052215" LIMIT 10;
+----------+--------------+----------------+------------+
| user_id  | bucket       | registration   | via_mobile |
+----------+--------------+----------------+------------+
| 25251662 | experimental | 20150521150038 |          0 |
| 25251673 | control      | 20150521150153 |          0 |
| 25251675 | control      | 20150521150208 |          1 |
| 25251680 | experimental | 20150521150245 |          0 |
| 25251684 | experimental | 20150521150309 |          0 |
| 25251687 | control      | 20150521150343 |          0 |
| 25251689 | control      | 20150521150427 |          0 |
| 25251690 | experimental | 20150521150430 |          0 |
| 25251696 | experimental | 20150521150500 |          0 |
| 25251700 | experimental | 20150521150502 |          1 |
+----------+--------------+----------------+------------+

Cool. Now to include whether ve_enabled property.

SELECT event_userId AS user_id, IF(event_userId % 2 = 0, "experimental", "control") AS bucket, timestamp AS registration, event_displayMobile AS via_mobile, ve.up_user IS NOT NULL AS ve_enabled FROM ServerSideAccountCreation_5487345 LEFT JOIN enwiki.user_properties ve ON event_userId = up_user AND up_property = 'visualeditor-enable' WHERE wiki = "enwiki" AND event_isSelfMade AND timestamp BETWEEN "2015052115" and "2015052215" LIMIT 10;
+----------+--------------+----------------+------------+------------+
| user_id  | bucket       | registration   | via_mobile | ve_enabled |
+----------+--------------+----------------+------------+------------+
| 25251662 | experimental | 20150521150038 |          0 |          1 |
| 25251673 | control      | 20150521150153 |          0 |          0 |
| 25251675 | control      | 20150521150208 |          1 |          0 |
| 25251680 | experimental | 20150521150245 |          0 |          1 |
| 25251684 | experimental | 20150521150309 |          0 |          1 |
| 25251687 | control      | 20150521150343 |          0 |          0 |
| 25251689 | control      | 20150521150427 |          0 |          0 |
| 25251690 | experimental | 20150521150430 |          0 |          1 |
| 25251696 | experimental | 20150521150500 |          0 |          1 |
| 25251700 | experimental | 20150521150502 |          1 |          1 |
+----------+--------------+----------------+------------+------------+
10 rows in set (0.00 sec)

Cool. It looks like we're enabling VE for mobile registrations. That'll be noise if they don't eventually edit from the desktop (no VE on mobile yet), so I'll keep 'em flagged so we can filter them out in the analysis later.

SELECT 
  event_userId AS user_id, 
  IF(event_userId % 2 = 0, "experimental", "control") AS bucket, 
  timestamp AS registration, 
  event_displayMobile AS via_mobile, 
  ve.up_user IS NOT NULL AS ve_enabled 
FROM log.ServerSideAccountCreation_5487345 
LEFT JOIN enwiki.user_properties ve ON 
  event_userId = up_user AND 
  up_property = 'visualeditor-enable' 
WHERE 
  wiki = "enwiki" AND 
  event_isSelfMade AND 
  timestamp BETWEEN "2015052115" and "2015052215";
$ wc datasets/pilot_users.tsv 
  4190  20950 161399 datasets/pilot_users.tsv

--Halfak (WMF) (talk) 12:43, 27 May 2015 (UTC)

Metrics![edit]

OK. Time to extract some metrics.

I used this library https://github.com/halfak/mwmetrics, to run the following bit:

datasets/pilot_user_metrics.tsv: datasets/pilot_users.tsv
	cat datasets/pilot_users.tsv | \
	mwmetrics new_users enwiki $(dbstore) > \
	datasets/pilot_user_metrics.tsv

It didn't take long at all. Now, I've created a repo for my own code[1] and loaded up my work in R.

> summary(user_metrics)
    user_id                  bucket      registration         via_mobile    
 Min.   :25251662   control     :2080   Min.   :2.015e+13   Min.   :0.0000  
 1st Qu.:25254181   experimental:2109   1st Qu.:2.015e+13   1st Qu.:0.0000  
 Median :25256579                       Median :2.015e+13   Median :0.0000  
 Mean   :25256593                       Mean   :2.015e+13   Mean   :0.2234  
 3rd Qu.:25259016                       3rd Qu.:2.015e+13   3rd Qu.:0.0000  
 Max.   :25261540                       Max.   :2.015e+13   Max.   :1.0000  
   ve_enabled     user_registration   day_revisions    
 Min.   :0.0000   Min.   :2.015e+13   Min.   :  0.000  
 1st Qu.:0.0000   1st Qu.:2.015e+13   1st Qu.:  0.000  
 Median :1.0000   Median :2.015e+13   Median :  0.000  
 Mean   :0.5058   Mean   :2.015e+13   Mean   :  0.925  
 3rd Qu.:1.0000   3rd Qu.:2.015e+13   3rd Qu.:  1.000  
 Max.   :1.0000   Max.   :2.015e+13   Max.   :149.000  
 day_reverted_main_revisions day_main_revisions day_wp_revisions 
 Min.   : 0.0000             Min.   : 0.0000    Min.   :0.00000  
 1st Qu.: 0.0000             1st Qu.: 0.0000    1st Qu.:0.00000  
 Median : 0.0000             Median : 0.0000    Median :0.00000  
 Mean   : 0.2344             Mean   : 0.6469    Mean   :0.01146  
 3rd Qu.: 0.0000             3rd Qu.: 0.0000    3rd Qu.:0.00000  
 Max.   :28.0000             Max.   :75.0000    Max.   :5.00000  
 day_talk_revisions day_user_revisions week_revisions   
 Min.   :  0.0000   Min.   :  0.0000   Min.   :  0.000  
 1st Qu.:  0.0000   1st Qu.:  0.0000   1st Qu.:  0.000  
 Median :  0.0000   Median :  0.0000   Median :  0.000  
 Mean   :  0.1015   Mean   :  0.2153   Mean   :  1.179  
 3rd Qu.:  0.0000   3rd Qu.:  0.0000   3rd Qu.:  1.000  
 Max.   :148.0000   Max.   :149.0000   Max.   :149.000  
 week_reverted_main_revisions week_main_revisions week_wp_revisions 
 Min.   : 0.0000              Min.   : 0.0000     Min.   : 0.00000  
 1st Qu.: 0.0000              1st Qu.: 0.0000     1st Qu.: 0.00000  
 Median : 0.0000              Median : 0.0000     Median : 0.00000  
 Mean   : 0.2616              Mean   : 0.8083     Mean   : 0.01743  
 3rd Qu.: 0.0000              3rd Qu.: 0.0000     3rd Qu.: 0.00000  
 Max.   :28.0000              Max.   :90.0000     Max.   :13.00000  
 week_talk_revisions week_user_revisions surviving       sessions      
 Min.   :  0.0000    Min.   :  0.0000    False:4094   Min.   : 0.0000  
 1st Qu.:  0.0000    1st Qu.:  0.0000    True :  95   1st Qu.: 0.0000  
 Median :  0.0000    Median :  0.0000                 Median : 0.0000  
 Mean   :  0.1115    Mean   :  0.2614                 Mean   : 0.4249  
 3rd Qu.:  0.0000    3rd Qu.:  0.0000                 3rd Qu.: 1.0000  
 Max.   :148.0000    Max.   :149.0000                 Max.   :18.0000  
 time_spent_editing
 Min.   :    0.0   
 1st Qu.:    0.0   
 Median :    0.0   
 Mean   :  469.3   
 3rd Qu.:  410.0   
 Max.   :40092.0

Looks like our users are relatively balanced. 22% of our users registered via mobile and and *probably* be discarded. All of the week measures and "surviving" are invalid due to the fact that we haven't been able to observe them for a week yet. ...and our timestamp is misinterpreted, so I'm going to have to clean that up. --Halfak (WMF) (talk) 17:05, 27 May 2015 (UTC)


Drumroll please:

         bucket via_mobile editing.k productive.k    n
1: experimental          0       534          272 1627
2:      control          0       544          271 1626
3:      control          1       117           59  454
4: experimental          1       115           51  482

So, no big deviations there. I can eyeball them as non-significant. That's an expected outcome at this point. Tomorrow I'm going to be looking at burden and log events. Until then. --Halfak (WMF) (talk) 17:31, 27 May 2015 (UTC)