Template A/B testing/ImageTaggingBot Analyses

From Meta, a Wikimedia project coordination wiki

Overview[edit]

ImageTaggingBot User Page


Templates invlolved in the experiment: z131 - z142 [1]


Timeframe: 2011-12-21 - 2012-02-21


Analyses Results[edit]

Counts: (Z-template - count)

131 - 107
132 - 107
133 - 114
134 - 207
135 - 188
136 - 219
137 - 42
138 - 41
139 - 43
140 - 18
141 - 23
142 - 24

Total Control: 374
Total Test: 759


Edit Activity[edit]

Case: z131 vs. z133[edit]

Below is the measurement of all users that went on to make at least one edit in namespace 6 after receiving the ImageTagging template given that they were not blocked after seeing the template.

Test response: 9.649%

Control response: 18.69%


Modelling Analysis, Registered Users, z133 (test) vs z131 (control) - R Output

Call:
glm(formula = template ~ metric, family = binomial(link = "logit"), 
    data = temp_df)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.250  -1.250   1.107   1.107   1.440  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)  
(Intercept)   0.1688     0.1456   1.159   0.2463  
metric       -0.7667     0.4026  -1.904   0.0569 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 306.15  on 220  degrees of freedom
Residual deviance: 302.37  on 219  degrees of freedom
AIC: 306.37

Number of Fisher Scoring iterations: 4

Test Users that edited namespace 6 after the posting[edit]

 [1] "Bolidist"              "Chrisyasay"            "Dborase"              
 [4] "Dj Ran"                "Dpsthota11"            "Einie101"             
 [7] "Erapo"                 "Fuse809"               "HahnMahlay"           
[10] "Hlakungpui"            "Iambonusboy"           "Ilovethe2000s"        
[13] "Imhungry4444"          "Iviscera"              "Jazzbeard"            
[16] "JLara-cl"              "Jrobin08"              "Kennethcooke609"      
[19] "Kimmeleshkolot"        "Kracoukas"             "Maslusar"             
[22] "MasterSmcd"            "Mishrasanjeew"         "NEWBRAIN"             
[25] "NickSCFC"              "Peppage"               "Pg 6475"              
[28] "Purificationwiki"      "QwerpoiXX"             "RTBoughner"           
[31] "Salilpawar1"           "Smallzc"               "Swaggie73"            
[34] "The 5th Silver Beatle" "TheBigJagielka"        "Timepricer"           
[37] "Wadamz"                "Warriorpoet23"         "WISEPHAROHPSN"        
[40] "Zachary.burrows"


Control Users that edited namespace 6 after the posting[edit]

 [1] "1000 Volts"            "Ansonjae"              "Balbon32"             
 [4] "Bloodios"              "Bubertov"              "D20120101"            
 [7] "Dborase"               "Dertogada"             "Diango"               
[10] "Duncan3dc"             "Emix.G.A"              "Foxonline"            
[13] "Fuse809"               "Galwaybuck"            "Hafeezullah2k"        
[16] "Ilovethe2000s"         "Iviscera"              "Jazzbeard"            
[19] "JLara-cl"              "K.yakovleva"           "Kennethcooke609"      
[22] "KingFredrick VI"       "Klr8"                  "Kracoukas"            
[25] "KristineValdheima"     "Malekalameh"           "MasterSmcd"           
[28] "MelanieBrown"          "MikelZap"              "Mohdwasi"             
[31] "Monica982"             "Mthen"                 "Muzammil786"          
[34] "Naveen Ramanathan"     "NEWBRAIN"              "Nick Rokk"            
[37] "Pg 6475"               "Pilch 84"              "Polo phil"            
[40] "Prz4587ill"            "Purificationwiki"      "Rcoyy"                
[43] "Rsamahamed"            "RTBoughner"            "Scribblerman"         
[46] "Smallzc"               "Sorayanwayne"          "SpongePappy"          
[49] "Sumitkumarjha75"       "Swaggie73"             "The 5th Silver Beatle"
[52] "TheDarkPyrano100"      "Victorianny"           "Voravitjessica"       
[55] "Wadamz"                "Willtoco"              "Zavila"               
[58] "Zywxn"


Blocking[edit]

Case: z131 vs. z132 (test) -- This result is no longer valid. For documentation purposes only. [edit]

Number of blocks after the posting was a marginally significant (p-value = .145) predictor of the template. The control template had a lower rate.


Test average blocks / user after posting: 0.06854839

Control average blocks / user after posting: 0.03212851


Modelling Analysis, Registered Users, z132 (test) vs z131 (control) - R Output

Call:
glm(formula = template ~ blocks_after, family = binomial(link = "logit"), 
    data = all_data)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.651  -1.165  -1.165   1.190   1.190  

Coefficients:
             Estimate Std. Error z value Pr(>|z|)
(Intercept)  -0.03018    0.09136  -0.330    0.741
blocks_after  0.54926    0.37683   1.458    0.145

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 688.99  on 496  degrees of freedom
Residual deviance: 686.60  on 495  degrees of freedom
AIC: 690.6

Number of Fisher Scoring iterations: 4


Case: z134 vs. z135 (test) -- This result is no longer valid. For documentation purposes only. [edit]

Number of blocks after the posting was a marginally significant (p-value = .120) predictor of the template. The test template had a lower rate.


Test average blocks / user after posting: 0.002645503

Control average blocks / user after posting: 0.01703163


Modelling Analysis, Registered Users, z132 (test) vs z131 (control) - R Output

Call:
glm(formula = template ~ blocks_after, family = binomial(link = "logit"), 
    data = all_data)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.147  -1.147  -1.147   1.208   1.931  

Coefficients:
             Estimate Std. Error z value Pr(>|z|)
(Intercept)  -0.07146    0.07156  -0.999    0.318
blocks_after -1.62419    1.04421  -1.555    0.120

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1092.4  on 788  degrees of freedom
Residual deviance: 1088.6  on 787  degrees of freedom
AIC: 1092.6

Number of Fisher Scoring iterations: 4


Case: z137 vs. z139 (test) -- This result is no longer valid. For documentation purposes only. [edit]

Number of blocks after the posting was a marginally significant (p-value = .19) predictor of the template. The test template had a lower rate.


Test average blocks / user after posting: 0.03947368

Control average blocks / user after posting: 0.109589


Modelling Analysis, Registered Users, z132 (test) vs z131 (control) - R Output

Call:
glm(formula = template ~ blocks_after, family = binomial(link = "logit"), 
    data = all_data)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.218  -1.218   1.137   1.137   1.494  

Coefficients:
             Estimate Std. Error z value Pr(>|z|)
(Intercept)   0.09591    0.16882   0.568     0.57
blocks_after -0.81601    0.62203  -1.312     0.19

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 206.50  on 148  degrees of freedom
Residual deviance: 204.48  on 147  degrees of freedom
AIC: 208.48

Number of Fisher Scoring iterations: 4


Overall Warnings[edit]

Average warnings / user before 0.8744589 and afte 0.8556999.


Fraction of users warned before: 0.4300144

Fraction of users warned after: 0.3347763


Th latter is a significant difference.