Template A/B testing/Welcome Templates Analyses

Analyses Results[edit]

PT Welcome Templates[edit]

Experimental templates: z4, z5

Number of non-duplicate z4 (control) recipients: 232

Number of non-duplicate z5 (test) recipients: 197

Fraction of recipients that edited after (test): 22.34%

Fraction of recipients that edited after (control): 25.43%

Result Conclusive? NO

Fraction of recipients that edited after (test) - filtered on users that made at least one edit before posting: 30.77%

Fraction of recipients that edited after (control) - filtered on users that made at least one edit before posting: 42.59%

Result Conclusive? Yes (marginal - 91.33%)

Fraction of recipients that were blocked after (test): 2.538%

Fraction of recipients that were blocked after (control): 3.448%

Result Conclusive? NO

Logistic Regression Analysis on edits events after the posting, Registered Users - R Output


Call:
glm(formula = template ~ metric, family = binomial(link = "logit"), 
    data = temp_df)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.126  -1.126  -1.056   1.230   1.304  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)
(Intercept)  -0.1229     0.1110  -1.107    0.268
metric       -0.1705     0.2280  -0.748    0.455

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 591.86  on 428  degrees of freedom
Residual deviance: 591.30  on 427  degrees of freedom
AIC: 595.3

Number of Fisher Scoring iterations: 3

[1] "Samples in test:"
[1] 197

[1] "Samples in control:"
[1] 232

[1] "Summary of metric for test:"
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0000  0.0000  0.0000  0.2234  0.0000  1.0000 

[1] "Summary of metric for control:"
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0000  0.0000  0.0000  0.2543  1.0000  1.0000

Logistic Regression Analysis on blocked users after the posting, Registered Users, Edit Event filtered on at least one edit before posting - R Output


Call:
glm(formula = template ~ metric, family = binomial(link = "logit"), 
    data = temp_df)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-1.1842  -1.1842  -0.9751   1.1706   1.3942  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)  
(Intercept)   0.0160     0.1789   0.089   0.9287  
metric       -0.5124     0.2991  -1.713   0.0867 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 274.42  on 198  degrees of freedom
Residual deviance: 271.44  on 197  degrees of freedom
AIC: 275.44

Number of Fisher Scoring iterations: 4

[1] "Samples in test:"
[1] 91

[1] "Samples in control:"
[1] 108

[1] "Summary of metric for test:"
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0000  0.0000  0.0000  0.3077  1.0000  1.0000 

[1] "Summary of metric for control:"
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0000  0.0000  0.0000  0.4259  1.0000  1.0000

Logistic Regression Analysis on blocked users after the posting, Registered Users - R Output


Call:
glm(formula = template ~ metric, family = binomial(link = "logit"), 
    data = temp_df)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.112  -1.112  -1.112   1.244   1.364  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.15541    0.09816  -1.583    0.113
metric      -0.27373    0.54028  -0.507    0.612

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 591.86  on 428  degrees of freedom
Residual deviance: 591.60  on 427  degrees of freedom
AIC: 595.6

Number of Fisher Scoring iterations: 3

[1] "Samples in test:"
[1] 197

[1] "Samples in control:"
[1] 232

[1] "Summary of metric for test:"
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.00000 0.00000 0.00000 0.02538 0.00000 1.00000 

[1] "Summary of metric for control:"
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.00000 0.00000 0.00000 0.03448 0.00000 2.00000