Template A/B testing/Twinkle Analyses

From Meta, a Wikimedia project coordination wiki

Overview[edit]

The Twinkle testing experiment descriptions can be found here.

The experiments themselves involved comparing template sets including a control and a test template. The control templates are made up of existing Huggle warning templates in use in the community while the test templates represent modified versions in measurable ways (length of warning, directives, personalization etc.) - the experiment definition can be found here.

Whether the changes made to test templates had an effect was primarily assessed using logistic regression analysis in R. Editor groups were pre-filtered based on:

  • registered / non-registered
  • minimum number of deleted edits made before the posting
  • maximum number of deleted edits made before the posting
  • minimum number of deleted edits made after the posting (usually at least 1 .. maybe)

The results of these analyses along with the conditions on the editor activity samples can be seen in the experiment results in the next section.

In the plots accompanying the analyses the effect on the mean ratio of edits before and after the posting as well as the mean absolute number of edits made after the posting are measured against the minimum number of edits before the posting used to select the sample group. Accompanying these plots are those including the sample sizes for each data point in the edit trend (both control and test).

Analyses Results[edit]

Data Munging / Filtering: 

Looking at all edit events
Blocks after = 0 (no blocks after seeing template)
namespace = 0
first_warning = TRUE

Findings:

The test was performed among registered users only as this group dominated the samples.  Instances of edits in Project namespace after the posting were measured.  There was not a significant effect measured between the test and control however the control did have a higher rate of edit events in this namespace.


Modelling Analysis, Registered Users, Instances of Edit Events in Namespace 4 (Project), z81 vs. z82 - R Output

execute.main(min_edits_before = 0, max_edits_before = Inf, registered = TRUE)

Call:
glm(formula = template ~ edit_event, family = binomial(link = "logit"), 
    data = all_data)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.269  -1.269   1.089   1.089   1.151  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)  
(Intercept)  0.21211    0.08736   2.428   0.0152 *
edit_event  -0.15074    0.17943  -0.840   0.4008  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 955.33  on 692  degrees of freedom
Residual deviance: 954.62  on 691  degrees of freedom
AIC: 958.62

Number of Fisher Scoring iterations: 3


Test Edit Events: 84 users edited in Project out of 377 Samples
Control Edit Events: 79 users edited in Project out of 316 Samples


Modelling Analysis, Registered Users, Edit Counts in Namespace 4 (Project), z81 vs. z82 - R Output

execute.main(min_edits_before = 0, max_edits_before = Inf, registered = TRUE)

Call:
glm(formula = template ~ edit_counts_all, family = binomial(link = "logit"), 
    data = all_data)

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.257  -1.257   1.100   1.100   1.523  

Coefficients:
                  Estimate Std. Error z value Pr(>|z|)  
(Intercept)      0.1843266  0.0768352   2.399   0.0164 *
edit_counts_all -0.0007911  0.0009528  -0.830   0.4064  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 955.33  on 692  degrees of freedom
Residual deviance: 954.59  on 691  degrees of freedom
AIC: 958.59

Number of Fisher Scoring iterations: 3


Test Edit Counts (377 Samples):

    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
   0.000    0.000    0.000    7.493    0.000 1223.000 


Test Edit Counts (316 Samples):

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   0.00    0.00    0.00   13.04    0.25 1391.00 


Plots[edit]

Plots - AfD (z78, z79) - edits after (0-3 days) and normalized edits after (0-3 days)


Summary[edit]

AfD[edit]

Legend of Terms:
================


Variables:

Rev_Deleted_Instances - Number of deleted revisions 0-3 days after posting
Edit_Event_NS_4 - Did the user edit project after seeing the template? (1 or 0)
Edit_Count_NS_4 - Edit counts in Prject namespace after posting


Registered Variable(s) Control Result Test Result Winner % Increase Confidence Sample Size Params
TRUE Edit_Event_NS_4 79 / 316 84 / 377 control (inconclusive) -- ~ 60% 377 (test), 316 (control) execute.main(min_edits_before = 0, max_edits_before = Inf, registered = TRUE)
TRUE Edit_Count_NS_4 13.04 7.493 control (inconclusive) -- ~ 60% 377 (test), 316 (control) execute.main(min_edits_before = 0, max_edits_before = Inf, registered = TRUE)


PROD[edit]

Below are the lists of users along with rev ids and rev timestamps of any subsequent revisions involving the PROD tag (e.g. "Proposed Deletion").


User List, Editors that received template warning and PROD tag and subsequent revisions involving tag, z78 vs. z79 - Control Group

user	rev_timestamp	rev_id
Lauranic	None	None
Piroroadkill	None	None
WotWeiller	None	None
WLU	None	None
Alex vamos 99	None	None
Baerc234	None	None
Dnfenner	None	None
Nika1693	None	None
Lyndonid	None	None
Serge.Belongie	None	None
Captain Q	None	None
GregoryErnst	None	None
Katrina Angela Gonzales	None	None
Tossnara	None	None
Sukhraj brar	None	None
Krishnapersonal	None	None
Chehticham	None	None
Lawqu	None	None
Kaswardy	None	None
Dragon Genius	None	None
Yammzsi	None	None
Keelyboom	None	None
Sonore	None	None
Almondb	None	None
CellarMist	None	None
Danger	478923477	20120226103723
HMHunt	None	None
Shaktipatel88	None	None
NNU-1-17100215	None	None
Mech5107	None	None
IGG8998	None	None
Simply14u	None	None
Alokprasad84	None	None
Whofan44	None	None
TehZorroness	None	None
Payyan-2000	None	None
Bealtes6789	None	None
Ziddirazan	None	None
KoolKellyGreen	None	None
Tomatool	None	None
Brandonandshae\'s	None	None
Vicentenavarre2	None	None
Speedtrace	None	None
Prentert	None	None
Adityavisionrx	None	None
Dell.bilik	None	None
Andrew22k	None	None
Frozenbrains	None	None
Bookworm857158367	None	None
Quidster4040	None	None
Mdl4235	None	None
Lizziecassmaran	None	None
Poppopcreator	None	None
YolentaShield	None	None
Uhl2016	None	None
Wetdogmeat	460353375	20111112224500
Wetdogmeat	460353779	20111112224813
Cwinslet	None	None
Biggleswiki	None	None
Joshua Guest	None	None
Gurjinder.Romana	None	None
Makky76	None	None
Siobhanhsm	None	None
TheAztec	None	None
Colorfulthrowup	None	None
Anshulkumardhiman	None	None
Pooltv	None	None
Jkim ucb	None	None
Damienfernando	None	None
SocialMarketingVA	None	None
Supuhstar	None	None
Robrox89	None	None
Kooh123	None	None
Ksimola	None	None
Bala8vijay	None	None
Tofer138	None	None
Kylemmenard	None	None
Universityuser	None	None
Edina-k	None	None
Benjaminevans82	None	None
Z\xc3\xbcrich1909	None	None
Magedogtag	None	None
Niko1297	None	None
James andrew lewis	None	None
Triwbe	None	None
Shiva Khanal	None	None
Cwilson66	None	None
Kerachian	None	None
Demonz587	None	None
Robotic564	None	None
Alexsautographs	None	None
Oi8uhybtfr5	None	None
Ilove96	None	None
John Abento	None	None
Kurtnardone	None	None
Roznar97	None	None
Stjetters	None	None
Horsey fart	None	None
Tward007uk	None	None
December2011	None	None
SolomanMcKenzie	None	None
Elsot39	None	None
142.227.207.193	None	None
SlickSy	None	None
Aus eagle	None	None
Adbennett	None	None
Fat pig73	None	None
Rajat Ghai	None	None
Teaandletters	None	None
Fez1321	None	None
Dipankan001	None	None
Kdianed	None	None
Sailor iain	None	None
Dagnytaggartmoxie	None	None
Astertreff	None	None
Asmolnikov02091997	None	None
Severnosianie	None	None
DJNutty	None	None
Pitchka	None	None
Tempurafried	None	None
Anna.kudlay	None	None
TheCustomOfLife	None	None
Dreamofpeaceinschool	None	None
QueenDemi	None	None
Travismac3	None	None
71.184.204.92	None	None
Katrinahalili	None	None
Jsteele12	None	None
NNU-11-zhanghongjing	None	None
Xploiter	None	None
Dailta	None	None
Abdulghaffoor	None	None
Charles Fitzgerald III	None	None
Homad89	None	None
Dravecky	465011371	20111209203327
Earl Andrew	None	None
MoQazalbash	None	None
Fracamauthor	None	None
Olesachem	None	None
ToranL7231	None	None
Ikestarr	None	None
Mr. D. E. Mophon	None	None
Peruginionio	None	None
Danieljpost	None	None
Muehlenau	None	None
Tigerbreath13	None	None
Rimianika	None	None
Ministrymedia	None	None
Matthead	None	None
Kryssantos1130	None	None
MobileMarketWatch	None	None
Lawbrarian	None	None
Richard Brooks MK	None	None
Nmyers223	None	None
Georgejdorner	None	None
Even23	None	None
Iuhoosier66	None	None
Le Bosch	None	None
TwinTowersNYC	None	None
LucileGoldstien	None	None
Lucia Black	None	None
Glovefish	None	None
Udaymanchanda	None	None
Mjohn127	None	None
Wtimrock	None	None
Tahliaheffernan	None	None
Rex10098	None	None
Kevinpcarter	None	None
Nethalcon	None	None
Gluontronic	None	None
Vasim mirza	None	None
Galworld	None	None
Rpirelli	None	None
Ksawatzky	None	None
Hellosmiles	None	None
Jackevee	None	None
Roshaniel21	None	None
Peter McE	None	None
Cazcazcaz	None	None
Gengel	None	None
DCabot	None	None
Ohjough	None	None
Ksfol9	None	None
Dustinslush	None	None
Suro seko	None	None
ProjectManhattan	None	None
Daniel.Vitor1	None	None
Pinkfloyd95209	None	None
Tripilohit	None	None
Ravindrajith	None	None
Mach240390	None	None
Mncline	None	None
RTG	None	None
Ath1973	None	None
Ferdinandude	None	None
Thefootballweekly	None	None
Renate.nyborg	None	None
Soccerzchelzlim	None	None
Amosbatto	None	None
OP CALIFORNIA	None	None
Fanfan1964	None	None
Dcampbell2011	None	None
Sirmikey111	None	None
QuixoteArmy	None	None
Double0agent	None	None
Vijila medknow	None	None
Mindchained	None	None
Tictictalk16	None	None
Kamolga22	None	None
Hiphopreina	None	None
Dlbu226	None	None
Bjdav23	None	None
Starzynka	None	None
Bootslewis	None	None
TatyanaR	None	None
Veronicarevier	None	None
Vmanzon	None	None
Ainsworth anderson	None	None
Chadialameddine	None	None
DrMattKerry	None	None
Ianhowlett	None	None
Theshywillraindeath	None	None
Nicholaus.hughes	None	None
Pioneersbrog	None	None
Danisamsoel	None	None
Unidue.hilfskraft	None	None
The Night\'s Watch	None	None
JournoJoeMcG	None	None
Applesnbananas	None	None
Beanscene	None	None
Neelix	463020495	20111129031721
Neelix	463021579	20111129032243
Neelix	468129185	20111228185537
Neelix	468138476	20111228195039
Neelix	468337894	20111229211251
Neelix	471074714	20120113024046
Neelix	472449337	20120121162552
Neelix	472450395	20120121163222
Neelix	472654157	20120122185334
Neelix	475322220	20120206013818
Neelix	475325976	20120206021713
Neelix	475504978	20120207021730
Neelix	476450492	20120212141311
Neelix	476752899	20120214011928
Neelix	477437157	20120217220647
Neelix	477644025	20120219022352
Neelix	478263108	20120222154038
Neelix	478264532	20120222155147
Neelix	478471933	20120223191544
Neelix	478520639	20120224002705
Neelix	479992997	20120303155650
Neelix	479997273	20120303162610
Neelix	482255945	20120316204835
Neelix	482793435	20120319215638
Neelix	484434246	20120328205919
Neelix	484470425	20120329013635
Neelix	484624098	20120330003106
Neelix	485229847	20120402212110
Neelix	487356157	20120414161709
Neelix	487716616	20120416193214
Neelix	487730610	20120416210152
Helpingfriendlybook	None	None
Theunknown24113	None	None
Fealiath	None	None
Elif Dervi\xc5\x9f	None	None
Kapilcm	None	None
Clydemaster555	None	None
Cwwiss	None	None
MatthewJS	None	None
Gdr05a	None	None
Bucsfannva	None	None
Stadiumseating	None	None
Mr.eurosong	None	None
Aer1n2026	None	None
Dogru144	None	None
Assassin\'s Creed	None	None
Bodwysty	None	None
Picard09	None	None
Gocek	None	None
Maccer	None	None
Michaelloves80sMusic	None	None
Hollywoodsuccessstory	None	None
Sliceservace	None	None
AlexSwanson	None	None
MJF2000	None	None
Yify	None	None
Mchaumont	None	None
Mikemor92	487583692	20120415234135
Dimitry-1983	None	None
Ram548	None	None
Kate Conroy	None	None
Pakhack	None	None
Theseus1776	477402188	20120217182246
Indiantvlover	None	None
Subratadhara	None	None
Luizfmaf	None	None
Alberoni otti	None	None
Cherrybabascrazy	None	None
Ffkks	None	None
Smukkai	None	None
Pepe061270	None	None
Hammster16	None	None
Gograded	None	None
Gjr rodriguez	None	None
Cost 4000	None	None
Jennybrown88	None	None
Vini1621	None	None
Vshehu	None	None
Coin945	464034636	20111204123915
Kukulich	None	None
Stgblu	None	None
Albanian222	None	None
Skullcrusher98	None	None
Battousai86	None	None
Entsal007	None	None
M@sk	None	None
Cassandralisa	None	None
Checkee	None	None
Shanghaitucker	None	None
Elinraun	None	None
Tyganlion	None	None
Ashpreet92	None	None
MsTingaK	None	None
BiT	None	None
Barraque	None	None
Hlearles	None	None
Dcarranza	None	None
Bridgetlin	None	None
88guy88	None	None
Adrianvirr	None	None
OddibeKerfeld	None	None
Raulaudelo	None	None
213.253.40.156	None	None
Manarg	None	None
Lynch3001	None	None
128.115.87.44	None	None
Unseen45	None	None
Brycenbeck	None	None
\xd0\x9d\xd0\xbe\xd1\x85\xd1\x87\xd0\x9e	None	None
Dave9862	None	None
Emergencyslider	None	None
Skycloud86	None	None
Gv23	None	None
Lolman1023	None	None
Michael ricardo	None	None
Watti Renew	None	None
Coloradical	None	None
Preeti226	None	None
Add925	None	None
Stellaplayer	None	None
Raspberry007	None	None
Stevertigo	None	None
WilliamJE	None	None
Chesterct	None	None
Mkhan-95	None	None
Anwoody	None	None
Anonymosityx	None	None
Writeformusic	None	None
NasihA139092	None	None
NSC-WISP	None	None
JayShaw TT	None	None
Girl786123	None	None
Cricketclubaus	None	None
Janawar	None	None
DerekB2010	None	None
Trend Education Foundation	None	None
Kshawberry	None	None
Ajsmith999	None	None
NNU-1-05100211	None	None
Bluesedit	None	None
Will. Henry Davies.	None	None
Sash\xc3\xa5 Fierc\xc3\xa9 Osta\xc3\xa7o	None	None
Paperweight78	None	None
Children\'sPhilanthropyCenter	None	None
Wimvandorst	None	None
Jazally	None	None
Ohconfucius	471838612	20120117094740
Ohconfucius	472921181	20120124035442
Ohconfucius	472921231	20120124035504
Ohconfucius	477140046	20120216062737
Ohconfucius	484513410	20120329092844
IronMidget	None	None
Pkornrei	None	None
Utopiantheorist	None	None
Butterfly2011	None	None
Knutbrinch	None	None
Animaajit	None	None
Llwyd66	None	None
Erik Kennedy	None	None
Rajuvinyasa	None	None
Rahul9386	None	None
ChemConnector	None	None
KidLotus	None	None
Dinoshark2000	None	None
Chris1321	None	None
John Vandenberg/Archive 13	None	None
Xiutwel-0003	None	None
Alcohology	None	None
Dinospartan117	None	None
Mbrstooge	None	None
Jgbackes	None	None
Pravisurabhi	None	None
Noodles&Wedges	None	None
Stanton86	None	None
Vickyheartwon	None	None
Emw	None	None
Adib Khaled	None	None
Smmohaghegh	None	None
Msajecki	None	None
Comm212communication	None	None
White uk	None	None
Marraii	None	None
Kc4williams	None	None
Reza.saeedi	None	None
Sp!ke	None	None
Miriam westen	None	None
Idguard	None	None
Mitchell033	None	None
Devanahalli2008	None	None
Yakuman	None	None
Marcus Tulius	None	None
Proxima Centauri	None	None
Jmgoncal	None	None
Edwardyu236	None	None
Ocean402	None	None
EQEditor	None	None
Sam Sloan	None	None
Glym	None	None
Josepiusnedumkallel	None	None
Cc60	None	None
Oprev	None	None
Kristinboconnell	None	None
Iainlindsay1981	None	None
Allusernamesareinuse	None	None
Brian0918	None	None
Arturyatsko	None	None
Farah26	None	None
Cooldude816	None	None
Ramone122	None	None
Timbouctou	None	None
Sasakubo1717/Archive 1	None	None
Vnixon	None	None
Black309	None	None
Gorrana	None	None
Namiba	463831385	20111203142722
Namiba	463946842	20111204055717
Namiba	464040417	20111204134236
Namiba	464093107	20111204202957
Namiba	464866892	20111209002511
Namiba	464866988	20111209002554
Namiba	464867065	20111209002627
Namiba	465179706	20111210213503
Namiba	465193115	20111210230411
Namiba	465834521	20111214160715
Namiba	467288784	20111223034145
Namiba	467350528	20111223152104
Namiba	467350716	20111223152231
Namiba	467350820	20111223152322
Namiba	467776559	20111226170138
Namiba	468771480	20111231135231
Namiba	473766920	20120128232950
FNHealthCouncil	None	None
Romanski1996	None	None
Rscarson	None	None
KlementyNagorny	None	None
Alan dierks	None	None
Dannyolivares	None	None
Taba s	None	None
Juliusreynoso	None	None
Sinatra3d	None	None
Peterraposo	None	None
Kngyger	None	None
Alokambar	None	None
Elvia.Y.	None	None
Brooksiderevival	None	None
Ingosb	None	None
Dundurn	None	None
Grovyle4life	None	None
Ivaneleckih	None	None
The 5th Silver Beatle	None	None
Kyokudokaikan Karate	None	None
Nirvana2013	None	None
Spitfire	None	None
Jonathan A Jones	None	None
Freshesh	None	None
Jspangler	None	None
Jle2199	None	None
Temasterson	None	None
Laurenlambs	None	None
Uptownteddy	None	None
Kristen7	None	None
L\'Espagnol	None	None
Persiangreatness	None	None
RealityFan	None	None
Belmonter	None	None
MedoSF	None	None
TylerMcAfee	None	None
MiluTz	None	None
Newsact11	None	None
Questions2answers	None	None
Epbr123	None	None
Elite compact	None	None
LC561	None	None
Bluedubs	None	None
Mmaaret	None	None
198.107.38.90	None	None
70.93.65.184	None	None
Targeman	None	None
Sold in Russia	None	None
Stoiko	None	None
Zigzig20s	None	None
Lithiym	None	None
MrYahooBen	None	None
Kenya360	None	None
Brudder Andrusha	None	None
Shadowcatfilms	None	None
Alex1991z	None	None
Buzzothecoolcat	None	None
Akbangura	463757856	20111203011925
Louis.mather21	None	None
Zineblayachi	None	None
Applejack88	None	None
Mista-X	None	None
Ananda	None	None
Jessmccabe	None	None
Dillonsimpsonfan	None	None
Orangemike	466222070	20111216214906
Orangemike	479683536	20120301174617
Orangemike	481024145	20120309163246
Orangemike	486842931	20120411162901
English Comp. Ward	None	None
Dreadarthur	None	None
Ctmadsen	None	None
Onkarsatam	None	None
Giric	None	None
Luciavilla	None	None
68.200.81.62	None	None
Tri-Edge99	None	None
Lotzoflaughs	None	None
Msalahinfo	None	None
Drews564	None	None
Sun Press	None	None
WBnChina	None	None
Jodrm	None	None
Jorgenev	460901564	20111116052430
67.150.36.131	None	None
Happyblossom	None	None
Jaytucker	None	None
Johny Stabler	None	None
Busi300	None	None
Canihuan300	None	None
Lars951	None	None
Phantomgames3456	None	None
Zeldacorner	None	None
Wormsgetsold	None	None
Gaspsiagore	None	None
BurnettsCumbria	None	None
Rsc.kidd	None	None
Cowart54	None	None
Abcgera	None	None
HSL 1232	None	None
Conquistador2k6	None	None
Ciclone	None	None
Ahmadusman	None	None
Dharlowe	None	None
The judge and jury	None	None
Saurabhwork	None	None
Toonatic7	None	None
Ea40783	None	None
Ronin.Paz	None	None
Spiwe	None	None
Ottawahitech	None	None
Avinashjayswal	None	None
Erastus Nekomba	None	None
Spottv	None	None
Nikosgreencookie	None	None
Jpp858	None	None
SubversiveVideo	None	None
Rajah2770	None	None
Ethereal Cheese	None	None
Bisigt	None	None
Richard Arthur Norton (1958- )	None	None
Nicksaltr	None	None
Cyrip	471246557	20120114012858
LANDKDESIGN	None	None
Gabriellebl	None	None
Newcrash	None	None
Uranian Institute	None	None
Esotericam	None	None
Virago250	None	None
Eric.slaw	None	None
JJ Ferrari	None	None
Jrmiller43015	None	None
Alexroller	None	None
Rite2binit	None	None
Dinoboy7788	None	None
Mrwebbin	None	None
ReverendSam	None	None
Orekhova	None	None
Mary Deering	None	None
Nzpyzik	None	None
Ccereal	None	None
Benmanninguk	None	None
Ksadarsh	None	None
Maxwellstentiford	None	None
Onesiepower	None	None
Kane841205	None	None
Ilovethebeano	None	None
Sivayogi	None	None
Airsan	None	None
Mark Lincoln	None	None
Necropenguin	None	None
SRSNipun	None	None
Fred Bauder	None	None
Pshelby17	None	None
84.177.239.141	None	None
Truthfull Liar	None	None
Thesilentplanet	None	None
ZalumoFilms	None	None
Miomio2011	None	None
Samsujata	None	None
Wlvresident	None	None
Pukkativa	None	None
Hm2k	None	None
Gocmedia	None	None
Glogger	None	None
Linelor	None	None
\xce\x91\xce\xbb\xce\xb5\xce\xbe\xce\xb1\xce\xbd\xce\xb4\xcf\x81\xce\xac\xce\xba\xce\xb7\xcf\x82	None	None
Simonetylee	None	None
Petrosjan	None	None
McMartinCummins	None	None
Justinboles	None	None
Weblugger	None	None


User List, Editors that received template warning and PROD tag and subsequent revisions involving tag, z78 vs. z79 - Test Group

user	rev_timestamp	rev_id
Jeyxxyej	None	None
Jony85	None	None
Renelj	None	None
Tabraham48	None	None
FlightTracker	None	None
Jimhodapp	None	None
Alcofribas1	None	None
Wikinovice1901	None	None
Debarunthepsychic	None	None
SOWz	None	None
PM800	None	None
Dimre01	None	None
Bobosen1	None	None
218.214.136.5	None	None
Biswajit sutradhar	None	None
Ashuputtur	None	None
Ckales1	None	None
Dirk4711	None	None
Mgraifman	None	None
Jamesjmyers	None	None
Beechee	None	None
Ran kurosawa	None	None
Sueolive	None	None
Jacobmovingfwd	None	None
Zoya&saara	None	None
R2y2a2n	None	None
Skeletor2112	None	None
Yoko santoso	None	None
Shaktipatel88	None	None
Rasho	None	None
KLepatric	None	None
No1	None	None
Alokprasad84	None	None
AlbertSM	None	None
Nutmegtree	None	None
Cathcon	None	None
Aurora369	None	None
Tranny.619	None	None
Jonrev	None	None
Shannon Tucker	None	None
Wawa22	None	None
Ticallion89	None	None
Moh32	None	None
Brie0109	None	None
Kpopperluver	None	None
V\xc3\xadctor Alexis Cantillano Oviedo	None	None
JackGuest	None	None
Jinjanjaa	None	None
Chetrasho	None	None
Dawniasboy	None	None
Alexstrazsa	None	None
Phazall	None	None
Mdl4235	None	None
Lucyrhodesbwb	None	None
Poppopcreator	None	None
Bhsfootballfan	None	None
Helltown19	None	None
Dmetz09	None	None
Wetdogmeat	460353375	20111112224500
Wetdogmeat	460353779	20111112224813
Samanthahawley	None	None
Miz072	None	None
Duotrope	None	None
Aryan1992	None	None
Gyan208	None	None
Philanit	None	None
Swayze2011	None	None
Hamham31	None	None
A loafy mother	None	None
TheDarkBlood	None	None
Commiegirl	None	None
Samradd	None	None
Atomic23112	None	None
Trendfabrik	None	None
TheRoro24	None	None
Bonnietylersave	None	None
Marshall Williams2	None	None
Piotrmoes	None	None
Pungul	None	None
Playhouse76	None	None
Snyderpickerill	None	None
Supuhstar	None	None
Elomis	None	None
RS485	None	None
Airjimmyk	None	None
Masterofdooze	None	None
Flossyfloating	None	None
Fondsdorgue	None	None
Pdsabooks	None	None
Exec8	None	None
Navneet.gugu	None	None
Ryderwiki	None	None
MortgageProf	None	None
Ryan-Jenkins97	None	None
Quasar200	None	None
Narayanraman	None	None
Lintu Ramachandran	None	None
Robinthant	None	None
Esthermhong	None	None
Esbendyr	None	None
Khat199	None	None
Alexsautographs	None	None
Pocorod11	None	None
Wikiwm	None	None
Pappy LaRoux	None	None
Rajeshbieee	None	None
John Abento	None	None
Rocketd20	None	None
Sayali sawant	None	None
Hannahison	None	None
1000MHz	None	None
Sporbul	None	None
Georgegreenrow	None	None
Lucapremier88	None	None
Ooshii333	None	None
WhisperToMe	470508325	20120109220131
Fat pig73	None	None
Jockmcgrumpy	None	None
Jfgueguen	None	None
Heather.Baxter	None	None
Nananabeattie	None	None
Jabes212	None	None
DGaw	461819049	20111121200622
DGaw	463006045	20111129020554
DGaw	463006046	20111129020555
DGaw	463934225	20111204033103
DGaw	464134240	20111205005836
DGaw	464141001	20111205014958
DGaw	464327519	20111206031630
DGaw	464775597	20111208150109
DGaw	464775599	20111208150110
DGaw	474204846	20120131130240
DGaw	475288782	20120205212444
DGaw	476929061	20120215010107
DGaw	478185086	20120222023722
1091student	None	None
Englshivy	None	None
Andy Fugard	None	None
Mr.TAMER.Shlash	None	None
Awosowunmi	None	None
Ashliveslove	None	None
Algologicinc	None	None
Aballister	None	None
Mohworks	None	None
Radiobums	None	None
Padalkar.kshitij	None	None
MrHat1065	None	None
Tukesh	None	None
Sophiehull	None	None
Stratman07	None	None
Cabanna	None	None
Iketaylor	None	None
Jd2718	None	None
MGhaith	None	None
MadMax	None	None
Fariaz009	None	None
Dreammancer	None	None
Stocky37	None	None
Jedim13	None	None
Humed90	None	None
AAOMnM2011	None	None
Wethreekingsscott	None	None
Eric hack	None	None
Rossinblu	None	None
Aspingley	None	None
Blackvegit	None	None
KittyKo\'sCute	None	None
TempleF	None	None
Spankybumchunks	None	None
Angieltmusic	None	None
EGeffroy	None	None
Nameless User	None	None
Bleifeld	None	None
Katrinahalili	None	None
Javakcao	None	None
Kristinehost	None	None
Ercamlikaya	None	None
11rcombs	None	None
Hy10sm	None	None
NMoops	None	None
Realkyhick	None	None
Sfernando50	None	None
Jlsnelson	None	None
BradPatrick	None	None
Jacinthamc	None	None
Jattwarcraft	None	None
Free8860	None	None
7upnone	None	None
Bdarwin	None	None
Gannic	None	None
Lucia Black	None	None
Engl101Pina	None	None
Nancisis	None	None
Supergabbyshoe	None	None
Jh10jcd	None	None
LexOnTheWater	None	None
Cbaur09	None	None
Hezzel	None	None
Dalderdj	None	None
SHerr	None	None
Veliveijari	None	None
Ostreicher	None	None
Kevinpcarter	None	None
Thetalkingheads	None	None
Oombutu	None	None
Qrc2006	None	None
Emettler	None	None
GVnayR	465288976	20111211140936
Krhuske	None	None
Aps1969	None	None
Bandmosh123	None	None
Filing Flunky	461968836	20111122174552
Filing Flunky	464002210	20111204101052
Filing Flunky	464207297	20111205131312
Filing Flunky	464238687	20111205171354
Filing Flunky	464557844	20111207121923
Filing Flunky	464625332	20111207201112
Filing Flunky	464625370	20111207201129
Jani47	None	None
Kaimaan	None	None
Gr57	None	None
Rofu	None	None
Hi99ins	None	None
Rumailzaheer	None	None
Bevhillswholesale	None	None
TheQuebin123	None	None
Suro seko	None	None
Liambroad98	None	None
Tarcha1115	None	None
Jblumrei	None	None
Colmulhall77	None	None
Jzaozeldajava	None	None
Zhele3	None	None
Bugskentucky	None	None
Cpg521	None	None
MasterYoda222007	None	None
Kks862003	None	None
Doctorwhofan400	None	None
Vijila medknow	None	None
At4470	None	None
Anshulkumardhiman	None	None
DavinciRed	None	None
Jasmine158	None	None
Yojay2000	None	None
Dsmith56	None	None
MineUnderworld	None	None
Kkarma	None	None
Carbonmotion	None	None
Fyodor Dostoevsky	None	None
Sociald10	None	None
Baxterewers	None	None
Bwatson23	None	None
Monster bloke	None	None
Gyandeep Kaushal	None	None
Ashleycudney	None	None
Gus77	None	None
Nycny	None	None
Mawma	None	None
Cheeseman123456789	None	None
DaniGurl13	None	None
StealthStar	None	None
67.182.60.246	None	None
DirkvdM	None	None
Mwvlyons	None	None
Spokane3	None	None
Jonty999	None	None
Mvc works	None	None
Iricp	None	None
Caleb Elliott	None	None
Dalerichards	None	None
Kevoy1	None	None
Writerslink	None	None
Kupirijo	None	None
Assassin\'s Creed	None	None
Badger No.1	None	None
Downtheroad69	None	None
Thelongwikihome	None	None
M4guir3	None	None
Dallasapps	None	None
Kabir1976	None	None
RJT12511	None	None
Friendshut	None	None
Fredler Brave	None	None
Mills gal	None	None
Jrevels	None	None
ChildofMidnight	None	None
Raghith	None	None
SportsMaster	None	None
Upinel	None	None
Tkuvho	None	None
Edesviktoria	None	None
Schakeco	None	None
Argos\'Dad	None	None
2679D	464777358	20111208151327
Bencathers	None	None
Faizanalivarya	None	None
Tokyo Watcher	None	None
Gjr rodriguez	None	None
Pallabisarkar25	None	None
WeKnowThis	None	None
GlenWalker01	None	None
UghItsAndrew	None	None
Simonpalakkal	None	None
Scaptor86	None	None
Coin945	464034636	20111204123915
JackShestak	None	None
L3ssm4n	None	None
LeelanQubic	None	None
Randomdude28	None	None
Bradthebear96	None	None
Buon professore	None	None
Tiptoety	None	None
M@sk	None	None
Lordgold138	None	None
ThreeQuills	None	None
Alisameredith	None	None
Williamleute	None	None
Nickdb93	None	None
Penale52	469577457	20120104202334
Penale52	470332550	20120108221341
Penale52	484230021	20120327193022
Shah 786	None	None
Ashsteury	None	None
Hlearles	None	None
Rivelino70	None	None
Internetbrowser	None	None
Bschanck	None	None
Wcability109	None	None
ManOfViolentCrimespsych	None	None
Blood Red Sandman	None	None
Dedark	None	None
Ea3wr	None	None
Sounders fan	None	None
Mkortoci	None	None
Integrated Research Limited	None	None
Walsh15	None	None
Mauger73	None	None
DaveMedia	None	None
Menoselmike	None	None
WildManKY	None	None
Smokeydude	None	None
Intothefire	None	None
Octorama	None	None
Sampozx	None	None
Squatj	None	None
Vmanzon	None	None
Wikiguru1318	None	None
Nathanf10	None	None
CrazyZA	None	None
Melbrgn	None	None
NSC-WISP	None	None
Frankieroberto	None	None
JayShaw TT	None	None
DarkMan66	None	None
Sdesalas	None	None
Trend Education Foundation	None	None
Charles Edward	466199697	20111216190223
JustAnotherAZGuy	None	None
Mannionpaul	None	None
Wikilove99ru	None	None
Dramaspeak1	None	None
CoreyBlake9000	None	None
Ericnotderek	None	None
Sigma0 1	None	None
Lovethevoid	None	None
Master son	None	None
Rrrr2699	None	None
Uddikamaho	None	None
Pkornrei	None	None
206.45.165.189	None	None
JesseHolmz	None	None
Tommsesas	None	None
Opokucharles73	None	None
GolfInfo	None	None
ChemConnector	None	None
Chrisshipley	None	None
Batcreekstone	None	None
ClaytonBW	None	None
AlexxxNestor	None	None
Ziofanale66	None	None
Postbil	None	None
12.196.37.227	None	None
Rickytrece	None	None
Emmadesa	None	None
GillesAuriault	None	None
Jzsaffa	None	None
Swinberg	None	None
Dburton21	None	None
Macnit	None	None
Cha2our	None	None
Esigc	None	None
Mbrstooge	None	None
Kvmenges	None	None
Prad2609	464231143	20111205162209
Rji	None	None
Kaybvin	None	None
Demon587	None	None
Ffrritz	None	None
\xd0\xa1\xd1\x82\xd0\xb0\xd1\x81 \xd0\xa4\xd1\x91\xd0\xb4\xd0\xbe\xd1\x80\xd0\xbe\xd0\xb2	None	None
Bearly541	None	None
Themarketingsociety	None	None
LoneWolf1992	None	None
IMPACTlaw	None	None
Makky76	None	None
AshwiniKalantri	None	None
ProjectNeo	None	None
Kc4williams	None	None
Vespristiano	None	None
5455oasis	None	None
Sujit900	None	None
Ct	None	None
Msauer4	None	None
110808049 nisarg	None	None
Tassedethe	487906626	20120417220917
Amirai0110	None	None
68.220.24.71	None	None
RobinMEvans	None	None
Frederick99	None	None
Leoca	None	None
Jc349108	None	None
Muscarella	None	None
TBrandley	None	None
Mrbananaturtle	None	None
Engl101warren	None	None
Daisyisthebest	None	None
DVMt	None	None
Pureenterprise	None	None
Wiesnerc	None	None
Shoegal73	None	None
Dmmaus	None	None
Mica97ab	None	None
Djmissinglinks	None	None
Amouage	None	None
Tarassemeniuk	None	None
Torinew	None	None
Leila00123	None	None
KlementyNagorny	None	None
HawkWang	None	None
Roberto116	None	None
M.C. Brown Shoes	None	None
Mech.bala	None	None
Katieleeann	None	None
Ritwik93	None	None
Elvia.Y.	None	None
Quentin D F Bargate	None	None
HKMiller	None	None
Willarmstrong11	None	None
Stevemarino	None	None
Tyc5858	None	None
Vibra805	None	None
Bananababie11	None	None
Skywalker80100	None	None
Islamocid	None	None
RVX	None	None
Pcap	None	None
Wandergirl13	None	None
Riya Kewlani22	None	None
Hergilfs	None	None
Senseless1logic	None	None
Timmy43	None	None
TyKendall	None	None
Gpolk50	None	None
Mdh1976	None	None
Trainadelaide2000	None	None
B.Lasonder	None	None
Adamcoltman	None	None
MiluTz	None	None
Shawnn2211	None	None
Belgrade011	None	None
Pethrischool	None	None
Elite compact	None	None
Ganeshk	None	None
Xboxmanwar	None	None
Robertscoble	None	None
When Geese Attack	None	None
Sn3aks	None	None
A M Khan121	None	None
Dara97	None	None
Blairleavitt	None	None
Mr.dobrica	None	None
Robertc555	None	None
D913010	None	None
Rogpyvbc	None	None
Fighterf4u	None	None
209.178.165.58	None	None
Runningonbrains	485930213	20120406164846
TylerJThomas87	None	None
VampireKilla	None	None
Akbangura	463757856	20111203011925
FashionSoroity	None	None
Bigpad	None	None
Mmoa1977	None	None
The Johnsonian	None	None
Marc darra	None	None
Imoon5	None	None
Steveswei	None	None
Blcklabel92	None	None
Sattilite1	None	None
JenniferHeartsU	None	None
Jozoisis	None	None
68.200.81.62	None	None
Vanilladude	None	None
Mmabelo	None	None
Sammy doug	None	None
Yusufe101	None	None
Clsalvaggio	None	None
Jorgenev	460901564	20111116052430
Der Golem	None	None
Akmenos288	None	None
Crazy-dancing	None	None
NNU-11-22100502	None	None
Johny Stabler	None	None
Lars951	None	None
CanadianResidential	None	None
Krohwer	None	None
Matt Gatenby	None	None
Alsimmons27	None	None
Jb351	None	None
WolfSearcher3	None	None
Ladyrose16	None	None
Drvw	None	None
TechCore	None	None
Conchboy242	None	None
Superkrizz	None	None
Monirugg	None	None
Ciclone	None	None
Ainsworth anderson	None	None
Michiharuharu	None	None
Lalacentral2	None	None
George of the georges	None	None
Spratton Hall School	None	None
TriXtan	None	None
Peel centrehalf	None	None
Nnu-1222100535	None	None
Imikool2005	None	None
Grakelin	None	None
Gunmetal Angel	None	None
Hup234	None	None
Le Bosch	None	None
Bob the Wikipedian	None	None
Tiffny18	None	None
DreamBeliever2238	None	None
Robives	None	None
Prococopon	None	None
Cyclonerampage	None	None
Igotask8board	None	None
Thirty5design	None	None
Hokaspokas	None	None
Nicksaltr	None	None
Hornet24	None	None
Finallly	None	None
Spojrzenie	None	None
DoubleMeDown	None	None
Forwardgear	None	None
Joonkee	None	None
R\'n\'B	463219272	20111130023507
Gatedear	None	None
Rickq	None	None
Ashu1997	None	None
Socrtwo	None	None
Manakkadanpanayamparambil	None	None
Ptsaklan	None	None
Ragudave	None	None
Wikibgs	None	None
Kchat101	None	None
IslandControl	None	None
Cduensing	None	None
Mion	None	None
Mibelz	None	None
WenonahRider	None	None
Tanvirchowdhury	None	None
Ovenkat	None	None
Stks3	None	None
Unixrevolution	None	None
Luckyme517	None	None
Abdalla abdirahman	None	None
Chaoticravenger	None	None
Knulclunk	None	None
Amlanpati	None	None
LosAngelesWriter	474675644	20120203004145
GMAKapusoMo	None	None
Ckennedy2505	None	None
Direct000	None	None
Cast chris	None	None
Alivemajor	None	None
NermalTheBunny	None	None
Brianbowie	None	None
Ngord	None	None
Island Roamer	None	None
Spencer1984	None	None
Sweetfootedness	None	None
Aprakash2000	None	None
The Genius 2000	None	None
Fatpig558	None	None
\xce\x91\xce\xbb\xce\xb5\xce\xbe\xce\xb1\xce\xbd\xce\xb4\xcf\x81\xce\xac\xce\xba\xce\xb7\xcf\x82	None	None
Bonhoffer	None	None
Osvaldo.silva	None	None
Mscott76	None	None


The response rates to the templates are as follows:


Total test : 577

Response test : 16 (2.7729636%)


Total control : 579

Response control : 13 (2.2452504%)


The difference in the rates above are not statistically significant.


Maryana went through the deleted revisions of users that received a PROD experimental template:


So it looks like our test *did* increase the likelihood of people removing the PROD tag... unfortunately, their articles still got deleted.
Looking at all deleted revisions, 70 out of 550 control users (13%) removed the PROD tag, versus 96 out of 550 (17%) test users. Note that the absolute numbers are also a lot higher than in non-deleted revisions (where roughly 2% of each :groups removed the tag). This makes sense -- most new articles on Wikipedia get deleted in one way or another...


Discussion & Lessons[edit]

The difference in measured instances of deleted revisions 0-3 days after the posting is inconclusive. However, there may be some fruit borne in measuring the differences between high numbers of deleted revisions and low numbers of deleted revisions before and after the posting.