Template A/B testing/Twinkle Analyses

Overview

The Twinkle testing experiment descriptions can be found here.

The experiments themselves involved comparing template sets including a control and a test template. The control templates are made up of existing Huggle warning templates in use in the community while the test templates represent modified versions in measurable ways (length of warning, directives, personalization etc.) - the experiment definition can be found here.

Whether the changes made to test templates had an effect was primarily assessed using logistic regression analysis in R. Editor groups were pre-filtered based on:

• registered / non-registered
• minimum number of deleted edits made before the posting
• maximum number of deleted edits made before the posting
• minimum number of deleted edits made after the posting (usually at least 1 .. maybe)

The results of these analyses along with the conditions on the editor activity samples can be seen in the experiment results in the next section.

In the plots accompanying the analyses the effect on the mean ratio of edits before and after the posting as well as the mean absolute number of edits made after the posting are measured against the minimum number of edits before the posting used to select the sample group. Accompanying these plots are those including the sample sizes for each data point in the edit trend (both control and test).

Analyses Results

Data Munging / Filtering: 
 
Looking at all edit events
Blocks after = 0 (no blocks after seeing template)
namespace = 0
first_warning = TRUE
 
Findings:
 
The test was performed among registered users only as this group dominated the samples.  Instances of edits in Project namespace after the posting were measured.  There was not a significant effect measured between the test and control however the control did have a higher rate of edit events in this namespace.

Modelling Analysis, Registered Users, Instances of Edit Events in Namespace 4 (Project), z81 vs. z82 - R Output

execute.main(min_edits_before = 0, max_edits_before = Inf, registered = TRUE) Call: glm(formula = template ~ edit_event, family = binomial(link = "logit"), data = all_data) Deviance Residuals: Min 1Q Median 3Q Max -1.269 -1.269 1.089 1.089 1.151 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) 0.21211 0.08736 2.428 0.0152 * edit_event -0.15074 0.17943 -0.840 0.4008 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 955.33 on 692 degrees of freedom Residual deviance: 954.62 on 691 degrees of freedom AIC: 958.62 Number of Fisher Scoring iterations: 3 Test Edit Events: 84 users edited in Project out of 377 Samples Control Edit Events: 79 users edited in Project out of 316 Samples

Modelling Analysis, Registered Users, Edit Counts in Namespace 4 (Project), z81 vs. z82 - R Output

execute.main(min_edits_before = 0, max_edits_before = Inf, registered = TRUE) Call: glm(formula = template ~ edit_counts_all, family = binomial(link = "logit"), data = all_data) Deviance Residuals: Min 1Q Median 3Q Max -1.257 -1.257 1.100 1.100 1.523 Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) 0.1843266 0.0768352 2.399 0.0164 * edit_counts_all -0.0007911 0.0009528 -0.830 0.4064 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 955.33 on 692 degrees of freedom Residual deviance: 954.59 on 691 degrees of freedom AIC: 958.59 Number of Fisher Scoring iterations: 3 Test Edit Counts (377 Samples): Min. 1st Qu. Median Mean 3rd Qu. Max. 0.000 0.000 0.000 7.493 0.000 1223.000 Test Edit Counts (316 Samples): Min. 1st Qu. Median Mean 3rd Qu. Max. 0.00 0.00 0.00 13.04 0.25 1391.00

Plots

Plots - AfD (z78, z79) - edits after (0-3 days) and normalized edits after (0-3 days)

Summary

AfD

Legend of Terms:
================


Variables:

Rev_Deleted_Instances - Number of deleted revisions 0-3 days after posting
Edit_Event_NS_4 - Did the user edit project after seeing the template? (1 or 0)
Edit_Count_NS_4 - Edit counts in Prject namespace after posting

Registered	Variable(s)	Control Result	Test Result	Winner	% Increase	Confidence	Sample Size	Params
TRUE	Edit_Event_NS_4	79 / 316	84 / 377	control (inconclusive)	--	~ 60%	377 (test), 316 (control)	execute.main(min_edits_before = 0, max_edits_before = Inf, registered = TRUE)
TRUE	Edit_Count_NS_4	13.04	7.493	control (inconclusive)	--	~ 60%	377 (test), 316 (control)	execute.main(min_edits_before = 0, max_edits_before = Inf, registered = TRUE)

PROD

Below are the lists of users along with rev ids and rev timestamps of any subsequent revisions involving the PROD tag (e.g. "Proposed Deletion").

User List, Editors that received template warning and PROD tag and subsequent revisions involving tag, z78 vs. z79 - Control Group

user rev_timestamp rev_id Lauranic None None Piroroadkill None None WotWeiller None None WLU None None Alex vamos 99 None None Baerc234 None None Dnfenner None None Nika1693 None None Lyndonid None None Serge.Belongie None None Captain Q None None GregoryErnst None None Katrina Angela Gonzales None None Tossnara None None Sukhraj brar None None Krishnapersonal None None Chehticham None None Lawqu None None Kaswardy None None Dragon Genius None None Yammzsi None None Keelyboom None None Sonore None None Almondb None None CellarMist None None Danger 478923477 20120226103723 HMHunt None None Shaktipatel88 None None NNU-1-17100215 None None Mech5107 None None IGG8998 None None Simply14u None None Alokprasad84 None None Whofan44 None None TehZorroness None None Payyan-2000 None None Bealtes6789 None None Ziddirazan None None KoolKellyGreen None None Tomatool None None Brandonandshae\'s None None Vicentenavarre2 None None Speedtrace None None Prentert None None Adityavisionrx None None Dell.bilik None None Andrew22k None None Frozenbrains None None Bookworm857158367 None None Quidster4040 None None Mdl4235 None None Lizziecassmaran None None Poppopcreator None None YolentaShield None None Uhl2016 None None Wetdogmeat 460353375 20111112224500 Wetdogmeat 460353779 20111112224813 Cwinslet None None Biggleswiki None None Joshua Guest None None Gurjinder.Romana None None Makky76 None None Siobhanhsm None None TheAztec None None Colorfulthrowup None None Anshulkumardhiman None None Pooltv None None Jkim ucb None None Damienfernando None None SocialMarketingVA None None Supuhstar None None Robrox89 None None Kooh123 None None Ksimola None None Bala8vijay None None Tofer138 None None Kylemmenard None None Universityuser None None Edina-k None None Benjaminevans82 None None Z\xc3\xbcrich1909 None None Magedogtag None None Niko1297 None None James andrew lewis None None Triwbe None None Shiva Khanal None None Cwilson66 None None Kerachian None None Demonz587 None None Robotic564 None None Alexsautographs None None Oi8uhybtfr5 None None Ilove96 None None John Abento None None Kurtnardone None None Roznar97 None None Stjetters None None Horsey fart None None Tward007uk None None December2011 None None SolomanMcKenzie None None Elsot39 None None 142.227.207.193 None None SlickSy None None Aus eagle None None Adbennett None None Fat pig73 None None Rajat Ghai None None Teaandletters None None Fez1321 None None Dipankan001 None None Kdianed None None Sailor iain None None Dagnytaggartmoxie None None Astertreff None None Asmolnikov02091997 None None Severnosianie None None DJNutty None None Pitchka None None Tempurafried None None Anna.kudlay None None TheCustomOfLife None None Dreamofpeaceinschool None None QueenDemi None None Travismac3 None None 71.184.204.92 None None Katrinahalili None None Jsteele12 None None NNU-11-zhanghongjing None None Xploiter None None Dailta None None Abdulghaffoor None None Charles Fitzgerald III None None Homad89 None None Dravecky 465011371 20111209203327 Earl Andrew None None MoQazalbash None None Fracamauthor None None Olesachem None None ToranL7231 None None Ikestarr None None Mr. D. E. Mophon None None Peruginionio None None Danieljpost None None Muehlenau None None Tigerbreath13 None None Rimianika None None Ministrymedia None None Matthead None None Kryssantos1130 None None MobileMarketWatch None None Lawbrarian None None Richard Brooks MK None None Nmyers223 None None Georgejdorner None None Even23 None None Iuhoosier66 None None Le Bosch None None TwinTowersNYC None None LucileGoldstien None None Lucia Black None None Glovefish None None Udaymanchanda None None Mjohn127 None None Wtimrock None None Tahliaheffernan None None Rex10098 None None Kevinpcarter None None Nethalcon None None Gluontronic None None Vasim mirza None None Galworld None None Rpirelli None None Ksawatzky None None Hellosmiles None None Jackevee None None Roshaniel21 None None Peter McE None None Cazcazcaz None None Gengel None None DCabot None None Ohjough None None Ksfol9 None None Dustinslush None None Suro seko None None ProjectManhattan None None Daniel.Vitor1 None None Pinkfloyd95209 None None Tripilohit None None Ravindrajith None None Mach240390 None None Mncline None None RTG None None Ath1973 None None Ferdinandude None None Thefootballweekly None None Renate.nyborg None None Soccerzchelzlim None None Amosbatto None None OP CALIFORNIA None None Fanfan1964 None None Dcampbell2011 None None Sirmikey111 None None QuixoteArmy None None Double0agent None None Vijila medknow None None Mindchained None None Tictictalk16 None None Kamolga22 None None Hiphopreina None None Dlbu226 None None Bjdav23 None None Starzynka None None Bootslewis None None TatyanaR None None Veronicarevier None None Vmanzon None None Ainsworth anderson None None Chadialameddine None None DrMattKerry None None Ianhowlett None None Theshywillraindeath None None Nicholaus.hughes None None Pioneersbrog None None Danisamsoel None None Unidue.hilfskraft None None The Night\'s Watch None None JournoJoeMcG None None Applesnbananas None None Beanscene None None Neelix 463020495 20111129031721 Neelix 463021579 20111129032243 Neelix 468129185 20111228185537 Neelix 468138476 20111228195039 Neelix 468337894 20111229211251 Neelix 471074714 20120113024046 Neelix 472449337 20120121162552 Neelix 472450395 20120121163222 Neelix 472654157 20120122185334 Neelix 475322220 20120206013818 Neelix 475325976 20120206021713 Neelix 475504978 20120207021730 Neelix 476450492 20120212141311 Neelix 476752899 20120214011928 Neelix 477437157 20120217220647 Neelix 477644025 20120219022352 Neelix 478263108 20120222154038 Neelix 478264532 20120222155147 Neelix 478471933 20120223191544 Neelix 478520639 20120224002705 Neelix 479992997 20120303155650 Neelix 479997273 20120303162610 Neelix 482255945 20120316204835 Neelix 482793435 20120319215638 Neelix 484434246 20120328205919 Neelix 484470425 20120329013635 Neelix 484624098 20120330003106 Neelix 485229847 20120402212110 Neelix 487356157 20120414161709 Neelix 487716616 20120416193214 Neelix 487730610 20120416210152 Helpingfriendlybook None None Theunknown24113 None None Fealiath None None Elif Dervi\xc5\x9f None None Kapilcm None None Clydemaster555 None None Cwwiss None None MatthewJS None None Gdr05a None None Bucsfannva None None Stadiumseating None None Mr.eurosong None None Aer1n2026 None None Dogru144 None None Assassin\'s Creed None None Bodwysty None None Picard09 None None Gocek None None Maccer None None Michaelloves80sMusic None None Hollywoodsuccessstory None None Sliceservace None None AlexSwanson None None MJF2000 None None Yify None None Mchaumont None None Mikemor92 487583692 20120415234135 Dimitry-1983 None None Ram548 None None Kate Conroy None None Pakhack None None Theseus1776 477402188 20120217182246 Indiantvlover None None Subratadhara None None Luizfmaf None None Alberoni otti None None Cherrybabascrazy None None Ffkks None None Smukkai None None Pepe061270 None None Hammster16 None None Gograded None None Gjr rodriguez None None Cost 4000 None None Jennybrown88 None None Vini1621 None None Vshehu None None Coin945 464034636 20111204123915 Kukulich None None Stgblu None None Albanian222 None None Skullcrusher98 None None Battousai86 None None Entsal007 None None M@sk None None Cassandralisa None None Checkee None None Shanghaitucker None None Elinraun None None Tyganlion None None Ashpreet92 None None MsTingaK None None BiT None None Barraque None None Hlearles None None Dcarranza None None Bridgetlin None None 88guy88 None None Adrianvirr None None OddibeKerfeld None None Raulaudelo None None 213.253.40.156 None None Manarg None None Lynch3001 None None 128.115.87.44 None None Unseen45 None None Brycenbeck None None \xd0\x9d\xd0\xbe\xd1\x85\xd1\x87\xd0\x9e None None Dave9862 None None Emergencyslider None None Skycloud86 None None Gv23 None None Lolman1023 None None Michael ricardo None None Watti Renew None None Coloradical None None Preeti226 None None Add925 None None Stellaplayer None None Raspberry007 None None Stevertigo None None WilliamJE None None Chesterct None None Mkhan-95 None None Anwoody None None Anonymosityx None None Writeformusic None None NasihA139092 None None NSC-WISP None None JayShaw TT None None Girl786123 None None Cricketclubaus None None Janawar None None DerekB2010 None None Trend Education Foundation None None Kshawberry None None Ajsmith999 None None NNU-1-05100211 None None Bluesedit None None Will. Henry Davies. None None Sash\xc3\xa5 Fierc\xc3\xa9 Osta\xc3\xa7o None None Paperweight78 None None Children\'sPhilanthropyCenter None None Wimvandorst None None Jazally None None Ohconfucius 471838612 20120117094740 Ohconfucius 472921181 20120124035442 Ohconfucius 472921231 20120124035504 Ohconfucius 477140046 20120216062737 Ohconfucius 484513410 20120329092844 IronMidget None None Pkornrei None None Utopiantheorist None None Butterfly2011 None None Knutbrinch None None Animaajit None None Llwyd66 None None Erik Kennedy None None Rajuvinyasa None None Rahul9386 None None ChemConnector None None KidLotus None None Dinoshark2000 None None Chris1321 None None John Vandenberg/Archive 13 None None Xiutwel-0003 None None Alcohology None None Dinospartan117 None None Mbrstooge None None Jgbackes None None Pravisurabhi None None Noodles&Wedges None None Stanton86 None None Vickyheartwon None None Emw None None Adib Khaled None None Smmohaghegh None None Msajecki None None Comm212communication None None White uk None None Marraii None None Kc4williams None None Reza.saeedi None None Sp!ke None None Miriam westen None None Idguard None None Mitchell033 None None Devanahalli2008 None None Yakuman None None Marcus Tulius None None Proxima Centauri None None Jmgoncal None None Edwardyu236 None None Ocean402 None None EQEditor None None Sam Sloan None None Glym None None Josepiusnedumkallel None None Cc60 None None Oprev None None Kristinboconnell None None Iainlindsay1981 None None Allusernamesareinuse None None Brian0918 None None Arturyatsko None None Farah26 None None Cooldude816 None None Ramone122 None None Timbouctou None None Sasakubo1717/Archive 1 None None Vnixon None None Black309 None None Gorrana None None Namiba 463831385 20111203142722 Namiba 463946842 20111204055717 Namiba 464040417 20111204134236 Namiba 464093107 20111204202957 Namiba 464866892 20111209002511 Namiba 464866988 20111209002554 Namiba 464867065 20111209002627 Namiba 465179706 20111210213503 Namiba 465193115 20111210230411 Namiba 465834521 20111214160715 Namiba 467288784 20111223034145 Namiba 467350528 20111223152104 Namiba 467350716 20111223152231 Namiba 467350820 20111223152322 Namiba 467776559 20111226170138 Namiba 468771480 20111231135231 Namiba 473766920 20120128232950 FNHealthCouncil None None Romanski1996 None None Rscarson None None KlementyNagorny None None Alan dierks None None Dannyolivares None None Taba s None None Juliusreynoso None None Sinatra3d None None Peterraposo None None Kngyger None None Alokambar None None Elvia.Y. None None Brooksiderevival None None Ingosb None None Dundurn None None Grovyle4life None None Ivaneleckih None None The 5th Silver Beatle None None Kyokudokaikan Karate None None Nirvana2013 None None Spitfire None None Jonathan A Jones None None Freshesh None None Jspangler None None Jle2199 None None Temasterson None None Laurenlambs None None Uptownteddy None None Kristen7 None None L\'Espagnol None None Persiangreatness None None RealityFan None None Belmonter None None MedoSF None None TylerMcAfee None None MiluTz None None Newsact11 None None Questions2answers None None Epbr123 None None Elite compact None None LC561 None None Bluedubs None None Mmaaret None None 198.107.38.90 None None 70.93.65.184 None None Targeman None None Sold in Russia None None Stoiko None None Zigzig20s None None Lithiym None None MrYahooBen None None Kenya360 None None Brudder Andrusha None None Shadowcatfilms None None Alex1991z None None Buzzothecoolcat None None Akbangura 463757856 20111203011925 Louis.mather21 None None Zineblayachi None None Applejack88 None None Mista-X None None Ananda None None Jessmccabe None None Dillonsimpsonfan None None Orangemike 466222070 20111216214906 Orangemike 479683536 20120301174617 Orangemike 481024145 20120309163246 Orangemike 486842931 20120411162901 English Comp. Ward None None Dreadarthur None None Ctmadsen None None Onkarsatam None None Giric None None Luciavilla None None 68.200.81.62 None None Tri-Edge99 None None Lotzoflaughs None None Msalahinfo None None Drews564 None None Sun Press None None WBnChina None None Jodrm None None Jorgenev 460901564 20111116052430 67.150.36.131 None None Happyblossom None None Jaytucker None None Johny Stabler None None Busi300 None None Canihuan300 None None Lars951 None None Phantomgames3456 None None Zeldacorner None None Wormsgetsold None None Gaspsiagore None None BurnettsCumbria None None Rsc.kidd None None Cowart54 None None Abcgera None None HSL 1232 None None Conquistador2k6 None None Ciclone None None Ahmadusman None None Dharlowe None None The judge and jury None None Saurabhwork None None Toonatic7 None None Ea40783 None None Ronin.Paz None None Spiwe None None Ottawahitech None None Avinashjayswal None None Erastus Nekomba None None Spottv None None Nikosgreencookie None None Jpp858 None None SubversiveVideo None None Rajah2770 None None Ethereal Cheese None None Bisigt None None Richard Arthur Norton (1958- ) None None Nicksaltr None None Cyrip 471246557 20120114012858 LANDKDESIGN None None Gabriellebl None None Newcrash None None Uranian Institute None None Esotericam None None Virago250 None None Eric.slaw None None JJ Ferrari None None Jrmiller43015 None None Alexroller None None Rite2binit None None Dinoboy7788 None None Mrwebbin None None ReverendSam None None Orekhova None None Mary Deering None None Nzpyzik None None Ccereal None None Benmanninguk None None Ksadarsh None None Maxwellstentiford None None Onesiepower None None Kane841205 None None Ilovethebeano None None Sivayogi None None Airsan None None Mark Lincoln None None Necropenguin None None SRSNipun None None Fred Bauder None None Pshelby17 None None 84.177.239.141 None None Truthfull Liar None None Thesilentplanet None None ZalumoFilms None None Miomio2011 None None Samsujata None None Wlvresident None None Pukkativa None None Hm2k None None Gocmedia None None Glogger None None Linelor None None \xce\x91\xce\xbb\xce\xb5\xce\xbe\xce\xb1\xce\xbd\xce\xb4\xcf\x81\xce\xac\xce\xba\xce\xb7\xcf\x82 None None Simonetylee None None Petrosjan None None McMartinCummins None None Justinboles None None Weblugger None None

User List, Editors that received template warning and PROD tag and subsequent revisions involving tag, z78 vs. z79 - Test Group

user rev_timestamp rev_id Jeyxxyej None None Jony85 None None Renelj None None Tabraham48 None None FlightTracker None None Jimhodapp None None Alcofribas1 None None Wikinovice1901 None None Debarunthepsychic None None SOWz None None PM800 None None Dimre01 None None Bobosen1 None None 218.214.136.5 None None Biswajit sutradhar None None Ashuputtur None None Ckales1 None None Dirk4711 None None Mgraifman None None Jamesjmyers None None Beechee None None Ran kurosawa None None Sueolive None None Jacobmovingfwd None None Zoya&saara None None R2y2a2n None None Skeletor2112 None None Yoko santoso None None Shaktipatel88 None None Rasho None None KLepatric None None No1 None None Alokprasad84 None None AlbertSM None None Nutmegtree None None Cathcon None None Aurora369 None None Tranny.619 None None Jonrev None None Shannon Tucker None None Wawa22 None None Ticallion89 None None Moh32 None None Brie0109 None None Kpopperluver None None V\xc3\xadctor Alexis Cantillano Oviedo None None JackGuest None None Jinjanjaa None None Chetrasho None None Dawniasboy None None Alexstrazsa None None Phazall None None Mdl4235 None None Lucyrhodesbwb None None Poppopcreator None None Bhsfootballfan None None Helltown19 None None Dmetz09 None None Wetdogmeat 460353375 20111112224500 Wetdogmeat 460353779 20111112224813 Samanthahawley None None Miz072 None None Duotrope None None Aryan1992 None None Gyan208 None None Philanit None None Swayze2011 None None Hamham31 None None A loafy mother None None TheDarkBlood None None Commiegirl None None Samradd None None Atomic23112 None None Trendfabrik None None TheRoro24 None None Bonnietylersave None None Marshall Williams2 None None Piotrmoes None None Pungul None None Playhouse76 None None Snyderpickerill None None Supuhstar None None Elomis None None RS485 None None Airjimmyk None None Masterofdooze None None Flossyfloating None None Fondsdorgue None None Pdsabooks None None Exec8 None None Navneet.gugu None None Ryderwiki None None MortgageProf None None Ryan-Jenkins97 None None Quasar200 None None Narayanraman None None Lintu Ramachandran None None Robinthant None None Esthermhong None None Esbendyr None None Khat199 None None Alexsautographs None None Pocorod11 None None Wikiwm None None Pappy LaRoux None None Rajeshbieee None None John Abento None None Rocketd20 None None Sayali sawant None None Hannahison None None 1000MHz None None Sporbul None None Georgegreenrow None None Lucapremier88 None None Ooshii333 None None WhisperToMe 470508325 20120109220131 Fat pig73 None None Jockmcgrumpy None None Jfgueguen None None Heather.Baxter None None Nananabeattie None None Jabes212 None None DGaw 461819049 20111121200622 DGaw 463006045 20111129020554 DGaw 463006046 20111129020555 DGaw 463934225 20111204033103 DGaw 464134240 20111205005836 DGaw 464141001 20111205014958 DGaw 464327519 20111206031630 DGaw 464775597 20111208150109 DGaw 464775599 20111208150110 DGaw 474204846 20120131130240 DGaw 475288782 20120205212444 DGaw 476929061 20120215010107 DGaw 478185086 20120222023722 1091student None None Englshivy None None Andy Fugard None None Mr.TAMER.Shlash None None Awosowunmi None None Ashliveslove None None Algologicinc None None Aballister None None Mohworks None None Radiobums None None Padalkar.kshitij None None MrHat1065 None None Tukesh None None Sophiehull None None Stratman07 None None Cabanna None None Iketaylor None None Jd2718 None None MGhaith None None MadMax None None Fariaz009 None None Dreammancer None None Stocky37 None None Jedim13 None None Humed90 None None AAOMnM2011 None None Wethreekingsscott None None Eric hack None None Rossinblu None None Aspingley None None Blackvegit None None KittyKo\'sCute None None TempleF None None Spankybumchunks None None Angieltmusic None None EGeffroy None None Nameless User None None Bleifeld None None Katrinahalili None None Javakcao None None Kristinehost None None Ercamlikaya None None 11rcombs None None Hy10sm None None NMoops None None Realkyhick None None Sfernando50 None None Jlsnelson None None BradPatrick None None Jacinthamc None None Jattwarcraft None None Free8860 None None 7upnone None None Bdarwin None None Gannic None None Lucia Black None None Engl101Pina None None Nancisis None None Supergabbyshoe None None Jh10jcd None None LexOnTheWater None None Cbaur09 None None Hezzel None None Dalderdj None None SHerr None None Veliveijari None None Ostreicher None None Kevinpcarter None None Thetalkingheads None None Oombutu None None Qrc2006 None None Emettler None None GVnayR 465288976 20111211140936 Krhuske None None Aps1969 None None Bandmosh123 None None Filing Flunky 461968836 20111122174552 Filing Flunky 464002210 20111204101052 Filing Flunky 464207297 20111205131312 Filing Flunky 464238687 20111205171354 Filing Flunky 464557844 20111207121923 Filing Flunky 464625332 20111207201112 Filing Flunky 464625370 20111207201129 Jani47 None None Kaimaan None None Gr57 None None Rofu None None Hi99ins None None Rumailzaheer None None Bevhillswholesale None None TheQuebin123 None None Suro seko None None Liambroad98 None None Tarcha1115 None None Jblumrei None None Colmulhall77 None None Jzaozeldajava None None Zhele3 None None Bugskentucky None None Cpg521 None None MasterYoda222007 None None Kks862003 None None Doctorwhofan400 None None Vijila medknow None None At4470 None None Anshulkumardhiman None None DavinciRed None None Jasmine158 None None Yojay2000 None None Dsmith56 None None MineUnderworld None None Kkarma None None Carbonmotion None None Fyodor Dostoevsky None None Sociald10 None None Baxterewers None None Bwatson23 None None Monster bloke None None Gyandeep Kaushal None None Ashleycudney None None Gus77 None None Nycny None None Mawma None None Cheeseman123456789 None None DaniGurl13 None None StealthStar None None 67.182.60.246 None None DirkvdM None None Mwvlyons None None Spokane3 None None Jonty999 None None Mvc works None None Iricp None None Caleb Elliott None None Dalerichards None None Kevoy1 None None Writerslink None None Kupirijo None None Assassin\'s Creed None None Badger No.1 None None Downtheroad69 None None Thelongwikihome None None M4guir3 None None Dallasapps None None Kabir1976 None None RJT12511 None None Friendshut None None Fredler Brave None None Mills gal None None Jrevels None None ChildofMidnight None None Raghith None None SportsMaster None None Upinel None None Tkuvho None None Edesviktoria None None Schakeco None None Argos\'Dad None None 2679D 464777358 20111208151327 Bencathers None None Faizanalivarya None None Tokyo Watcher None None Gjr rodriguez None None Pallabisarkar25 None None WeKnowThis None None GlenWalker01 None None UghItsAndrew None None Simonpalakkal None None Scaptor86 None None Coin945 464034636 20111204123915 JackShestak None None L3ssm4n None None LeelanQubic None None Randomdude28 None None Bradthebear96 None None Buon professore None None Tiptoety None None M@sk None None Lordgold138 None None ThreeQuills None None Alisameredith None None Williamleute None None Nickdb93 None None Penale52 469577457 20120104202334 Penale52 470332550 20120108221341 Penale52 484230021 20120327193022 Shah 786 None None Ashsteury None None Hlearles None None Rivelino70 None None Internetbrowser None None Bschanck None None Wcability109 None None ManOfViolentCrimespsych None None Blood Red Sandman None None Dedark None None Ea3wr None None Sounders fan None None Mkortoci None None Integrated Research Limited None None Walsh15 None None Mauger73 None None DaveMedia None None Menoselmike None None WildManKY None None Smokeydude None None Intothefire None None Octorama None None Sampozx None None Squatj None None Vmanzon None None Wikiguru1318 None None Nathanf10 None None CrazyZA None None Melbrgn None None NSC-WISP None None Frankieroberto None None JayShaw TT None None DarkMan66 None None Sdesalas None None Trend Education Foundation None None Charles Edward 466199697 20111216190223 JustAnotherAZGuy None None Mannionpaul None None Wikilove99ru None None Dramaspeak1 None None CoreyBlake9000 None None Ericnotderek None None Sigma0 1 None None Lovethevoid None None Master son None None Rrrr2699 None None Uddikamaho None None Pkornrei None None 206.45.165.189 None None JesseHolmz None None Tommsesas None None Opokucharles73 None None GolfInfo None None ChemConnector None None Chrisshipley None None Batcreekstone None None ClaytonBW None None AlexxxNestor None None Ziofanale66 None None Postbil None None 12.196.37.227 None None Rickytrece None None Emmadesa None None GillesAuriault None None Jzsaffa None None Swinberg None None Dburton21 None None Macnit None None Cha2our None None Esigc None None Mbrstooge None None Kvmenges None None Prad2609 464231143 20111205162209 Rji None None Kaybvin None None Demon587 None None Ffrritz None None \xd0\xa1\xd1\x82\xd0\xb0\xd1\x81 \xd0\xa4\xd1\x91\xd0\xb4\xd0\xbe\xd1\x80\xd0\xbe\xd0\xb2 None None Bearly541 None None Themarketingsociety None None LoneWolf1992 None None IMPACTlaw None None Makky76 None None AshwiniKalantri None None ProjectNeo None None Kc4williams None None Vespristiano None None 5455oasis None None Sujit900 None None Ct None None Msauer4 None None 110808049 nisarg None None Tassedethe 487906626 20120417220917 Amirai0110 None None 68.220.24.71 None None RobinMEvans None None Frederick99 None None Leoca None None Jc349108 None None Muscarella None None TBrandley None None Mrbananaturtle None None Engl101warren None None Daisyisthebest None None DVMt None None Pureenterprise None None Wiesnerc None None Shoegal73 None None Dmmaus None None Mica97ab None None Djmissinglinks None None Amouage None None Tarassemeniuk None None Torinew None None Leila00123 None None KlementyNagorny None None HawkWang None None Roberto116 None None M.C. Brown Shoes None None Mech.bala None None Katieleeann None None Ritwik93 None None Elvia.Y. None None Quentin D F Bargate None None HKMiller None None Willarmstrong11 None None Stevemarino None None Tyc5858 None None Vibra805 None None Bananababie11 None None Skywalker80100 None None Islamocid None None RVX None None Pcap None None Wandergirl13 None None Riya Kewlani22 None None Hergilfs None None Senseless1logic None None Timmy43 None None TyKendall None None Gpolk50 None None Mdh1976 None None Trainadelaide2000 None None B.Lasonder None None Adamcoltman None None MiluTz None None Shawnn2211 None None Belgrade011 None None Pethrischool None None Elite compact None None Ganeshk None None Xboxmanwar None None Robertscoble None None When Geese Attack None None Sn3aks None None A M Khan121 None None Dara97 None None Blairleavitt None None Mr.dobrica None None Robertc555 None None D913010 None None Rogpyvbc None None Fighterf4u None None 209.178.165.58 None None Runningonbrains 485930213 20120406164846 TylerJThomas87 None None VampireKilla None None Akbangura 463757856 20111203011925 FashionSoroity None None Bigpad None None Mmoa1977 None None The Johnsonian None None Marc darra None None Imoon5 None None Steveswei None None Blcklabel92 None None Sattilite1 None None JenniferHeartsU None None Jozoisis None None 68.200.81.62 None None Vanilladude None None Mmabelo None None Sammy doug None None Yusufe101 None None Clsalvaggio None None Jorgenev 460901564 20111116052430 Der Golem None None Akmenos288 None None Crazy-dancing None None NNU-11-22100502 None None Johny Stabler None None Lars951 None None CanadianResidential None None Krohwer None None Matt Gatenby None None Alsimmons27 None None Jb351 None None WolfSearcher3 None None Ladyrose16 None None Drvw None None TechCore None None Conchboy242 None None Superkrizz None None Monirugg None None Ciclone None None Ainsworth anderson None None Michiharuharu None None Lalacentral2 None None George of the georges None None Spratton Hall School None None TriXtan None None Peel centrehalf None None Nnu-1222100535 None None Imikool2005 None None Grakelin None None Gunmetal Angel None None Hup234 None None Le Bosch None None Bob the Wikipedian None None Tiffny18 None None DreamBeliever2238 None None Robives None None Prococopon None None Cyclonerampage None None Igotask8board None None Thirty5design None None Hokaspokas None None Nicksaltr None None Hornet24 None None Finallly None None Spojrzenie None None DoubleMeDown None None Forwardgear None None Joonkee None None R\'n\'B 463219272 20111130023507 Gatedear None None Rickq None None Ashu1997 None None Socrtwo None None Manakkadanpanayamparambil None None Ptsaklan None None Ragudave None None Wikibgs None None Kchat101 None None IslandControl None None Cduensing None None Mion None None Mibelz None None WenonahRider None None Tanvirchowdhury None None Ovenkat None None Stks3 None None Unixrevolution None None Luckyme517 None None Abdalla abdirahman None None Chaoticravenger None None Knulclunk None None Amlanpati None None LosAngelesWriter 474675644 20120203004145 GMAKapusoMo None None Ckennedy2505 None None Direct000 None None Cast chris None None Alivemajor None None NermalTheBunny None None Brianbowie None None Ngord None None Island Roamer None None Spencer1984 None None Sweetfootedness None None Aprakash2000 None None The Genius 2000 None None Fatpig558 None None \xce\x91\xce\xbb\xce\xb5\xce\xbe\xce\xb1\xce\xbd\xce\xb4\xcf\x81\xce\xac\xce\xba\xce\xb7\xcf\x82 None None Bonhoffer None None Osvaldo.silva None None Mscott76 None None

The response rates to the templates are as follows:

Total test : 577

Response test : 16 (2.7729636%)

Total control : 579

Response control : 13 (2.2452504%)

The difference in the rates above are not statistically significant.

Maryana went through the deleted revisions of users that received a PROD experimental template:

So it looks like our test *did* increase the likelihood of people removing the PROD tag... unfortunately, their articles still got deleted.

Looking at all deleted revisions, 70 out of 550 control users (13%) removed the PROD tag, versus 96 out of 550 (17%) test users. Note that the absolute numbers are also a lot higher than in non-deleted revisions (where roughly 2% of each :groups removed the tag). This makes sense -- most new articles on Wikipedia get deleted in one way or another...

Discussion & Lessons

The difference in measured instances of deleted revisions 0-3 days after the posting is inconclusive. However, there may be some fruit borne in measuring the differences between high numbers of deleted revisions and low numbers of deleted revisions before and after the posting.