User talk:James Salsman/Inactive administrators survey

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search

Discussion[edit]

In 2009-10 the Community Health Strategic Planning Task Force produced an inactive administrators survey to try to determine why so many administrators were leaving the English Wikipedia. Last we heard from the Foundation, it had been approved and was awaiting "resourcing." Because there is now statistically significant evidence that admin attrition is causing the decline in active editors, it is more important than ever to attempt to address administrator attrition. The first step in doing so is to follow through and survey the inactive administrators. The LCA announcement does not entirely help determine whether the survey is a "community advocacy" or "editor retention" function, so perhaps this is something for Zack Exley's department. But the survey itself is up and ready for distribution on Google Docs Forms. If the Foundation can not find the resources to do this, I intend to distribute it to inactive admins by email this month and tabulate the data myself. I would much prefer that the Foundation follow through and do it to save me the time and effort. James Salsman 11:52, 10 February 2012 (UTC)

James, given the timeline you lay out, there's absolutely no way that this team would be in a position to help. As we mentioned in the announcement, we don't anticipate being fully ramped up for six months to a year, at the earliest. In fact, we intend for our immediate engagement to be "planning", not "doing". I see a zero percent chance that we'd be in a position to help with this "this month". And that's leaving aside the question of whether it's even in scope. Sorry to be the bearer of bad news on this one. Philippe (WMF) 11:43, 10 February 2012 (UTC)
I missed the 6-12 months part, sorry. Well, the survey is up, I've got the list of inactive admins, and I'm told they almost all have email enabled. I guess I should just send out the links, share the spreadsheet permissions with Zack, and summarize the results in a couple weeks? James Salsman 12:03, 10 February 2012 (UTC)
Update: I've completed distribution to 308 inactive admins and have given Steven Walling access to the raw results. There is a very encouraging response rate, with 23 respondents so far. That should allow for statistically significant answers to most of the questions people will have about the responses.
I will post my analysis of the initial results at Inactive administrators survey in a week or two. James Salsman 01:37, 12 February 2012 (UTC)
In the interests of not distracting this page, would you mind terribly if I move the discussion about this survey (which we've established wouldn't be within the purview of this team) to another location on meta? I agree with you that the teaser is fascinating to see. Philippe (WMF) 05:22, 12 February 2012 (UTC)
Sure.... James Salsman 06:41, 12 February 2012 (UTC)

Re the claim "There is statistically significant evidence that admin attrition is causing the decline in active editors", all I saw was a Granger test, which is in no way significant at this stage of analysis. A Granger test presupposes that the two variables tested are the ONLY variables in operation. In the case where a third variable is driving the series on a different lag basis, Granger tests are actually quite misleading, as they are unable to rule out (or even consider) that a third variable might be the cause. A Granger is typically used as a litmus test to see in what direction further hypotheses should be developed. But until there is clear evidence that there are no other variables in effect, no causality has been reliably demonstrated. Manning 12:36, 13 February 2012 (UTC)

I think we can safely say that "administrator attrition is useful in forecasting editor attrition" without making any claims about direct causality. If there is a third common cause, it would have to be something that affects admins about six months before it affects editors on average. Any ideas for what that could be? I'm convinced that the archive time on WP:ANI has a vast amount of influence over objective measures of editor interest and general civility. It will take me some time to prove it. I'm glad I went ahead with the survey, though. The results have already been very encouraging, and responses are still coming in. If you have any particular questions you want me to examine after looking at the survey questions, please let me know and I will do my best. James Salsman 16:50, 13 February 2012 (UTC)
Oh I agree that it certainly appears useful in forecasting. However the claim was that there is direct causality and that has not been established. As far as other potential variables, length of service, degree of interpersonal conflict, edit count... just off the top of my head. Manning 19:54, 13 February 2012 (UTC)
Agreed. This statistical test is not appropriate for determining causality. It is merely determining correlation with a time-lag. You must consider confounding factors when attempting causal analysis. --EpochFail 23:14, 13 February 2012 (UTC)
I think Granger causality is somewhat stronger than that, as Granger detailed in his 2003 Nobel Memorial Prize in Economic Sciences lecture (see pp. 365-6.) But as I said above, thought that there was a third common cause: the length of time sections remain on w:WP:ANI, but I've just looked at it from 2005 to the present and conclusively ruled it out. How could length of service, degree of interpersonal conflict, or edit count be a factor? Those are uniformly smoothed out across the admin population due to the law of large numbers. I'm seeing no correlation between the days since first edit and either the days since last edit or edit count at all. What should I be looking at for those? James Salsman 23:46, 13 February 2012 (UTC)

Sent to[edit]

The survey was distributed to the following 308 inactive administrators listed at w:Wikipedia:Former_administrators#Desysopped_for_inactivity and w:Wikipedia:List_of_administrators/Inactive (listed as inactive since up until the end of August, 2011.) Note that these do not include those who voluntarily resigned their adminships before leaving, who might number in the hundreds too

17Drew 52_Pickup ABCD Abecedare Acetic_Acid Adambiswanger1 Adashiel AdjustShift Admiral_Norton Admrboltz Aecis Akradecki Aksi_great Alai Alex756 Alhutch Alphachimp Alteripse Amberrock Anas_Salloum Andrew_Norman Andris Android79 AndyZ AnonEMouse Anonymous_editor Appraiser Astronautics Awolf002 Backslash_Forwardslash Bellhalla Bjarki_S Bobet Bookofjude Bratsche Brockert Bubba_hotep Buchanan-Hermit C12H22O11 CBDunkerson CJLL_Wright CSTAR Caribbean_H.Q. Carlosguitar Catbar CattleGirl Chmod007 Cholmes75 Chris_Roy Christopher_Mahan Chunky_Rice Ck_lostsword Cnwb Cobaltbluetony Coelacan Conscious Consumed_Crustacean Coralmizu DESiegel DJ_Clayworth Dan100 DanMS Daniel Daniel_Olsen Daveydweeb David_Kernow DeadEyeArrow Decumanus Denni Dfrg.msc Diberri Dijxtra Dina DoomsDay349 Dori DrBob Duja Duk Dvyost EWS23 Elf-friend Encephalon Enochlau Espresso_Addict Ev Evilphoenix FCYTravis FF2010 Fantasy Feco Femto Fir0002 Firsfron Flcelloguy Flowerparty Freakofnurture Fred_J FreplySpang Fropuff G-Man GDonato Gaz Gazimoff Gladys_j_cortez Goodoldpolonius2 Greeves GregAsche GregRobson Grm_wnr Guettarda Hajor Hall_Monitor Heah Hiding Humus_sapiens Husond Ikiroid Improv InShaneee Interiot Irishguy Isomorphic Isotope23 J-stan JForget JPD JWSchmidt Jallan James086 JamieS93 Jaxl Jedi6 Jersey_Devil Jersyko Jimregan Jkelly Joe_Beaudoin_Jr. Jogers Joke137 Joshbuddy Jrdioko Jreferee Jza84 Katefan0 Kcordina Keilana Kimchi.sg KnightLago KnowledgeOfSelf Knowledge_Seeker Ktsquare Kukini Kungfuadam L'Aquatique Lee_Daniel_Crocker Leebo Lexi_Marie Lightdarkness Ligulem LiquidGhoul Lommer Loren36 LouI Luigi30 Lupin MBK004 MPF Mahanga Mallanox Mangojuice MarkSweep Martyman Marumari Masamage Matt_Britt Matt_Crypto Mattbr MatthewUND Mdd4696 Meegs Mel_Etitis Merphant Miborovsky Mirv Mkweise Monotonehell Moreschi Morwen Mr._Lefty MrKIA11 Myleslong Nae'blis Nakon Ngb Nickptar Niteowlneils Nixeagle Nousernamesleft Nufy8 Okiefromokla Olessi Orioane Oxymoron83 PDH PS2pcGAMER Pablo-flores Pagrashtak Paxse Pcb21 Pepsidrinka Persian_Poet_Gal Pgk Phaedriel Phil_Bordelon Phil_Sandifer Picaroon Pilotguy PinchasC Plange Poeloq Poor_Yorick Psy_guy Quercusrobur R3m0t Radiant! Ram-Man Ramallite Ran Redux Redvers Reflex_Reaction Remember_the_dot RexNL ReyBrujo Rhobite Rholton Riana Rjd0060 Rje Rlandmann Robth RobyWayne Rockero RoseParks Rune.welsh Sam_Korn Sam_Vimes Samuel_Blanning Sango123 Scm83x Searchme Seb26 Selket Sephiroth_BCR Seresin Shauri SheffieldSteel Shimeru Shirahadasha Singularity Sjorford Slowking_Man SmthManly Sn0wflake Snoyes Sortior Soumyasch Sue_Anne TFOWR TKD TSO1D TUF-KAT Talrias Tangotango Tanner-Christopher ThaddeusB TheProject The_Epopt The_Placebo_Effect Theresa_knott Thorpe Thunderboltz Thunderbrand Tillwe TomTheHand Trebor Turnstep UkPaolo Vassyana Veinor Verrai Visorstuff Wafulz Wernher Where Wiki_alf Wikibofh Winhunter Xiner Y0u Yacht YellowMonkey Yonatan Zedla ZimZalaBim Zoney Zsinj

Not sent to[edit]

No email[edit]

The following inactive admins had no email configured in their preferences

--_April @pple Aarktica Accounting4Taste Ams80 Aranel Arminius Asbestos Benc Blankfaze Bluemoose Borisblue Canadian-Bacon CanadianCaesar Cdc Cedrus-Libani Cgs Chadloder Clarkk Clifford_Adams CryptoDerk Curps DF08 Darthgriz98 DropDeadGorgias Duncharris Dustimagic Dysprosia Efghij Evil_Monkey Filiocht Francs2000 Func Goatasaur Grunt Hcheney Hephaestos Hermione1980 Imran JHK Jdavidb Jeronimo JonMoore Joolz Josh_Grosse Karen_Johnson Khendon LC LittleDan Lord_Emsworth Lradrama MC_MasterChef Mackeriv Malcolm_Farmer Mark_Christensen Mark_Richards Marshman Maximus_Rex Mbecker Mic Mintguy Mustafaa Notheruser Opabinia_regalis Optim PeaceNT Peter_Winnberg Pfortuny RadicalBender Recurring_dreams Rlquall Royboycrashfan SD6-Agent Salsa_Shark Scimitar Sean_William Seglea Shadow1 Someone_else Steinsky TPK Tannin Tarquin Texture The_Singing_Badger TimShell Tom- Wapcaplet Water_Bottle WojPob Yelyos

Refusing email[edit]

Presumably the Foundation system administration staff could email these 24 users if they were so inclined, but administrators who voluntarily resigned before becoming inactive would probably be a far more fruitful source of respondents if any further inquiry is necessary

Cantthinkofagoodname Chancemill CyborgTosser Galwhaa Gator1 Hdt83 Izehar J_Di Jmlk17 Lachatdelarue Misza13 Mo0 Musical_Linguist Oberiko PZFUN Pastordavid Phils Sannse Scipius Sfoskett Trilobite WODUP Woggly Alex_S

Finally, User:Midom is a personal friend who still volunteers for the Foundation. James Salsman 02:07, 12 February 2012 (UTC)

Excuse me?[edit]

James, did you really send this survey from a banned user account, using a Foundation staff member as the contact, and without going through the research committee? That's a gross violation of research ethics, and discounts the whole study. This is a flagrant disregard for the community's norms and trust. I have blanked the content page until this issue is resolved. Philippe (WMF) 22:02, 13 February 2012 (UTC)

Sorry, I don't understand how the content of the page is considered in violation of anything, could you explain? As far as I can guess, some users didn't understand that a main namespace page is not necessarily an official one, and I've hence moved it in user space, although I considered it clear enough when I saw it the first time. As for the way users were contacted, if it's an abuse of the email feature, other sort of "spam" not authorised by the RCom or whatever it should be reported to the appropriate venues. Nemo 22:19, 13 February 2012 (UTC)
Hey Nemo. There's a few problems here, as Philippe pointed out...
  1. No one should be using Meta to run a survey that hasn't been approved by the volunteers at RCom, in order to both reduce frivolous surveys that annoy the community and to ensure ethical research.
  2. James sent out this survey as a banned editor on English Wikipedia, using a sock. If he wants to participate in meta discussions here, that's fine. But he can't be using Meta as a staging ground for evading a community ban on any Wikipedia. In light of that, we may just want to delete it in order to press home the point that it's not okay.
  3. He sent out this survey with a footer that said I personally was the Foundation point of contact for it, even though I never said I was okay with that. He sent out a hundred emails before I even replied, and when I asked him not to do it more, he completed the 300 emails to admins.
Thanks for moving this to userspace. Philippe is just trying to save people headache here, especially from anyone who might think that by having a page on Meta, that it's official or okay in any way. (See also: [1]) Steven Walling (WMF) • talk 22:43, 13 February 2012 (UTC)
  1. [edit conflict] I disagree. Where is this stated?
  2. Ok, but I don't see what he made wrong here. If the page was (is?) used to misrepresent his position, moving it to user space should certainly be enough. Or is more needed?
  3. Thanks for pointing this out. I understand this happened on en.wiki (right?). Such an abuse of the email feature, anyway, should be prevented on any wiki as far as possible. If the user is likely to do it again, a sysop should be asked on WM:RFH to investigate and consider further action here. I've blocked the account User:Survey in the meanwhile. Nemo 00:07, 14 February 2012 (UTC)
Steven, I had no idea that there was even a "Research Committee" until the first time you asked me to stop, by which time -- as I plainly told you -- all of the emails had already gone out. When two days earlier I had asked who could be the point of contact, you said I could "talk to" you. When I told you that I was going to go ahead, you did not reply. You and I had a detailed IRC discussion about the general unwillingness of the Foundation to examine admin attrition about five months ago. The survey itself has been discussed with extensive feedback for two and a half years now. There is nothing on Notes on good practices on Wikipedia research which is contrary to the way in which this was conducted. Banned and blocked users on the English Wikipedia are explicitly allowed to send emails to administrators. Is there something in particular that you think I did wrong? James Salsman 23:59, 13 February 2012 (UTC)
Can't comment the rest, but I'm quite sure «Banned and blocked users on the English Wikipedia are explicitly allowed to send emails to administrators» is a misrepresentation of the policies. This should be discussed on en.wiki, and if they consider it an abuse then yes it is such. Nemo 00:10, 14 February 2012 (UTC)
WP:UNBAN specifically allows it. I consider this survey to be part of my appeal. James Salsman 00:17, 14 February 2012 (UTC)
The purpose of that line in the policy is for you to communicate with the blocking admin in question, not to send email surveys. You're wikilawyering here. Steven Walling (WMF) • talk 01:20, 14 February 2012 (UTC)
I don't think taking an action which the Foundation explicitly approved 2.5 years ago in order to bolster an appeal case is anything of the sort. James Salsman 01:33, 14 February 2012 (UTC)
In addition to the above, because personal information may have been disclosed to an unauthorized third party (James), at the instruction of the WMF's legal team, I have deleted the page as an OFFICE action. Such actions should not be reverted without going through the Foundation's Legal and Community Advocacy department. Philippe (WMF) 00:14, 14 February 2012 (UTC)

Lack of RCom review[edit]

This survey has not been reviewed by the Research Committee and as such it's in violation of our Research:Subject recruitment. When we receive a recruitment request (whether it's from external researchers or community members) we ask the author of the proposal to specify, prior to starting any form of recruitment:

  • the purpose of the study
  • the affiliation and contact details of the main investigator
  • the target sample size for the study
  • the proposed recruitment method
  • the terms under which the data is being collected (including references to any privacy policy or IRB approval, if available)
  • the terms and licenses under which the data and the findings will be shared

Projects are then categorized by type of support request and marked as pending review until the Research Committee has carefully reviewed them and given its green light. Individuals who attempted in the past to bulk-mail or bulk-contact editors via user talk pages without permission have been immediately blocked/pointed to the RCom pages to have their proposal documented. This is an example of a recent request that we received and reviewed with the author. To my knowledge, the author of the inactive administrator survey didn't seek any support from RCom to review this survey. As such it should be expected that the proposal be flagged as not approved or removed from areas of Meta that may give readers the impression that this project was approved by RCom or by the Wikimedia Foundation. --DarTar 00:35, 14 February 2012 (UTC)

Thank you for this explanation about subject recruitment and this particular case. I personally knew all this, what puzzled me is that someone seemed to say that in general you can't even publish any "research" on Meta without RCom authorisation. As research latu sensu on Wikimedia projects is one of the main and oldest scopes of this wiki, that would be very surprising. Can you confirm, please? Grazie, Nemo 00:51, 14 February 2012 (UTC)
There are several informal research proposals or even research stubs on Meta and we should definitely allow anybody to freely discuss research ideas on the Research Index (that's the purpose of it and yes, it's a wiki so it's ok to draft and refactor), but that's different from posting information on a completed research project with data collected without any formal review and with WMF staff information indicated as an (unsolicited) contact point for this project. --DarTar 02:02, 14 February 2012 (UTC)
Unsolicited?
[09:41am] jsalsman okay, well I guess the first thing I need is to know who in Zack's department will be point of contact for editor recruitment efforts
[09:41am] Philippe jsalsman: that hasn't changed.
[09:41am] jsalsman who then?
[09:41am] StevenW jsalsman: you can talk to me and Maryana
[09:41am] jsalsman okay
...
[09:59am] jsalsman StevenW: I'm going to go ahead with the three-year old inactive admins survey and send you access to the results spreadsheet
-- http://meta.wikimedia.org/wiki/IRC_office_hours/Office_hours_2012-02-10
Steven's claim that I sent emails after I was asked to stop sending them is easily disproved. The last email went out Saturday at 4:54 PM and Steven asked me to stop sending them at 9:41 PM. I await your reply to my email. James Salsman 04:26, 14 February 2012 (UTC)
DarTar, would you please send me an email so that I can contact you directly? James Salsman 01:00, 14 February 2012 (UTC)
No problem, mail for you --DarTar 02:02, 14 February 2012 (UTC)
Replied. James Salsman 03:03, 14 February 2012 (UTC)

I am awaiting a reply from DarTar, but in the mean time I would like to point out that the "research policy for subject recruitment" which he links to and describes as having been violated is actually a FAQ question which links in turn to this page which says, "This page describes how to get support for research projects from the Wikimedia Foundation.... This page exists to describe a process by which researchers can obtain community approval to recruit research participants for surveys, interviews and experiments. Until an official policy is approved by the Wikimedia Research Committee regarding subject recruitment, individual requests can be submitted following these instructions." (emphasis added) I question the ethics of referring to a process which is clearly marked as optional and apparently has not been officially approved by the Research Committee as having been violated. I have asked some related questions on the wiki-research-l list. James Salsman 06:29, 14 February 2012 (UTC)