User:ATDT/E3Position

Notes towards an E3 position statement about development practices.

Bugzilla

Bug 38798

Our experiments are live for a short period of time, and they are optimized for
insight and speed rather than polish. They are not permanent features. They
will be taken down regardless of how well-liked they are, because they are
there to answer questions, not to be liked.

In April of 2011, the board unanimously approved Resolution:Openness[1], which
designates editor engagement and retention as the top priority for the
foundation. We won't fix the editor decline problem unless we know what is
causing it, and we won't know what is causing it unless we run a set of
experiments that test a range of hypotheses.

Multivariate testing *always* takes a toll. Think, for example, how
disorienting it would be to ask a question on IRC on the presumption that the
interface you just interacted with is universal, only to find that no one knows
what you're talking about, because they haven't been bucketed into the same
experimental group as you. We are aware of these effects and strive to minimize
them, but we will allow ourselves to behave like elephants from time to time
and we expect to be suffered.

#wikimedia-tech

July 29, 2012

13:47 (Jarry1250) ori-l: Something to get rid of the &pe= section that's confusing everyone
13:48 (ori-l) Jarry1250: but where would you redirect to if not the article page? and how would you signal post-edit state to the client except by adding a query param?
13:49 (ori-l) apologies if i'm being daft.
13:49 (Jarry1250) ori-l: Well, firstly, this seems to be appearing to people who aren't in the target group.
13:49 (Jarry1250) So surely you could just drop the parameter if the user wasn't in the target group?
13:49 (Jarry1250) I think that's the main thing that's intriguing me here
13:50 (ori-l) Jarry1250: i'd rather deal with each issue in sequence rather than jump around. if we move on to this, should i assume $wgOut->redirect would be an improvement after all?
13:50 (ori-l) wouldn't, rather
13:50 (ori-l) if you still think it would be, could you explain?
13:51 (Jarry1250) That was my proposed solution before Steven correctly pointed out you're doing most of this in JS so it wouldn't work
13:51 (ori-l) oh, okay, sorry -- i missed that
13:51 (ori-l) let me answer your other question then
13:52 (ori-l) the short answer is yes, you could drop the parameter for people not in the experiment, and that would almost certainly be cleaner.
13:53 (ori-l) the slightly longer answer entails a slight detour, which i hope you'll brook
13:53 (Jarry1250) ori-l, sure, go on
13:54 (ori-l) the E3 team is tasked with designing and deploying experiments that are 1) short-term and 2) provide insight on editor engagement and retention
13:55 (ori-l) we therefore consciously optimize for insight and development speed, occasionally at the cost of committing kludges
13:55 (ori-l) our assumption is that everything we do will be disruptive and annoying to some extent. the sheer fact of having different interfaces for different users is confusing.
13:56 (ori-l) imagine a new editor came on irc and asked about this pe=1 and you had no idea what they were talking about.
13:56 (ori-l) our conviction is that the payout from the insights generated by these experiments will offset the annoyance
13:57 (Jarry1250) ori-l: Oh, I get the generals. But I think the specifics here point towards a five minute fix
13:57 (Jarry1250) I say five minutes, I mean, I image it is.
13:57 (Jarry1250) I haven't actually checked.
13:58 (ori-l) Jarry1250: but the fix would come at the cost of debuggability
13:59 (Jarry1250) Hmm?
13:59 (ori-l) Jarry1250: deploying it to everyone makes it easier for us to test and means that if there are any problems, they'd get picked up very quickly by savvy members of the community.
14:00 (Jarry1250) Oh, I though Steven said non-new editors +anons weren't int he target group so nothing would happen for them?
14:00 (ori-l) deploying an experiment requires quite a lot of coordination between ux/ui / data / community / dev people
14:00 (Nemo_bis) ori-l> imagine a new editor came on irc and asked about this pe=1 and you had no idea what they were talking about. <-- which is likely given that they're usually told to go to #wikipedia instead of #wikimedia-tech
14:00 (Jarry1250) Also, they wouldn't notice
14:00 (Jarry1250) ^ re: nemo
14:01 (ori-l) Jarry1250: there are lots of ways things could break
14:01 (ori-l) that would be visible to everyone
14:01 (ori-l) suppose we forgot to prefix the query parameter with '&' / '?' and you got something like ?action=viewpe=1
14:01 (StevenW) re: debugging explanation etc. see https://www.mediawiki.org/wiki/Extension:E3_Experiments/Testing
14:02 (Jarry1250) ori-l: I think perhaps you're misunderstanding me
14:02 (Jarry1250) All I'm suggesting is that people who aren't bucketed don't get &pe uselessly appended to their URLs
14:02 (Jarry1250) s/don't/shouldn't/
14:03 (ori-l) Jarry1250: and what i'm saying is it's already hideously complicated to alter behavior and interface at run-time to different users based on criteria and we don't want to make it more granular than we need to because it's harder for us to reason about and makes it less likely our experiments will work
14:04 (ori-l) That was desperately short on punctuation. I can rephrase if that wasn't clear.
14:04 (Jarry1250) Which extension is this in?
14:04 (ori-l) Jarry1250: E3Experiments
14:04 (Jarry1250) I'll have a poke. Code is clearer than English
14:04 (ori-l) see /experiments/postEditFeedback
14:05 Myrii is now known as Myr
14:11 (Jarry1250) ori-l: Oh, it's all logged-in users? (Or I'm misreading that?)
14:13 (Jarry1250) Oh, no, I am
14:15 (Jarry1250) ori-l: Ah, I see, you're deciding eligibility in JS rather than PHP; I'm guessing you made that decision late because you have pef_eligibility coded for, but I can't find where it's assigned
14:15 (ori-l) Jarry1250: depends what you mean. Data collection and UI/UX changes will only happen for logged in, newly-registered editors. These changes are gated by JavaScript code that check the current environment (is the user logged in? are they newly registered? etc.) against the experiment's criteria. That gating code will run for all users, and the query string fragment is an artifact of that that is world-visible.
14:16 (ori-l) Jarry1250: "pef_eligibility" is a debug hook -- it's never added to URLS programmatically
14:16 (ori-l) Jarry1250: it's something we can use to force a coerce a certain condition to be true because (for example) it's annoying to have to keep creating new accounts while we're testing
14:16 (Jarry1250) Ah, I see. Well, I can see why this isn't easy for you. I'd assumed you'd be deciding eligibility in PHP rather than doing it all client side.
14:17 (ori-l) Jarry1250: that was a tough design decision to make, but the direction of web development is to push state down to the client
14:18 (ori-l) Jarry1250: it's easier for the back-end to send bits of information down to the client than it is for the client to report on state that may be decoupled from discrete http requests / responses
14:19 (Jarry1250) You're having the backend send down data and then reconstructing it at the moment, rather than calculating first and then sending a single thing. Anyway, what's done is done.
14:19 (ori-l) Jarry1250: and scattering bucketing logic in both PHP and JS code is too confusing
14:20 (ori-l) Jarry1250: yup.
14:21 (ori-l) Jarry1250: let me add one last point -- though we apply different standards to experiments than we do to permanent features, it isn't our intention to be cavalier about breaking things
14:21 (ori-l) if what we're doing is seriously disruptive, it's not acceptable, and we should fix it
14:23 (ori-l) we're principled about being kludgy :)
14:23 (Jarry1250) Yeah, I'm just trying to work out if there's actually a fix to this...
14:23 (Jarry1250) It seems like there *should* be, but apparently not...
14:23 (spagewmf) ori-l , is ?pe=1 related to en wiki edits redirecting to /w/index.php?title=Page_name&pe=1 link rather than a more palatable /wiki/Page_name?pe=1
14:24 (Jarry1250) spagewmf: which of this is chosen is decided by MW, I think
14:26 Jarry1250 looks up the ArticleUpdateBeforeRedirect hook
14:27 (ori-l) yeah, that's my understanding. i presume the addition of an unfamiliar query param is the trigger but i don't know for certain.
14:29 (Jarry1250) I *think* it's in Title::getLocalUrl but it's too late for me to be able to follow that
14:29 (Jarry1250) I *think* it's in Title::getLocalUrl but it's too late for me to be able to follow that
14:30 (Jarry1250) Anyway, to answer the original point, you can't hide it because it's not actually related to eligibility as I had assumed, but merely to the fact that it's a post-edit page at all.
14:31 (Jarry1250) (I mean, you could related it to eligibility but that would mean porting the detection from JS to PHP which would be a pain for reasons previously stated)
14:31 (ori-l) Jarry1250: right. you could tidy it up by delegating some of the eligibility logic to PHP-land, but that would suck
14:31 (ori-l) yes, exactly
14:32 (Jarry1250) Okay, well that's my curiosity sated. :)

See also

VPT discussion (PEF-specific)