Sampling (ca. September 2016-August 2017)
Sampling will be derived by passing in the session-pegged psuedorandom mw.user.sessionId() value (ephemeral cookie) and potential buckets into mw.experiments.getBucket() and determining whether to enroll the user for event logging based on the bucket response. For example, if 10% of sessions should be enrolled for event logging, the groups might be named Control (with a value of 0.9)and B (with a value of 0.1) and if mw.experimentsgetBucket() returns B the session is in scope for event logging.
Do note that devices without sendBeacon support will not be eligible for sampling. This is by design because without sendBeacon there is a lower probability of correctly conveyed clickthroughs. This sendBeacon constraint at the moment means the data will be biased toward Firefox and Chrome users.
- Sampling rates varying per a/b test. For it and ru a/b tests, we have no sampling for logged-in users. Sampling rates are set at 0.01 for ruwiki and 0.02 for itwiki for anon users.
- - OVasileva (WMF) 09.37, 23 September 2016 (UTC)
- The Popups instrumentation is enabled on all wikis that have the Popups extension loaded, i.e sewiki and all wikis that have the Beta Features extension loaded. Per 340706, the sampling rate is currently 0.05% of all distinct browser sessions. Phuedx (WMF) (talk) 14:58, 2 March 2017 (UTC)
Bucketing (August 2017-)
At the time of writing [August 25, 2017], no EventLogging is occurring for any of our projects.
Each anonymous user will be assigned one of the following buckets:
off. These buckets have the following behaviours:
- If the user is in the
onbucket, then Page Previews will be enabled by default and the EventLogging instrumentation will be enabled.
- If the user is in the
controlbucket, then previews will be disabled by default and the instrumentation will be enabled.
- If the user is in the
offbucket, then previews will be disabled by default and the instrumentation will be disabled.
The bucket is calculated by passing in the session-pegged psuedorandom mw.user.sessionId() value (an ephemeral cookie) and buckets into mw.experiments.getBucket(). The
control buckets are defined as
0.5 * $wgPopupsAnonsExperimentalGroupSize (X), with the
off bucket capturing the rest of the population (i.e.
1 - X).
For example, if
$wgPopupsAnonsExperimentalGroupSize is 0.1 then 0.05 user sessions will be in the
on and control buckets.
Do note that devices without Beacon API support will not be eligible for the EventLogging instrumentation. This is by design because without the Beacon API there is a lower probability of correctly conveyed clickthroughs. Currently, this constraint means the data will be biased toward Firefox and Chrome users.
enwiki and dewiki A/B test v2 (October 18th - November 15h)
On Wednesday, October 18th, we re-enabled the Page Previews enwiki and dewiki A/B test with the following bucket sizes:
|Wiki||Bucket Sizes (on:off:control)|
These bucket sizes were chosen to achieve an average rate of circa 200 events/second over the course of two weeks. They're slightly tweaked versions of the bucket sizes derived in T172291#3500535.
The instrumentation for the test was deactivated on November 15 (T178500).
enwiki and dewiki A/B test v3 (December 12-)
Per task T170169, Page Previews (Popups) can be configured to not load on a page. We refer to these pages as "blacklisted pages". If a page is blacklisted, it follows that the Page Previews instrumentation won't execute and we won't collect data regardless of whether we've been doing so (and will continue to do so) for the rest of the user's session.
Should ReferenceTooltips be logged in a separate schema
@Thiemo Kreuz (WMDE): - I see the schema has been updated to include ReferenceTooltips. I wonder if it might make sense to use a new schema, given you'll likely want to log reference tooltip previews separately from page previews. I'd imagine reference tooltips will display less often then link previews so if you are leaning heavily on the existing logging code, the sample size for reference tooltips is likely to be very small in comparison to link previews. Does it make sense to "fork" this schema and update the code so that it doesn't log link previews at all, or is data relating to link previews important for the questions you are trying to answer? Jdlrobson (talk) 18:37, 22 January 2019 (UTC)
- Thanks for asking. These are all excellent questions we don't have an answer for, yet. I'm aware my edit does not have an effect until the code is updated to use the new revision ID. I just wanted to leave this here so we don't forget. I created phab:T214493 for us to keep track of this. --Thiemo Kreuz (WMDE) (talk) 17:18, 23 January 2019 (UTC)