Jump to content

Growth/Activation

From Meta, a Wikimedia project coordination wiki

These requirements apply to PEF as well as any other experiment with (1) an eligibility check based on registration date (for example experiments only targeting new editors or editors with accounts older than N days) and (2) a limited and consistent duration of the treatment for each participant in the experiment.

Definitions

[edit]
Milestones
start of the experiment (deployment completed)
activation buffer period
beginning of eligibility period
end of eligibility period
eligibility period
registration time
duration of treatment
treatment period
end of treatment
end of the experiment
Terminology
experiment the duration of the experiment for the experimenter, spanning the whole process of deploying the code to production, collecting data and disabling the code
eligibility the set of conditions that determine the population of participants, i.e. whether a user can participate at all in the experiment
bucket the actual condition to which a participant is assigned to if in the eligible population (i.e. an experimental treatment, a control group or nothing if groups are a sample of the eligible population)
treatment the actual experimental condition any individual participant is exposed to during the test

Experiment activation workflow

[edit]
  • the code gets deployed at , we allow a buffer of time before the experiment is activated and anyone can participate in the experiment;
  • is the period that determines if a user is eligible as a participant in the experiment. If the user registration date falls within this period the user is eligible, otherwise nothing happens
  • if the user is eligible, than a bucketing function determines which condition the user is assigned to. Conditions can be two (in case of a split test) or multiple (in case of multiple treatments).
  • The duration of the treatment is calculated from the time at which the user becomes eligible as a participant (the registration date) and ensures that every participant is exposed to the treatment for the same duration regardless of when she became eligible. This will ensure, in particular, that users who registered towards the end of the eligibility period will still get exposed to a sufficiently long treatment to produce usable data.
  • The overall duration of the experiment will need to extend at least days after the eligibility period is over.