User:MPopov (WMF)/Sandbox/Measurement plans and instrumentation specifications

New data collection activities (e.g. analytics/telemetry instruments) required review & approval from Legal, Security, and Safety teams before they can be deployed. When submitting a request for review & approval, the request needs to provide answers to the following:

• What data do you want to collect?
• Why do you want to collect it?
• How will you collect it?
• What will you do with the data once you've collected it?

Measurement Plan

A measurement plan describes how you intend to evaluate the success/impact of a product/feature. It documents the hypotheses you have:

For [user group(s)], if we release [feature] it will have [effect] because [rationale] as measured by [metric].

(Example of a hypothesis phrasing.)

In it, you specify what data you need to calculate all the metrics:

• key metrics: your goals, what you're trying to move the needle on
• (optional) leading indicators: proxy metrics that are related to the key metrics but more sensitive to changes and faster to measure / quicker to check after deployment
• (optional) guardrail metrics: what you don't want to negatively impact

Example: T346086

You document what data already exists (and what those sources are) and what additional instrumentation needs to happen to collect the data you need. You also document what retention policy you're planning to use (e.g. standard 90 days or request for extended retention).

Instrumentation Specification

An instrumentation specification (template) documents all the additional data you'll need to collect and is useful for engineers to make sure they've instrumented everything that is needed so that all the metrics of interest will be measurable. This is also useful for:

• pre-deployment QA to make sure all events are being produced when they should be and with the necessary event data
• post-deployment data QA to make sure we're receiving all events and that the event data attached to them is what we would expect to see

When you submit a request to Legal, Safety, and Security for review & approval, you should provide both because together they describe why you need to collect data, what data you need to collect, and how you will collect it:

• Measurement Plan: why + what
• Instrumentation Spec: what + how