Help:CentralNotice/Metrics

Like most platforms that provide banner advertising tools, it is possible to measure the impressions of users on Central Notice to evaluate the effectiveness of banners. This data is currently sampled on all banners, and collected internally within the Wikimedia Foundation.

To help the community see the impact of banners they run, we will provide this access to Stewards who are involved with the Central Notice request process.

In order for a Steward to access banner metrics and use them in decision making, you need to understand the following step: 1) getting to an LDAP NDA system, 2) using the Turnilo Board to do analysis of the Central Notice banner impressions, and 3) subsequently choosing which data to share with Banner requesters or with the community.

Getting Access: Request an LDAP NDA

In order to access data for Central Notice Impressions, you need LDAP NDA access. You will need to create a Developer Account for Wikimedia servers, that is associated with a unified identity that then gets these additional access rights.

This process is managed on Phabricator, with instructions here: https://wikitech.wikimedia.org/wiki/Volunteer_NDA

This includes several steps:

Requesting LDAP NDA access through the second stage of the process documented at https://wikitech.wikimedia.org/wiki/Volunteer_NDA (Typically @KFrancis Phabricator). In the request cc Fiona Romeo (@FRomeo_WMF Phabricator) as the sponsoring staff for the request.
Confirming this access with a manager at the Wikimedia Foundation (for Central Notice Admins we are managing this approval through the Director of Fundraising Tech @greg on Phabricator ).

If you encounter any complications during the process, please email campaignswikimediaorg .

Using Turnilo’s Banner Activity Minutely Board

Metrics for Central Notice are available through Turnilo. To see documentation of the Wikimedia Foundation’s Turnilo dashboard, check out: https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Turnilo

The metrics important to Central Notice Admins are available as part of #banner_activity_minutely (https://turnilo.wikimedia.org/#banner_activity_minutely/). This table includes tracking of banner impressions through sampling of users seeing banners and you can learn more about the data structure as part of the the Central Notice Extension (https://www.mediawiki.org/wiki/Extension:CentralNotice)

On right is the initial dashboard visible in Turnilo. Remember to use the status code "6" as a filter and "Normalized request counts" as the Measure when setting up your analysis. Do not screenshot any data from the dashboard when publishing analysis.

Analyzing the actual impressions of a banner campaign

To begin the analysis, you will need to configure two settings that need to be added to all banners:

Set your Measure to “Normalized Request Count”. This tells us a “real” number of banner impressions, not the sampled estimate provided by the Request Count metric.
Add a filter for “Status Code” 6 which indicates that the banner was shown to a user. This sets the filter to only show banners that were actually shown to readers. You can learn about other status codes here. Other status codes such as 2 (2.1, 2.2, etc) can help explain why users might not have seen the banner.

With these initial settings, now it’s time to do analysis! You can either “Split” or “Filter” by various other data fields stored in the underlying table.

Useful variables for evaluating community campaigns might include:

Time -- the period of time from which you want to evaluate the data
Campaign -- this describes which campaign was shown to users
Banner -- this shows what banner was shown to users
Uselang -- this describes the language of the browser, which is what filters
Country -- this field uses ISO_3166-1 codes
Subregion -- for some countries, the banners track a specific subregion as well.
Anonymous -- this field evaluates if the user was logged in or not

Other data points are described here and the meaning of some of this data can be better understood if you examine the extension documentation here.

Hint: You can have more than one Filter or Split in the dashboard. This allows you to evaluate a campaign or banner against other variables in the table.

Note: the data is updated daily, with refreshes of the data monthly and Sanitization of two of the more sensitive data points "Region" and "Device" every 90 days. Please see the next section for guidelines on how to share the data from the tool.

Considerations for sharing the data from Turnilo

Now that you have done some analysis, you probably want to share some of the data to inform further decisions with the requesters for the banners.

Generally speaking, data accessed in Wikimedia Servers, before being published or shared, needs to be evaluated for risk against two major policies: the Data Protection Guidelines and the Wikimedia Foundation Country and Territory Protection List.

As someone with access to NDA protected data, you are responsible for evaluating the potential risk of this data before sharing it. To mitigate risk, you will share only what is necessary for the purpose specified (“data minimization”) and only when the data publishing is “low risk”. Higher risk data publication needs to be vetted by the Wikimedia Foundation’s Legal Department. In the context of the guidelines, you specifically should aim not to share data that could lead to identification of users or other privacy or safety risks.

In line with data minimization principles, please only share data publicly about community request initiated banners in your role as a Central Notice Admin. Please consult WMF or respective affiliates if you want to share analyzed data related to banners focused on fundraising, governance or surveys. Your NDA access is only intended for analysis of the #banner_activity_minutely/ board.

In the context of the Turnilo board for Banner Activity Minutely Board, be especially aware of the following concerns when sharing data:

Please do not take a screenshot of the Turnilo dashboard and publish that screenshot directly. Instead, analyze the data, removing countries from the protection list and other identifying information following the guidelines above, before sharing.
Please be extra mindful of the guidelines for publishing data related to medium risk, high risk and “do not publish” countries.
Focus on publishing large numbers which help requestors and central notice admins look at high level impacts of campaigns. Note the guidelines only allow publishing high level data of pageviews of at least 250 pageviews, larger numbers are always better.

If you have any questions about how to interpret the guidelines for publishing data please contact campaignswikimediaorg, with your question and they can find help within the Foundation for interpreting the guidelines.