Research:Standard metrics

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
Metrics standardization, Wikimedia Research & Data Showcase, March 2014

Researchers, analysts, and product managers use a wide variety of metrics (from "monthly active editors" to "user's giving proportion in the dictator game"[1]) track and evaluate phenomena related to the Wikimedia projects. This page collects metrics which are suitable for wide use, which will make it faster to develop new research projects and easier to compare existing ones.

These metrics are mostly quantitive, but qualitive metrics are worth standardizing too. For example, researchers sometimes survey Wikimedia users and contributors about their subjective satisfaction with software. It would be sensible to devise a standard, well-considered way of asking such questions.

A high-level overview of the design of Rolling Monthly Active Editors, June 2014

Background[edit]

An example of sensitivity analysis for the new editor definition: monthly count of newly registered users on the German Wikipedia performing at least one edit in their first day/week in the article namespace or across all namespaces.
Analysis example. An example of sensitivity analysis for the new editor definition: monthly count of newly registered users on the German Wikipedia performing at least one edit in their first day/week in the article namespace or across all namespaces.

Overview[edit]

One way to group standard metrics is into 5 categories:

New users
these metrics provide indicators on the acquisition, activation and productivity of users joining Wikipedia or other Wikimedia projects for the first time.
Community
these metrics measure the overall composition, growth and volume of activity of existing communities, including both human and automated activity by bots.
Content
this category of metrics measures the growth and dynamics of content creation, including edits, new articles, uploads.
Curation
these metrics measure the quantity and quality of curation and moderation activities, such as reverts, deletions, blocks.
Traffic
these metrics measure traffic and readership of Wikimedia projects.

Evaluation[edit]

Each metric and user class definition comes with supportive analysis whose goal is to understand how sensitive its definition is to specific parameter choices and whether the metric captures the same phenomenon in different projects. We strive to run sensitivity analysis across projects in different languages and of varying levels of maturity, but we welcome feedback to improve these definitions and to identify edge cases, particularly for smaller projects or projects with uncommon policies, where the proposed definition may not accurately capture the quantity it attempts to represent.

We also expect the use of these metrics in the first iterations of the design of Editor Engagement Vital Signs to reveal anomalies and interesting facts that are hard to anticipate until series for each metric are automatically generated for each Wikimedia project.

New users[edit]

Newly registered user[edit]

A is a previously unregistered user creating a username for the first time on a Wikimedia project.

Depends on
none
Used in
New editor

New editor[edit]

A is a newly registered user completing edits to pages in any namespace of a Wikimedia project within days since registration ().

New editor
Standardized definition
  • = 1 edit
  • = 1 day
Depends on
Newly registered user
Used in
Productive new editor

Productive new editor[edit]

A is a new editor who completes at least productive edit(s) within time since registration ().

Productive new editor
Standardized definition
Depends on
New editor
Used in
none

Surviving new editor[edit]

A is a new editor who completes at least edits within time since registration () and also completes edits in the survival period .

Surviving new editor
Standardized definition
  • = 1 edit
  • = 1 edit
  • = 1 day
  • = 30 days (~ one month)
  • = 30 days (~ one month)
Depends on
New editor
Used in
none

Community[edit]

The editor model[edit]

The editor model is a suite of metrics which include subcslasses of and funnel rates for monthly active editors.

Rolling active editor[edit]

A is a registered user who completed edits to pages in any namespace of a Wikimedia project between and .

Active editor (rolling)
Standardized definition
  • = 5 edits
  • = 30 days

Rolling new active editor[edit]

A is a newly registered user who both registered and completed edits to pages in any namespace of a Wikimedia project between and .

New active editor (rolling)
Standardized definition
  • = 5 edits
  • = 30 days
Depends on
Newly registered user
See also
Rolling active editor

Rolling surviving new active editor[edit]

A is a newly registered user who both registered and completed edits between and and continued to complete edits between and .

Surviving new active editor (rolling)
Standardized definition
  • = 5 edits
  • = 30 days
Depends on
Newly registered user
Rolling new active editor
See also
Rolling active editor

Rolling recurring old active editor[edit]

A is a user registered before , completed edits between and and continued to complete edits between and .

Recurring old active editor (rolling)
Standardized definition
  • = 5 edits
  • = 30 days
See also
Rolling active editor

Rolling re-activated editor[edit]

A is a user who completed less than edits between and and completed edits (but was not a R:newly registered user) between and .

Reactivated editor (rolling)
Standardized definition
  • = 5 edits
  • = 30 days


Other community metrics
[edit]

The following metrics do not form part of the Editor Model and are computed daily. These metrics will be delivered in stage 3 (2015-Q1)

Daily unique registered editors[edit]

A is a user who is not a flagged bot and completed at least edits on date .

Standardized definition
  • = 1 edits

Daily unique anonymous editors[edit]

A is an unregistered user who completed at least edits on date via the same IP address.

Standardized definition
  • = 1 edits

Daily unique bot editors[edit]

A is a user who is a flagged bot and completed at least edits on date .

Standardized definition
  • = 1 edits

Daily unique page creators[edit]

A is a user who completed at least page creations across all namespaces on date .

Standardized definition
  • = 1 page creation

Daily unique media creators[edit]

A is a user who completed at least media creations on date .

Standardized definition
  • = 1 media creation

Content[edit]

these metrics will be delivered in stage 3 (2015-Q1)

Daily edits[edit]

is a count of the number of edits saved by any users on date .

Standardized definition

no parameters

Daily edits by registered users[edit]

is a count of the number of edits saved by non-bot-flagged registered users on date .

Standardized definition

no parameters

Daily edits by anonymous users[edit]

is a count of the number of edits saved by anonymous editors on date .

Standardized definition

no parameters

Daily edits by bot users[edit]

is a count of the number of edits by flagged bot users on date .

Standardized definition

no parameters

Daily pages created[edit]

is a count of the number of page creations across all namespaces on date .

Standardized definition

no parameters

Daily media created[edit]

is a count of media creations on date .

Standardized definition

no parameters

Curation[edit]

these metrics will be delivered in stage 4 (2015-Q2)

Traffic[edit]

Page views[edit]

See Research:Page view.

Unique devices[edit]

See Research:Unique devices.

Supplementary resources[edit]

Notes[edit]

  1. Yann Algan et al. (2014), "Cooperation in a peer production economy: experimental evidence from Wikipedia."