Grants:IdeaLab/Measure replacement rate among the admins/Notes

From Meta, a Wikimedia project coordination wiki
An early mockup of a special page for health metrics.

It is possible to graph the metrics, and thus it could be an option to graph it at Wikimedia Statistics, but interpretation of a graph is difficult. It is easier to explain the numbers on a special page, although not much easier. It seems also like the community is more apt to accept pages generated on the wiki rather than pages on some external service.

The mockup should give an idea, but it does not have to look like this, and it does not have to be a special page.

Considerations for where to place the metric[edit]

Where to attach the health metric "Admin Replacement Rate" (ARR) in Wikistats 2.0

alignment
  • ARR is about users, and so would align with editors
  • ARR is about groups, and so would not align with editors
normalization
  • Editors has no really clean normalization except against size of internet community within a language
  • Health metric for an admin group (migration) has an internal normalization against the admin group and an external normalization against the larger group being managed
split
  • Editors can be split in user types, where the admin group can be one of several
  • Health metric for an admin group can be split in immigrants and emigrants, and an overall value

It seems like there are more reasons for splitting it out in a separate metric than trying to adapt the editors metric.

How to define the metric[edit]

This is the core idea. There should be a separate category "Health" (why?) with a metric "Admin replacement rate".

There should be a normalize pane with entries "admins" and "users", both limited to active admins and active users. Not quite sure if this should be limited to active users, but this is closer to the real group size. It will give higher numbers if there are inactive admins.

If no choice are provided for normalizing factors, then #Minimal replacement migration can be used as an argument for only using the internal change rate as a metric. This gives users leaving and joining the group as or the change in the total population, and then it is possible to calculate or the necessary replacement migration to avoid a decline in administrators.

There should be a split pane entry "Migration", with numbers for immigration, emigration, and migration. The entry "Overall total" uses number for total migration with normalization.

If no choice are provided for normalizing factors, then an additional entry can be provided for "minimal replacement migration".

Implementation[edit]

There are several possible ways to implement this, but given the existence of Wikistats 2 it would be better to reuse this. The user part of the existing system is described at Analytics/Systems/Wikistats 2 and the server part of the existing system at Analytics/AQS/Wikistats 2. Those two subsystems will be extended with a small additional user interface for visualizing the collected data, and one or more new endpoints for the server part.

The proposed metrics does not use any private data, and it should not be possible to extract or identify any particular user under normal use. As a border case it is although possible to measure a single admin, if the community has a single admin over a long enough period. This could be solved by dismissing data from periods with fewer than a set limit of admins. Census bureaus typically use a three or five persons as limits. If the limit is reached, then a longer period might have enough data.

User interface[edit]

In the user interface there will be an "admins" section similar to "editors", with a "totals" view. (example) This would be close to identical to editors, but the statistics would only be the number of admins. In the "Filter and split" section the previous mentioned statistics will be listed, possible with options to turn on and off additional graphs. Only a single graph for each each of the implemented statistics is part of this proposal.

Metrics to be computed

There will be no additional dashboard as part of this project.

* this point may be dropped if the project run out of time

Server API[edit]

In the server API there will be an endpoint for each of the necessary statistics for the aggregates, that is the health metrics mentioned above.

Statistics to be computed

  • the Number of active admins
  • the net Migration rate for admins
  • the Replacement rate for admins*
  • the Conversion rate into admins*
  • the minimal replacement migration for admins*
  • the Number of active users (or really non-admin users)*
  • the Number of promoted admins (or really users emigrating from non-admins and immigrating to admins)*
  • the Number of demoted admins (or really users emigrating from admins and immigrating to non-admins)*

* this point may be dropped if the project run out of time

The statistics provided at the endpoints should align with the metrics visualized in the user interface.

To implement this an endpoint like the following is necessary

get /metrics/admins/aggregate/{project}/{admin-type}/{activity-level}/{granularity}/{start}/{end}

where

  • admin-type is one of "total", "active", "emigrating", "immigrating", or "migrating"

or (most likely)

  • admin-type is one of "total", "net-migration-rate", "replacement-rate", "conversion-rate", or "minimal-replacement-migration"

Health metrics[edit]

These are some of the possible health metrics. I have only listed those that can be solved by slight variations over the same code.

Totals[edit]

Standard metric to mimic the other metrics, in particular the "editor" metric. It will only be a single metric, consisting of statistics provided by a single server endpoint.

is the Number of active admins
Net migration rate[edit]

The first one is the net migration rate into the active admin group. The trend for this number says whether the admin group as such are stable. This metric is the only one where it could be argued that it should be normalized against the total number of admins.

where

is the net Migration rate for admins
is the Number of promoted admins (users emigrating from non-admins and immigrating to admins)
is the Number of demoted admins (users emigrating from admins and immigrating to non-admins)
is the Number of active admins
is an index for the bin at the given time
The number is unitless.
Replacement rate[edit]

The second one is the replacement rate into the active admin group. The trend for this number says whether new admins are stable compared to the group of active admins. It can also be described as internal change rate.

where

is the Replacement rate for admins
is the Number of promoted admins (users emigrating from non-admins and immigrating to admins)
is the Number of active admins
is an index for the bin at the given time
The number is unitless.

The metric for active admins is a nearly identical to the present active users, which is already measured, except it has an additional filter to include only the admin group. The metric number of migrating admins is immigrating admins or newly appointed admins, but a separate metric for emigrating admins or newly demoted admins could be interesting too. It is not obvious how the two metrics should be combined, that is the "total" in a split.

Conversion rate[edit]

The third is the user conversion rate into the active admin group. It is virtually the same formula as replacement rate, yet the actual numbers will be quite different. The trend for this number says whether new admins are stable compared to the group of active users. It can also be described as external change rate.

where

is the Conversion rate for admins
is the Number of promoted admins (users emigrating from non-admins and immigrating to admins)
is the Number of active users (or really non-admin users)
is an index for the bin at the given time
The number is unitless.

The metric for active users is a nearly identical to the present active users, which is already measured, except it has an additional filtering to exclude only the admin group. The metric conversion rate can also be called replacement rate, turnover rate, or transition rate. A conversion rate is between specific population groups, while the migration rate is into or out of a specific group.

Minimal replacement migration[edit]

The fourth is the necessary migration into the admin group, given the number of active admins. minimal replacement migration. The trend for this number says whether the community are able to cope with changes in the admin community.

where

is the Number of users to promote to admins to avoid a decline in the group
is the retention rate of admin immigrants defined as; (1 - instantaneous departure rate)
is the difference in the admin population over the time interval;
is an index for the bin at the given time
The number has "admins" as units.

The metric for minimal replacement migration use nearly the same statistics as replacement rate, except the number of demotes must also be calculated.

Alternate definitions for admin health metric[edit]

It seems like this is several slightly different metrics

Replacement rates[edit]

The number of real changes in the group normalized over the total number of members in the group (internal change rate)
This is number the number of new members joining the group plus the number of old members leaving the group normalized over two times the total number of members in the group. Those numbers are straight out of the user rights log given the members of the group. This is perhaps the easiest metric.
Whether previous members should be counted as new members can be debated, but in my opinion they should not. They will not create a real influx of new members, they are only re-appointments of the same members. By counting over a long enough time period re-appointments can easily be canceled out. Still note that the time period must not be longer than the fixed term used at some of the projects.
This metric is probably the one most users would use as a metric for a stale group.
Note that internal change rate is easier to calculate and interpret than replacement rate. The later can only be calculated for incoming new members, as old members leaving the group is nearly impossible to calculate. When has a member truly left the group?
The number of real changes in the group normalized over the total number of active users (external change rate)
This is the number of new members joining the group plus the number of old members leaving the group normalized over two times the total number of active users or the log of the active users.
It is not clear whether the internal or external change rate should be used. The internal one is correct whether the project is oligarchic or not, while the external one should be normalized over the log of the active users if the project is oligarchic. (Ie to be able to predict correct numbers, although if the project is oligarchic it has serious problems.)
Note that this isn't a probability.

Minimal replacement migration[edit]

The rate for minimal replacement migration is given by

where

  • = Replacement Migration avoiding the decline of population in year
  • = retention rate of immigrants year , defined by (1 - instantaneous departure rate)
  • = change in the total population in the time interval

This is the minimum to sustain a population over time. The retention rate at time t is compared with a known difference at a future time t+1 and gives the necessary replacement migration to avoid a decline in the population.

It is comparable to the problem given as internal change rate above, but a group does not have changes in the population except for immigration into the group and emigration out of the group. That is immigration and emigration must balance each other over time. It become slightly more complex when modelling retention of admins as users leaving straight out of the group. Eg. a group member "dies on service", but note dies as in leave the project.

Mean time[edit]

The mean time the members stay in the group (mean member time)
This the accumulated time for all members in the group, divided by the total number of members in the group, given the current active members. This metric comes from the user rights log.
This is a variant of the change rate as a low number for that will give a long time for mean member time. At some projects this number can approach the lifetime of the project.

Promote[edit]

The probability of one new group member in a lower group being promoted to an higher group (promote probability)
For most groups this is simply an extract from the user rights log. Unfortunately it is not possible to do this for the autoconfirmed group, so an alternate method to extract the new group members must be created. Also note that the influx to the higher group must be truly new members, and not just re-appointment of previous members (admins).
Usually this number is very low on Wikipedia, and because it is so low users have stopped expecting to become an admin. This leads to a higher status of being an admin, and people are then less likely to accept being wrong. Ie the social fall has become to large.
The mean time before promotion from one group to another group (mean time before promotion)
This number is simply a variant of promote probability but it is a lot easier to understand.
Because the promote probability is so extremely small the mean time before promotion grows so large it is nearly impossible for an ordinary user to be promoted during his or her lifetime on Wikipedia.
This number can be adjusted for the conditional probability that the mean time is for those that is in fact promoted. The number is still very high.

Activity ratio[edit]

The normalized activity level for a group against the total activity in recent changes (activity level)
The activity level of a group can be measured and normalized against the overall activity level in the recent changes feed, thereby giving a measure that say whether there are enough members in the group. That is the activity in the recent changes feed is used as a proxy for the groups overall activity. It does not say anything about the group actually doing their job, but it does say whether the group is large enough compared to some limit.
This number can be calculated from activity in the recent changes feed, with some additional lookup.

Response time[edit]

There is a bunch of actions that various groups should take, and some of these has (or should have) a maximum response time. Such actions are quite heterogeneous, but a lot of them are now commonly marked with some kind of Done template. Playing on this it could be possible to detect such markers on a page and calculate response times.

One possibility could be to add a tag on the revision if a special parser variable is incremented, that is a new call is added, and then log the time for the first occurrence of a comment inside a summary and the time for a done-tag on a revision with the same comment. A list of such pairs could be used for creating statistics about the response times on the specific page.

Statistics for a page would include minimum, maximum, mean, and median times. They could be made available as individual values, or as a block of stats. The later could be added automatically if any parser variable is added.

Note that it is not necessary to have a complete picture of response time for all actions to be able to give an estimate for response time, as we can use a proxy into some of the actions.

Alternate definitions for user type health metric[edit]

Oligarchiness[edit]

A few metrics from the paper of Shaw and Hill (that is the hypothesis’ about oligarchiness reformulated)

The number of new group members (aka administrators) over number of active contributors
Hypothesis 1; "The probability of adding new administrators declines as wikis’ contributor bases grow". The number should probably be normalized over active contributors and not number of registered users.
The number of contributions to project pages by group members (aka admins) over total number of contributions to project pages
Hypothesis 2; "Controlling for the total number of contributions to administrative pages, administrators will contribute more to administrative pages as wikis’ contributor bases grow".
The number of reverts by administrators of experienced contributors over the total number of contributions made by experienced contributors
Hypothesis 3; "Controlling for the total number of contributions made by experienced contributors, the number of reverts by administrators of such contributions will increase as wikis’ contributor bases grow".
These three indicators can be used as trends, comparing over an interval.

Egalitarian contributor[edit]

An egalitarian contributor can be modeled as a contributor with nearly linear behavior in H3, while an oligarchian contributor must be modeled as a contributor with non-linear behavior. Any given contributor will be a mixture of the two models, and the optimum mixture to describe the contributor is an optimization problem.

The reasoning goes like "An egalitarian contributor will be equally influenced by any other contributor as he/she is focused on his/her own contributions, thus an increase in other contributions will scale linearly given the users own normalized contributions." Contrary to this "An oligarchian contributor will be increasingly disturbed by other contributors as he/she is focused on his/her own goals, thus an increase in other contributions will scale non-linearly given the users own normalized contributions."

The non-linearness in H3 will probably be a pretty good metric for oligarchiness for a group of users. For a specific user the metric will be quite noisy. More data, that is sampling over longer time, will usually give less noise.

There will be an increase in H2, but I wonder if that is more a result of necessity than an expression of oligarchiness. Individual users will in any case have a wildly different metric for H2. For example will an user active on the deletion pages be flagged as oligarchian according to H2, but i'm not sure this is correct.

Meritocrat–oligarch–egalitarian plane[edit]

Over time there should be a part of the community that has clear signs on turning into a w:meritocracy. Those users will show up as ordinary users in group memberships, but they tend to be involved in project discussions too. Meritocrat differ from oligarchs by lesser use of revert tools, they contribute more. Meritocrat get their social capital from contributions, while oligarchs seems to rely more on power-structures.

Simply being an normal user with high activity is not a clear indication of being a meritocrat, the activity must involve project work. There seems to be a kind of plane spanned by meritocrat–oligarch–egalitarian

I'm not sure whether a deletionist can be a meritocrat, but it seems so. Perhaps the inclusionist–deletionist is somewhat orthogonal to the meritocrat–oligarch–egalitarian plane, or slightly oriented along the oligarch–egalitarian axis. A deletionist is more oligarch-like.

If the total group of oligarchs gets to large they will easily overwhelm meritocrats and take control over the wiki.

To much of either meritocracy or oligarchy would probably create a defunc community. A community on the meritocrat-side is probably a lesser evil than an overwhelmingly oligarchic community. An extreme egalitarian community would probably turn into chaos, as it will have little coordination.

Alternate definitions for new editors health metric[edit]

Retention of new editors[edit]

There is a pretty simple health metric for new editors; relative retention of promising new editors. Assume some new editors arrive at time t0, and then their activity levels are measured at t1 and t2. If the activity level at t2 is normalized over t1, then the number should scale pretty well with whatever activity level the wiki has. If it drops to much, the retention gets to low, then something weird is going on and the community is to harsh to newcomers.

A better formula could be

This relative retention of promising new editors will probably be somewhat dependent on the number of active users, that is normalizing against the number of active users should be enough

It is likely that there will be some kind of non-linear dependency, so it should probably be modeled as a regression like the oligarchyness.

See also[edit]