Research:New page reviewer impact analysis/Incorrect deletion decisions

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search

How has the implementation of the New Page Reviewer right affected the proportion of 'incorrect' review decisions by new page patrollers?

The stated purpose of the implementation of the New Page Reviewer right was to reduce the number of incorrect review actions that resulted in inappropriate articles being marked as reviewed, and appropriate articles being marked for deletion. If the New Page Reviewer right had the desired effect, we should see a decrease in the 'false positive' rate for new articles that are marked for deletion by reviewers: in other words, more of the articles that reviewers mark for deletion should be subsequently deleted by an administrator (other than the reviewer).

Results[edit]

Percent of articles marked for deletion by new page reviewers that were subsequently deleted.

Before the new user right was implemented in mid-November 2016, approximately 83% of articles marked for deletion by patrollers were subsequently deleted by administrators. After November 2016, the 'hit rate' for deletion reviews declined by approximately 10%: only 73% of new articles marked for deletion by patrollers with New Page Patroller user right were subsequently deleted.[1]

This suggests that the new restriction on who could review articles may have had the opposite effect than what was intended. Instead of increasing the consistency and correctness of review actions, it actually decreased it.

This finding has implications for quality control and for reviewer workload.

  • If more of the deletion requests from reviewers are being rejected, it may be that reviewers are being too stringent in their application of deletion criteria, resulting in the deletion of good faith, promising articles (unless that article is subsequently 'saved' by an admin).
  • If more of the deletion requests from reviewers are being rejected, admins and other reviewers may need to re-review more articles, resulting in extra work for them.

Results table[edit]

Year Month article marked for deletion by patroller was deleted? (True/False) # of articles marked for deletion % of articles marked for deletion that were later deleted by an admin
2015 12 FALSE 433
2015 12 TRUE 1544 78.09812848
2016 1 FALSE 225
2016 1 TRUE 1196 84.16608023
2016 2 FALSE 208
2016 2 TRUE 1161 84.80642805
2016 3 FALSE 251
2016 3 TRUE 1487 85.55811277
2016 4 FALSE 276
2016 4 TRUE 1548 84.86842105
2016 5 FALSE 233
2016 5 TRUE 1070 82.1181888
2016 6 FALSE 163
2016 6 TRUE 783 82.76955603
2016 7 FALSE 169
2016 7 TRUE 758 81.76914779
2016 8 FALSE 274
2016 8 TRUE 1259 82.12654925
2016 9 FALSE 231
2016 9 TRUE 1166 83.46456693
2016 10 FALSE 228
2016 10 TRUE 1100 82.8313253
2016 11 FALSE 290
2016 11 TRUE 831 74.13024086
2016 12 FALSE 442
2016 12 TRUE 924 67.64275256
2017 1 FALSE 300
2017 1 TRUE 958 76.15262321
2017 2 FALSE 342
2017 2 TRUE 1071 75.79617834
2017 3 FALSE 320
2017 3 TRUE 819 71.90517998
2017 4 FALSE 244
2017 4 TRUE 751 75.47738693

References[edit]