Objective Revision Evaluation Service

From Meta, a Wikimedia project coordination wiki
(Redirected from ORES)
Jump to: navigation, search
Objective Revision Evaluation Service logo.svg
The ORES Logo. 

The Objective Revision Evaluation Service (ORES) is a web service that provides machine learning as a service for Wikimedia Projects. The system is designed to help automate critical wiki-work -- for example, vandalism detection and removal. This service is developed as part of the revision scoring as a service project.

By keeping contribution open, but being good at quality control, open knowledge projects maximize productivity and quality -- and this works for large wikis that are well supported by quality control tools (e.g., English and German Wikipedia), but remain a burden for small wikis. ORES is intended to provide a generalized service to support quality control and curation work in all wikis.

Models[edit]

The primary way that a user of ORES will interact with the system is by asking ORES to apply a 'scorer model' to a particular revision.

Support table[edit]

This table provides a summary overview of which models are supported in which wikis. The sections below discuss these models in more detail.

context edit quality article quality
damaging goodfaith reverted wp10
arwiki Arabic Wikipedia Yes check.svg
cswiki Czech Wikipedia Yes check.svg Yes check.svg Yes check.svg
dewiki German Wikipedia Yes check.svg
enwiki English Wikipedia Yes check.svg Yes check.svg Yes check.svg Yes check.svg
enwiktionary English Wiktionary Yes check.svg
eswiki Spanish Wikipedia Yes check.svg
eswikibooks Spanish Wikibooks Yes check.svg
etwiki Estonian Wikipedia Yes check.svg Yes check.svg Yes check.svg
fawiki Persian Wikipedia Yes check.svg Yes check.svg Yes check.svg
fiwiki Finnish Wikipedia Yes check.svg
frwiki French Wikipedia Yes check.svg Yes check.svg
hewiki Hebrew Wikipedia Yes check.svg Yes check.svg Yes check.svg
huwiki Hungarian Wikipedia Yes check.svg
idwiki Indonesian Wikipedia Yes check.svg
itwiki Italian Wikipedia Yes check.svg
kowiki Korean Wikipedia Yes check.svg
nlwiki Dutch Wikipedia Yes check.svg Yes check.svg Yes check.svg
nowiki Norwegian Wikipedia Yes check.svg
plwiki Polish Wikipedia Yes check.svg Yes check.svg Yes check.svg
ptwiki Portuguese Wikipedia Yes check.svg Yes check.svg Yes check.svg
rowiki Romanian Wikipedia Yes check.svg
ruwiki Russian Wikipedia Yes check.svg Yes check.svg Yes check.svg Yes check.svg
svwiki Swedish Wikipedia Yes check.svg
trwiki Turkish Wikipedia Yes check.svg Yes check.svg Yes check.svg
ukwiki Ukrainian Wikipedia Yes check.svg
viwiki Vietnamese Wikipedia Yes check.svg
wikidatawiki Wikidata Yes check.svg Yes check.svg Yes check.svg
 Add your wiki

Edit quality models[edit]

A descriptive diagram of edits flowing from "The Internet" to Wikipedia depicts the "unknown" quality of edits before ORES and the "good", "needs review", "damaging" labeling that is possible after ORES is made available.
ORES edit quality flow. A descriptive diagram of edits flowing from "The Internet" to Wikipedia depicts the "unknown" quality of edits before ORES and the "good", "needs review", "damaging" labeling that is possible after ORES is made available.

One of the most critical concerns about Wikimedia's open projects is the review of potentially damaging contributions. There's also the need to identify good-faith contributors (who may be inadvertently causing damage) and offering them support. These models intended to make the work of filtering through the recentchanges feed easier. #ORES edit quality flow shows how the stream of edits can be labeled as "good", "needs review" and "damaging" by the machine learning models.

  • damaging -- predicts whether or not an edit causes damage. The higher the "true" probability, the more likely the edit was damaging.
  • goodfaith -- predicts whether an edit was saved in good-faith. The higher the "true" probability, the more likely the edit was saved in good faith.
  • reverted -- predicts whether an edit will eventually be reverted. The higher the "true" probability, the more likely the edit will be reverted.


Article quality models[edit]

A screenshot of the English Wikipedia assessment table generated by WP 1.0 bot is presented. This screenshot was taken on Dec 23rd, 2014.
English Wikipedia assessment table. A screenshot of the English Wikipedia assessment table generated by WP 1.0 bot is presented. This screenshot was taken on Dec 23rd, 2014.

The quality of encyclopedia articles is a core concern for Wikipedians. Currently, some of the large Wikipedias roughly follow the Wikipedia 1.0 assessment rating scale when evaluating the quality of articles. Having these assessments is very useful because it helps us gauge our progress and identify missed opportunities (e.g., popular articles that are low quality). However, keeping these assessments up to date is challenging, so coverage is inconsistent. This is where the wp10 machine learning model comes in handy. By training a model to replicate the article quality assessments that humans perform, we can automatically assess every article and every revision with a computer. This model has been used to help WikiProjects triage re-assessment work and to explore the editing dynamics that lead to article quality improvements.

  • wp10 -- predicts the Wikipedia 1.0 assessment class of an article or draft


API service[edit]

See https://ores.wikimedia.org for information on how to use the API service.

If you're querying the service about a large number of revisions, it's recommended to batch 50 revisions in each request as described below. It's acceptable to use up to four parallel requests.

Web interface[edit]

There is a basic web interface for ORES at https://ores.wikimedia.org/ui.

Licensing[edit]

ORES, revscoring and related software that we develop is freely available under an MIT license. All scores produced by the service are licensed to the public domain (CC0.)

False positives[edit]

False positives can be reported at Research:Revision scoring as a service/Misclassifications/Edit quality.

See also[edit]