Jump to content

Expert review

From Meta, a Wikimedia project coordination wiki

Purpose of this page: Elaborate on a lightweight model which will allow us to receive and store reviews by credentialed experts of specific Wikipedia[1] article revisions, and to surface such reviews through the UI and through the API.

Rationale

[edit]

Assertions by credentialed experts that an article does or does not represent the best available scholarship on a topic can help readers (by making it easier to find high quality content) and editors (by incentivizing high quality work, surfacing problems, etc.), as well as individuals or groups seeking to use subsets of Wikipedia for educational purposes (by allowing extraction and use of reviewed subsets of Wikipedia content). This does not require any changes to Wikipedia's editorial process, and indeed, capturing, storing and making available such reviews does not imply any additional valuation of the credentialed expertise or even its relevance to the article at hand.

Precedents

[edit]
Magnus Manske's experimental JavaScript widget to display article reviews
Phase II design for the reader feedback tool (does not support expert reviews yet)

Actual practice:

Proposals and research:

What you can do right now

[edit]

If you're a hacker:

  • Make Magnus Manske's user script better and help develop it to gadget quality (documentation and known issues)
  • Create other proof-of-concept applications that tie into the existing toolserver database / API developed by Magnus
    • Proof-of-concept application for allowing authenticated users to submit reviews
  • Help extend the ArticleFeedback MediaWiki extension to support product requirements articulated below

If you're good at analyzing and documenting:

  • Help us develop draft product specifications for how expert reviews should be supported by MediaWiki (UI and workflow mock-ups very much welcome)

If you're great at outreach into academia:

  • Help identify opportunities for mobilizing large numbers of experts to join the review process

User research

[edit]

Wikipedians

[edit]
This section is mainly based on a review of the existing essays and views about expert editors and credentials on the English Wikipedia[2] and the work of Logan et al.[3].

The commonly established consensus among Wikipedians is that "experts" are welcome to edit articles, but should do so just like any other editor. They shouldn't get any preferential treatment because of claimed credentials. They must provide reliable sources for their changes, and can't use their authority as an "expert" to try to win an argument. Like any other Wikipedian, they have to follow the same policies, and must be particularly careful about writing from a neutral point of view, and avoiding self-promotion, conflicts of interest, and original research.

This assessment leads to the conclusion that it is likely to be difficult to recruit editors from targeted "expert" circles like academics and professionals. It is likely that, just like any other population, only a fraction of the "expert" population will demonstrate the qualities and motivations required to be a Wikipedian.

If "experts" are to be specifically involved in the improvement of Wikimedia content, it needs to be through other venues and processes, like independent, externally-hosted reviews.

"Experts"

[edit]
This section is mainly based on preliminary results from the Research Committee survey on expert involvement.

The main barrier identified by "experts" is the lack of professional benefit from contributing to Wikipedia. It doesn't provide credit, nor does it count for tenure, or advances one's career, so they feel they don't have time for it. Because there is no professional incentive to contribute to Wikipedia, it is not a priority in the busy schedule of academics and professionals.

The other main barrier is social: academics are generally used to be treated with deference. The interaction with a community of egalitarian Wikipedians can rapidly be disconcerting, even frustrating, when their edits are disputed, edited, or removed. "Experts" are also reluctant to engaging in long, difficult, and sometimes hostile discussions with "amateurs" who don't accept the argument of authority.

Product requirements

[edit]

Very basic notes.

  • Should surface expert review information in an unobtrusive manner, ideally integrated with other review information such as reader feedback, FlaggedRevs, etc.
    • Recognizing reviewer organizations with logos and links would be a nice incentive for organizations to join the program
  • Should allow for short and extended comments
  • Should enable a simple yes/no decision on whether an article should e.g. be included in an offline distribution
  • Should recognize that multiple fields of expertise may apply to any given article
  • Should support e-mail as a delivery mechanism for new articles requiring review (to simulate peer review process)
  • Should capture conflicts of interest by reviewers
  • Should initially require only minimal effort towards credentials validation, likely by offloading this requirement to other organizations
    • e.g. in the current Encyclopedia of Life prototype, EOL has its own criteria for curator expertise
  • Should have one or multiple authentication mechanisms
    • based on rights-assignment to individual users, possibly granted at sign-up through separately authenticated sign-up form?
    • based on URL keys that can be used for e-mails etc.?
    • based on open authentication/authorization standards such as Shibboleth?
  • May need to interface with WikiProjects for purposes such as selecting articles requiring review, outreach, etc.
  • Should not cannibalize editing - reviews should be coupled with an invitation to improve the article directly
    • But: reviewers must be independent and not substantially involved in the article's development. Reviewing a revision you haven't touched and improving it going forward would be less problematic than reviewing a revision you've substantially contributed to.

See also

[edit]

Notes

[edit]
  1. In future, we may extend the model further to cover other projects, especially Wikibooks, Wikiversity
  2. Wikipedian views and essays on credentials & expert participation: Wikipedia:Expert retention, Wikipedia:Credentials are irrelevant, Wikipedia:Credentials matter, Wikipedia:Ignore all credentials, User:Jimbo Wales/Credential Verification, Wikipedia:Expert editors, User:LinaMishima/Experts Problem, User:Jnc/Astronomer vs Amateur, User:Misza13/Nobody cares about your credentials, Wikipedia:There is no credential policy, Wikipedia:Anti-elitism, Wikipedia:Randy in Boise, Wikipedia:Competence is required, Wikipedia:Expert rebellion
  3. Logan DW, Sandal M, Gardner PP, Manske M, Bateman A, (2010). Ten Simple Rules for Editing Wikipedia. PLoS Comput. Biol., 6(9): e1000941. doi:10.1371/journal.pcbi.1000941