Talk:ARTT

Welcome! We would love to hear from you below. Please share your questions, thoughts, and opinions on the forthcoming Analysis and Response for Trust Tool (ARTT) and its potential uses for Wikipedia.

Reliable sources

Question: Can ARTT strengthen a reliable source guide (perennial source guide)?

You are probably already know about w:WP:MEDRS and w:WP:RS. A good starting point in "strengthening" is probably understanding it. You probably also want to look at this paper on the topic . w:WP:DUE is also an often used tool for dealing with "misinformation".

In summary, our findings reveal a trade off between timeliness and scientificness in regards to scientific literature: most of Wikipedia’s COVID-19 content was supported by references from highly trusted sources - but more from the general media than from academic publications. That Wikipedia’s COVID-19 articles were based on respected sources in both the academic and popular media was found to be true even as the pandemic and number of articles about it grew. Our investigation further demonstrates that despite a surge in preprints about the virus and their promise of cutting-edge information, Wikipedia preferred published studies, giving a clear preference to open-access studies. A temporal and network analysis of COVID-19 articles indicated that remaining up-to-date did come at a cost in terms of quality, but also showed how preexisting content helped regulate the flow of new information into existing articles

It's worth noting that for good or ill wikipedia will lag behind primary research and focus on systematic reviews or other information that contextualizes a variety of primary sources.

I suspect you might like to examine some talk pages on issues that you consider relevant to ARTT, and might find useful for some general context about the environment in which your tool plays out (though note that this is out of date). Talpedia (talk) 14:50, 24 July 2022 (UTC)

@Headbomb has a user script that vets sources according to the name of the publisher (i.e., at the level of "MDPI is suspicious, but Elsevier is good"). A further refinement along these lines might be to provide journal-specific information (e.g., "MDPI's Foods is good, but Elsevier's Archives of Cardiovascular Diseases Supplements is not") or article-specific information (e.g., recent reviews are great, and original papers and papers 10+ years old are not).

I do not think that the "perennial source" approach is a good one. I think it is a harmful concept in general: a source's actual reliability cannot be determined without looking at the sentence it is connected to; a source can be good for one subject and bad for another; a source's reputation might change over time, so prior discussions might not be relevant. (It's called "perennial" because it was supposed to be a handy cheat sheet of sources that were frequently debated on wiki – the discussion that wouldn't die. It has since [probably as a result of the 2020 US presidential elections] become a way to pre-emptively block the use of sources that you dislike, even if they were never discussed before, or never even used.)

However, for this specific purpose, the problem is that there are tens of thousands of medical and medical-adjacent journals, and we've talked about less than 1% of them. I can tell you that a journal article in the lowest quintile typically has a larger number of copyright violations than journals in the other four quintiles, but it's very rare that I could tell you anything specific about any individual journals, especially in the lowest quintile.

In terms of vaccine-related subjects, one of the fastest ways to evaluate sources is to consider the type of source. Anything with {{cite web}} – except when it is pointing to certain government websites like the UK's NHS or the US's NIH – is riskier than most things using {{cite journal}} or {{cite book}}.

Also, I want to echo @Talpedia on the DUE problem: The problem often isn't reliability per se. It's usually a problem of w:en:WP:Balancing aspects to put in the things that are relevant and important, and to leave out trivia and ephemeral things. A link to Donald Trump's Twitter account would be a perfectly reliable (even "authoritative") source for a statement like "Donald Trump tweeted about horse de-wormer". The problem is that Trump's views on horse de-wormer don't belong in any article (except possibly w:en:Veracity of statements by Donald Trump), not that the source doesn't support the statement. WhatamIdoing (talk) 16:11, 25 July 2022 (UTC)

"Headbomb has a user script that vets sources according to the name of the publisher", not quite what it does. It checks for domain-level (e.g. lulu.com, arxiv.org, biorxiv.org) or and DOI prefixes. Both publishers and individual journals, as well a general websites, can be targeted (subject to certain technical limitations). If the exact details are important, you can check en:WP:UPSD for the specifics.

And while I'm here, I'll say I'll disagree with It has since [...] become a way to pre-emptively block the use of sources that you dislike, even if they were never discussed before, or never even used.). That's not been my experience with en:WP:RSP. Headbomb (talk) 17:55, 25 July 2022 (UTC)

It happens from time to time. See, e.g., w:en:WP:RSN/Archive_323#State-sponsored_fake_news_sites, which proposes banning seven sites, six of which were "not used" in any article, according to one editor who looked into it. WhatamIdoing (talk) 02:22, 26 July 2022 (UTC)

Article quality

Question: Is it possible to increase the quality of a vaccine-related Wikipedia articles to the point where ARTT may recognize them as reliable sources?

What would it take for an article, itself, to become a reliable source?

Please discuss.

• ARTT could systematically report multiple dimensions of quality based on online Wikimedia information without new human interventions.
  • Are there enough sources? Are there so many that it's a problem? (That means the list may call for pruning, since many/obscure sources are hard for anyone to fully review. An article may be more coherent if we cite a few reliable secondary/survey sources.) Are the sources mainly from peer-reviewed or otherwise reliable sources? ARTT will have its own curated information about which medical sources are reliable. Are there too many citations to dubious sources?
  • Organization of the text, per the en.wp Manual of Style for medical articles; does it have the expected sections and are they of appropriate length?
  • Article evaluations apart from its medical content: presence of warning banners; ORES score; numbers of wikilinks to and from the article; presence of images or Wikidata or Commons or Wikisource materials
  • Something about whether there are enough editors, or too many who are not taking charge and getting it organized?
  • Talk page rating by a real human editor, and the context and date of that rating, and whether changes have been made since then; links to relevant WikiProjects
• Those evaluations can be reported as scores, and they give guidance on people might do to improve the article. -- econterms (talk) 17:34, 22 June 2022 (UTC)

• At the Wikimedia Hack-a-thon (May 21-22), seven attendees completed a survey regarding what they consider when assessing article quality. Their answers showed that they are more likely to consider an article low-quality if it has:
  • few sources
  • dubious sources
  • warning banners
  • one editor and/or too few editors

• Less important markers were:
  • low ORES scores
  • that it was recently created
  • many reverted edits
  • little talk page engagement
  • lack of Wikiprojects tags -Ariel Cetrone (WMDC) (talk) 13:57, 24 June 2022 (UTC)

  @Ariel Cetrone (WMDC), the item about "warning banners" is discouraging to me. First, those are "maintenance banners", and they are not meant to 'warn the reader'.

  Second, many of those are out of date. To give one example, a significant proportion of banners saying that the article needs more footnotes were added a very long time ago, by a bot, when there were no ref tags in the article. (It started as a category, and another bot replaced it with a maintenance banner in December 2009.) Then, some years later, another bot was sent around to see whether ref tags had been added to the article. Thousands of articles had their "unref" template bot-replaced by a "more footnotes" template... no matter how many refs were on the page. These tags are almost never removed, because there's always room for a few more refs, right? And we all assume that a human made a judgment that this particular article needed more refs. Look at the July and December 2009 bot edits in w:en:Preceptor, and then the January 2012 bot edit that changed the tag. This article has multiple unsourced sections, but no human has ever thought it enough of a problem to bother adding any requests for additional sources. WhatamIdoing (talk) 16:25, 25 July 2022 (UTC)

The articles are probably already more reliable than the articles you consider reliable, such as those written by WHO and doctors during the pandemics due to the political bias in these sources, propaganda associated with them, simplifications, norms of misleading patients embedded within medical communication, variety of editors, and variety of sources used.

That said. I do agree with some of Econterms and Ariel's suggestions about about distinguishing articles. From what I can tell some articles have few editors, and you sort of get "hoards" of editors moving between article. If an article us under the eye of "the hoard" it will likely be of better quality, as was the case for many COVID articles. You could likely create a proxy for this by tracking certain editor names in recent edits. I don't approve of this metric - but I suspect it would work. Talpedia (talk) 14:59, 24 July 2022 (UTC)

Talk pages

Question: Can the tool encourage collegial exchanges among editors? For example, if there was a tool that recommends tactics for effective communication with other editors, would you be inclined to use it?

I suspect tools to encourage cordial debate might be useful. You might like to look at as I have cited above. Reviewing what materials you have put online at the moment, I have misgivings. I would far prefer a framing of "codiscovery / cocreation" of knowledge for this sort of thing. It's worth noting that this shared vision of creating an legitimate encyclopedia already exists within wikipedia, and there are various rules such as w:WP:CONSENSUS, w:WP:VERIFY and w:WP:NOFORUM to back this up.

Your framing in terms of "misinformation", "responders" and "interventions" seems to borrow the language and methodology of disease and public health. As a group of correct people, quoting trust worthy authorities to correct misinformation. No doubt, at it's extremes, this framing becomes accurate, but I think it should be avoided. From what I've seen your framing is an ill-fit for most conversation, and this is even more true for wikipedia. It feels like you view yourself as having "the truth" the aim of the process is to manipulate your colocutor into agreeing with you.

Wikipedia has a well-documented problem with "nastiness", problems with attracting new editors and retention; and some of your approaches could no doubt help here. I reviewed https://artt.cs.washington.edu/response/ and don't really disagree with it, and I imagine your to-be-released database of sources from communication science is valuable in this regard. The issue from by perspective is that there appears to be "no subjectivity" or "learning" in how you are approaching the problem.

Talpedia (talk) 15:18, 24 July 2022 (UTC)

I recommend "My Father, The Fool" by Richard Russo in The Atlantic to anyone who is interested in this subject. People don't pass along bad information because they are trying to harm others. They (incorrectly but unknowingly) believe that they – for once – have the inside story, the One True™ Truth, the real deal from truly trustworthy sources ('cause they trusted them, and they wouldn't have trusted them if they weren't trustworthy, right?), and they are genuinely trying to help.

When you have two people, each of whom believe they hold The Truth™, but their "truths" are opposites, then disagreement is inevitable. It's nice to encourage collegial exchanges, but there may be limits to what's possible. We sometimes have success with telling people that Wikipedia doesn't want the truth; it wants what researchers put in journal articles, or it wants the view that a typical physician holds, rather than the unusual views promoted by their special "Lyme literate" source. Some people accept Wikipedia's self-imposed limitations, and others don't.

There is also the problem of compassion fatigue. I might be able to empathize with the first half-dozen people to put bad science in an article, but by the time the seventh person adds the same bit of stupid, I might run out of compassion. As an example of one that I solved, w:en:Multiple chemical sensitivity had a problem for years with English speakers adding a link to a scanned copy of a letter written in German, and claiming that it says something that the letter does not say. Even though none of them (obviously) could read it, they believed they knew what it said because some MCS websites told them that the letter proves recognition of MCS by the government of Germany. I finally solved this problem by using it as an example in the article of how healthcare providers can get paid for providing care to MCS patients. Now readers both get accurate information and an opportunity to let go of the misinformation they had collected. I don't remember having any problems with this since making that change to the article. WhatamIdoing (talk) 16:48, 25 July 2022 (UTC)

[Wikipedia] wants what researchers put in journal articles, or it wants the view that a typical physician holds a concern is that these two things can be very different. What I'm even more concerned about is "what a science communication expert will say to the public (to ensure compliance)" or "what a communication science approved by the WHO will say to the public to ensure compliance". I want wikipedia to be able to distinguish science communication, guidance, journalism, and research sources and find the best sources for particular claims. I'm suspicious that ARTT wants you to stop talking if you disagree with the latest authorative piece of guidance written by a science communication expert, or say something that might undermine the "messaging".

In fact ARTT might have it's own version of "The authorative TRUTH, supported by experts" based on authorative guidance. from a WHO approved list of sources, that is sure to comment positively on the general effectiveness of vaccines.

When wikipedia disagreement is at it's best (at least for me, for personal growth) it encourages a detailed search and reading of the best secondary sources from a range of topics and a nuanced understanding of this information that can even change your view. But I am aware that a thorough reading of the best sources might not be what a lot editors want to do and perhaps there are needs to better approaches when this doesn't happen.

I suppose there's another angle here, that I had not considered, that wikipedia as well as being somewhere where people create articles together could be a venue for allowing the kind of "responses" that ARTT is suggesting in order to encourage better edits or at least less drama. If I were to spin some of the seeming aims of ARTT in the most positive way, its goals might be maximise mutually growth from an interaction while reducing unnecessary and boring noise. I have my doubts about this however, it sort of violates w:WP:FORUM, the personal growth on wikipedia is perhaps better explored through collaborative editing, and there are certainly some things you can get from editing that you cannot get from mere conversation, and I sort of feel the objectification and patronising attitude that comes along with something like ARTT. Is it really better to be told that someone thinks you a conspiracy-wielding, misinformation spreader who can't understand the facts or argument by having them show patronising empathy for something that is not actually your opinion? Or is it better for someone to quote some policies, discussions, sources and principles that will take you hours of reading to understand. Perhaps it's a matter of arrogance and misanthrope on my part but I would prefer the latter, since I suspect I will gain more from the interaction.

I guess my preferred "liberal" solution would be to make it as easy as possible to interact with the literature and wikipedia policies and the basis for them, I'm not sure it would work. Talpedia (talk) 00:05, 31 July 2022 (UTC)

A typical researcher's POV and a typical clinician's POV can differ substantially. I imagine that's true in many fields (e.g., teaching children to read).

@Talpedia, I wonder if you feel sometimes like science communication could easily stray into (well-intentioned) propaganda. WhatamIdoing (talk) 03:19, 1 August 2022 (UTC)

I do indeed. In fact I almost started writing something here entitled "public health propaganda" talking about how an aim of public health at times seem to be to alter people's understandings of facts to make them do things for the public good (not individual interest) regardless of what the evidence is and how I was concerned that this the intent behind the use of the concept of "authoritative"... but then thought it might come across as too "conspiracy theory" sounding. There is a scenario where you try to build a machine to propagate public health "truth" from your designated government sanctioned authority to quickly change people's behaviour. I have no issues with having such an apparatus for guidance, but it starts to become problematic when it's about truth.

I guess my opinion is based on personal interaction with medical literature and the corresponding guidance where I've repeatedly found that that guidance's "simplifications" tend to hide quite important details, often seemingly designed to get people to do what doctors want (or worse what other people think that doctors want). Talpedia (talk) 10:16, 1 August 2022 (UTC)

I suspect that this risk is more obvious to someone in an individualistic culture vs a community-oriented culture.

I'm not sure that it's necessary or even relevant to alter people's understandings of scientific facts. The facts may be little more than decoration. Some of the research suggests that adults (at least in the US) are no more immune to "everybody else is doing it" than teenagers. If you want to change teen smoking rates, you put up a poster at school that says "Most kids in this school don't smoke"; if you want to change adult conservation efforts, you mail them a letter that says "Most of your neighbors are trying to conserve energy". WhatamIdoing (talk) 15:23, 1 August 2022 (UTC)

That seems quite plausible. There can be degree of "the tail-wagging the dog" going on regarding ideology/tribe and facts, and even if one accepts facts it may only be because you have creative ways to fit them into your ideology. That doesn't stop damage as a result of people *trying* to distort and censor facts.

This does seem like a unspoken aim of AART... or perhaps their aim is to make people feel part of a community regarding this issue to obtain compliance that way. Or maybe it's all to do with offering community regarding issues because speech on the internet is indicative of a need for community.

Another aspect seems to be linking preexisting community up to the public health / societal interest community. Talpedia (talk) 18:05, 1 August 2022 (UTC)

Which IMO wouldn't be a bad idea. We would all benefit from public health people having a couple of wiki-savvy friends.

I suspect that AART and other groups are reacting primarily to the COVID-19 vaccine situation. Normally (e.g., DPT vaccines), the vaccines are fairly old and very well studied. The Scientific™ Facts don't change often. The social facts (e.g., whether your child can go to school) don't change often, either. When they do change (e.g., when the DPT vaccine switched from whole-cell to acellular for pertusiss), there's a bit of a kerfuffle, but (a) it usually affects a small group of people, and (b) the risks and alternatives are pretty well known and generally accepted through human-scale/social methods (i.e., "My sister's kids both got this vaccine years ago, and they're fine" or "Do you remember when the neighbor kids got whooping cough in 1967?", not to mention "Yeah, pneumonia admissions always go up this time of year, but this is expected and we're staffed up for it"). As a result, the debate is generally small scale and personal ("I could never forgive myself if I deliberately gave my child a vaccine and he developed any side effects, but if he dies of whooping cough, then that was just 'natural', 'random', and 'not my fault', and besides, it'll probably be somebody else's baby who dies").

With COVID-19, people were trying to make sense of information without their usual supports. People didn't have any settled background knowledge or acceptance. The scientific facts changed as more things were learned plus as the virus mutated. Individual choices had outsized effects on others (e.g., overwhelmed hospitals and overstuffed morgues, because the anti-vax and vaccine-hesitant people not only insisted upon their individual right to refuse the vaccines, but also their right to make other people provide medical care, life insurance payments, etc. whenever the natural consequences of their first choice turned out to be unfavorable).

If the US were more like most of Europe and China in the sense of having a more positive relationship with "official" health information, then the US might have had a different response to the COVID-19 vaccine. But I wonder: do the responses from COVID-19 really apply to normal, everyday decisions about whether to get a typical childhood vaccine or an annual influenza jab? And if we're in a world in which some people believe anything they read in their internet bubble and nothing outside it, why do we think that providing good information outside the bubble will help the people who need the most help? WhatamIdoing (talk) 01:20, 3 August 2022 (UTC)

Training

Question: Would a tool that recommends quality sources be of use to trainers?

I would very much like a tool to link to "quality sources" and if you've managed to make it work would be very keen. If you could make something like DrQA that opperated over academic books and secondary sources, that worked well, and was trained on wikipedia sentences rather than questions that would be wonderful - but is perhaps at the moment impossible. The question then is what is possible.

I suspect the sources you use will be of poor quality and biased given your framing "fighting misinformation". Wikipedia prefers academic sources (preferrably secondary :see w:WP:MEDRS) (w:WP:ABIAS) to those written by medical communication experts for the general public.

Talpedia (talk) 15:31, 24 July 2022 (UTC)

Yes. Also, if you could match a good source (e.g., a recent review article in a decent journal) with relevant items in Citation Hunt, then those sources might actually get used. WhatamIdoing (talk) 16:50, 25 July 2022 (UTC)

Wikipedian workshop, medical expertise:

On June 15, 2022, Wikimedia DC and Hacks/Hackers held a workshop where Wikipedians with medical expertise discussed quality markers for vaccine-related articles. The goal of the session was to determine what it would take for an article to be high-quality enough for the ARTT tool to recommend it as a reliable source.

The general consensus was that high ORES scores and word counts do not indicate quality. Instead, a quality article should be measured by a variety of factors including, but not limited to: the completeness of text, the age and lack of redundancies among its sources, and its readability.

Further discussion is required to determine if it is feasible to raise the bar on vaccine-related articles.

In light of these findings, additional uses for the ARTT are now being considered. For example, if the tool recommends an article as a source, even if it is not truly a high-quality one, could the tool include an educational features to help readers to assess the article? -Ariel Cetrone (WMDC) (talk) 15:15, 24 June 2022 (UTC)

I'm not sure that it's a good idea to declare a Wikipedia article as a high-quality source. Even if we create a great article, someone could remove good information or add bad information immediately after it's endorsed by the ARTT group. I think you will want to talk to User:Mikael Häggström or User:Doc James or other people in the WikiJournal User Group about this problem. WhatamIdoing (talk) 17:17, 25 July 2022 (UTC)

Trust

Prior research on how readers perceive Wikipedia indicates that people don't need to trust an information source to find it useful. I wonder whether the focus on "trust" is misplaced. Maybe we should try to make articles be "useful" instead. On the subject of vaccines, that probably means things like mentioning all the things they care about (e.g., menstrual irregularities). WhatamIdoing (talk) 00:58, 28 July 2022 (UTC)

Parallel projects

Let's keep an eye on View it! Tool and show this project on meta analogously to that one. -- econterms (talk) 17:22, 13 August 2022 (UTC)