Wikimedia Futures Lab/Dashboard/Experiment Tracker/Twinkle AI: AI-supported feedback to newcomers

Title

Twinkle AI: AI-supported feedback to newcomers

Hypothesis

I believe that admins and patrollers are overwhelmed and end up biting newcomers. By having an AI tool that can generate a nice, instant contextualized and personalized message explaining their actions, like deletions and rollbacks, and post it to talk pages, we would expect newcomers retention to increase and the quality of edits to improve.

Experiment owners

• ━ Albertoleoncio Who, me?
• Laurentius (talk)

Anticipated completion date

TBD

Description

Problem: Often times administrators and patrollers take actions, like deleting pages, reverting edits, etc., but don't give an explanations to the users who made the edit, or do so in a harsh way. Part of the reason is that they have a lot of work to do and feel overwhelmed, and writing personalized messages is burdensome. This is the kind of problem that LLM can solve.

For instance, imagine that, when deleting an article, an administrator is presented with the following form:

• a field where you can write a short motivation of the action: like the current edit summary field
• a checkbox: do you want to generate a talk message?
• an optional prompt customization
• review and post buttons

If the admin checks the box, a message is automatically generated, and he can review and post it.

Expected outcomes:

• increased editor retention
• lower amount of violations
• reduced burden for administrators

Success metrics:

• retention
• lower number of repeated mistakes (including doing the exactly the same thing over and over)
• how many messages receive a thank you
• we could ask the receiving users whether it's helpful

To run this experiment, we should focus on one Wikipedia edition (likely a medium-large sized one), and engage some admins/patrollers in the experiment. A limited number of people is enough to test and validate the idea.

Additional discussion notes

Notes:

• We already have tools and templates to send messages to talk pages, but they are not personalized and they feel like a parking ticket
• do admins and patrols want to be nice? Maybe not all of them - but probably at least half of them do.
  • Sometimes they want to be harsh for good reasons (when talking with trolls). That's also ok.
  • But sometimes even people who started as trolls can become good editors!
  • Maybe it's not about being nice: it's assuming good faith.
• an LLM could be able to use more context - like previous pages
• The message could be tailored to use the local dialect of the sending user (e.g., portugal or brazilian portuguese)
• Will the community be against the use of AI for messages?
  • In this case it's transparent, and optional
  • It should be clear to the sending user that this is just a suggestion - the user is responsible for it and can change it
  • We could mark the messages with hashtags
• Which information should be feed to the AI? (integrated in the prompt)
  • Generic context
    • Policies
    • Articles for deletion discussions and consensus
  • Specific context:
    • Page history and content
    • Deletion reason
    • Recipient talk page
    • Recipient context (how many edits they made, etc.)

How can this experiment fails?

• Admins and patrols may not be interested in being nice.
• People don't use it.
• Admins gets overwhelmed by the amount of conversations that are opened
• The tool may be too slow

Open questions:

• do we have an LLM in Wikimedia Cloud that we can use?

People present in the initial discussion:

• Martin Urbanec

• Herschal Jackson

• Dennis

• Patricio Lorente

• Joao

• Alberto Leoncio

• Lorenzo Losa