Research:Guided article creation in Visual Editor

A primary motivation for newcomers to join Wikipedia is to create articles,^[1] yet historically this has been a complex and challenging process. Prior research has shown that newcomers’ often struggle with policy comprehension^[2] and unfamiliarity with the mechanics of editing, resulting in deletion of potentially valid articles. Providing editors with structure and guidance may improve editor retention, increase the likelihood of continued contributions,^[3] and address high deletion rates observed in certain wikis. From previous research supporting the Article Guidance initiative,^[4] we found that the proposed ideas explored in the early-stage design concepts showed promise in providing useful article creation guidance to new editors.

As a next step to support the Article Guidance Initiative, WMF product teams plan to integrate selected features from the initial intervention into Visual Editor. With this integration, it is important to understand whether this guidance is still useful when adapted/integrated with Visual Editor. We also need to assess if this guidance fits seamlessly within, and supports, the Visual Editor experience. Additionally, reviewers often monitor contributions made by new editors, mobile editors, and those who use assistive technologies such as ‘add a citation’ that they track with the help of edit tags. As a result, it is important to get reviewers’ perspectives on the proposed guidance and how it might affect their workflows and workloads.

Research goals

Assess to what degree the design proposal remains useful as guidance to junior editors when concepts are adapted/integrated with Visual Editor
Understand to what degree this guidance fits seamlessly within and supports the Visual Editor experience
Learn reviewers’ perspectives on any guidance that might impact their workflows and workload

Approach and project phases

Methodology

In order to test these prototypes, we will conduct moderated interviews with selected participants to understand their current usage of the Visual Editor, their impressions and perception of the concept, and the usefulness and effectiveness of the prototype in providing guidance. We will use the think-aloud approach to give participants the opportunity to explore the prototype along with some structured probing to get detailed feedback on the features and uncover any challenges in using these prototypes as well as unmet needs. We will also dedicate an initial portion of the interview to understand the participant familiarity with the Visual Editor features relevant to a guided experience to provide context to the later prototype exploration. Analysis will make use of a mix of inductive and deductive coding where some codes and themes are pre-determined based on the research questions but there is scope to derive themes directly from the data.

Phases

[completed] Scoping: Finalize research brief and plan
[completed] Preparation: Materials development, translation, secure support resources, recruitment
[completed] Data collection
[completed] Analysis and reporting

Privacy policy

All research sessions will be accompanied by a privacy statement, detailing what data is collected and in what ways the data will be used. Any research participant quotes or close paraphrases will be anonymized by default. Participants who have been contacted for this study may, at any point, choose to opt-out of the study. For research participant recruitment, we also use a short survey to gather a limited amount of information needed to know if individuals qualify to participate. Similarly, this survey is accompanied by a privacy statement, detailing what data is collected and in what ways the data will be used.

Results

Full detailed reporting

Full results are available here, including recommendations: https://commons.wikimedia.org/wiki/File:(Final_Report)_Guided_Article_Creation_in_Visual_Editor.pdf

Executive summary of results

Guided workflows were viewed as significantly more effective, helping editors to create articles while also providing support and enabling learning of wiki policies. Features such as EditCheck were valuable as they present wiki policies in a user-friendly way, something that both editors and reviewers preferred since policy pages can be verbose and easily ignored.
From among the new guidance concepts introduced, ‘Outlines’ was successful as a feature, providing structured guidance to article creation. Editors understood the concept and were able to use it with ease. (please reference full report for details and images of various proposed features such as 'outlines')
Other concepts in the prototype showed promise but need some level of iteration. These were introducing the menu via ‘+’ icon in the side rail, and adding verified facts to support the content.
The features that need further consideration are the ‘Complete section’ alert and introducing ‘References’ through the menu. Editors were unaware of the ‘Complete section’ alert, nor did they understand they needed to resolve it to continue editing. Editors didn’t explore ‘References’ through the menu as it’s not surfaced during the guidance.
The existing Visual Editor features like ‘EditCheck’ and ‘Add a citation’ worked successfully - even when explored in the combined setting of the prototype. Editors could discover them, understand the concept, and use them as needed. However, with ‘Add a citation’, editors used it in a limited capacity since they were unaware of some functionalities, including ‘Discover’ that was introduced in this prototype.
Editors and reviewers were comfortable with using content from sister projects, such as Wikidata and Wikimedia Commons, as they believed it's verified and aligned with wiki policies. Additionally, using these sites for source content addressed the editors’ challenge of finding and adding reliable sources. However, a concern of reviewers was that a lack of associated Wikidata might give editors the incorrect impression of the topic lacking notability.
Visual cues for discovering the menu and onboarding to the guided experience will help editors discover and interact with the features, reduce ambiguity and help users understand how to use the guidance effectively.
A challenge that editors face with article creation that remains to be addressed is adding images through mobile Visual Editor. Incorporating the addition of visuals via the guidance would be helpful to those who are new to editing and unfamiliar with wikitext.

References / notes

↑ Data from the 2022 Welcome Survey show that approximately 28% of newcomers create accounts with the goal of creating new Wikipedia articles.
↑ For example, during the Revise tone study, only 1 of 5 participants could identify "peacock terms" without guidance, demonstrating that abstract policies need concrete, in-context examples.
↑ For example, the 'Add a Link' structured task increased the first edit probability and retention by roughly 16%. The 'Add a Link' structured task also resulted in contributions with a 9.7% revert rate vs. 34.9% for unstructured tasks.
↑ https://meta.wikimedia.org/wiki/Research:Guided_article_and_section_creation

[1] Data from the 2022 Welcome Survey show that approximately 28% of newcomers create accounts with the goal of creating new Wikipedia articles.

[2] For example, during the Revise tone study, only 1 of 5 participants could identify "peacock terms" without guidance, demonstrating that abstract policies need concrete, in-context examples.

[3] For example, the 'Add a Link' structured task increased the first edit probability and retention by roughly 16%. The 'Add a Link' structured task also resulted in contributions with a 9.7% revert rate vs. 34.9% for unstructured tasks.

[4] ttps://meta.wikimedia.org/wiki/Research:Guided_article_and_section_creation

[1]

[2]

[3]

[4]

Background and context