Research:Community portal redesign/Opentask

The purpose of this experiment is to measure the effects of modifications to the open task list on the Community portal on English Wikipedia.

Background

The Community portal attracts thousands of visitors each day, the majority of whom are anonymous and new editors. A relatively high percentage of these users click on articles in the open task list, which is refreshed hourly by a bot with articles from various maintenance categories.

While we know how many users are clicking on these articles, we don't have a good sense of how many of them are attempting and/or completing an edit. Qualitative coding of a day's worth of data suggests that few to no edits emerge from these visits. Measuring the number of task attempts and completions in a discrete period of time will give us a good baseline for other task recommendation focused experiments, as well as provide insight for iteratively improving the usability of this and other task recommendation lists.

Extended content
Week 1 Click results Total clicks 1206 Task # of total clicks # of unique clicks # of returned clicks Wikify 49 92% 8% Copyedit 60 82% 18% Remove original research 37 95% 2% De-orphan 183 88% 12% Verify 41 90% 10% Merge 61 90% 10% Split 91 93% 7% Expand 166 90% 10% Make neutral 36 94% 6% Update 78 92% 8% Discuss AfD 178 93% 7% Clean up 55 87% 13% Add image 7 100% 0% Translate 104 95% 5% Create 60 88% 12% Total 1206 Week 2 Click results Total clicks 2125 (50% increase from week 1) Task # of total clicks # of unique clicks # of returned clicks Wikify 435 90% 10% Copyedit 345 85% 15% Remove original research 186 86% 14% De-orphan 148 89% 11% Verify 101 92% 8% Merge 74 89% 11% Split 80 90% 10% Expand 94 93% 7% Make neutral 101 90% 10% Update 86 95% 5% Discuss AfD 72 90% 10% Clean up 128 96% 4% Add image 71 99% 1% Translate 92 92% 8% Create 183 85% 15% Total 2191 Summary Increasing the whitespace and visibility of the list and removing excess links and formatting doubled the number of clicks on open tasks. Week 3 Click results Total clicks Task # of total clicks # of unique clicks # of returned clicks Wikify 156 94% 6% Copyedit 187 82% 18% Remove original research 75 91% 9% De-orphan 79 90% 10% Verify 78 87% 13% Merge 53 92% 8% Split 64 97% 3% Expand 80 84% 16% Make neutral 110 90% 10% Update 94 93% 7% Discuss AfD 61 93% 7% Clean up 100 97% 3% Add image 90 90 100% Translate 79 86% 14% Create 82 89% 11% Total 1388 Summary Moving the Collaborations section to the top of the page reduced clicks on opentask articles, though there were still more clicks generated than in the original page layout.

Measuring the funnel

Funnel of task acceptance, attempt, and completion

Funnels are ways of measuring user conversion through multiple steps and tasks. We're interested in how many unique users who view the landing page go on to view and attempt one of the tasks. Additionally, we're interested in seeing what kinds of tasks most users are attempting, and whether they successfully save their edits to those task articles.

Event	Definition	What's tracked	Properties
Opentask click	User clicks on any article in the open task section of the Community portal	anonymized token, timestamp, click target	task type (e.g., "copyedit," "npov", etc.)
Edit click	User clicks on the edit button of the task page	anonymized token, timestamp, click target	task type
Edit completion	User closes the editing interface either through a successful save or cancel action	anonymized token, timestamp, click target	save, task type
CP return	User returns to the CP after passing through any stage of the funnel	anonymized token, timestamp, click referrer	none

Results

Pre-redesign (November 3-14)

The above chart shows unique users converted along each of the following steps: 1) clicking any link on the Community portal, 2) clicking on an opentask link (accepting a task), 3) clicking the edit button on the article, 4) saving an edit to the article. Except for the first column, each column is the percentage of users converted from the previous step (the number of users who completed that step divided by the number of users who completed the step to the left).*

*Absolute numbers: 3239 users accepted a task, 136 clicked edit, 27 saved edit.

It appears that the major drop-off point in the funnel is between accepting a task (clicking on one of the articles in the opentask list) and finding/clicking the edit button. This suggests that users are interested in trying out a task, but either don't find the edit button or can't figure out how they can improve the article.

After the redesign (November 15-26)

*Absolute numbers: 1193 users accepted a task, 56 attempted an edit, and 19 saved. This represents a small increase (from 4% to 5%) for edit attempts, and a larger increase (from 0.1% to 2%) of successful saves per click-through to task.

Conversion by task

The above image shows absolute numbers of unique users selecting, attempting, and completing the different kinds of tasks in the opentask list. Copyediting and wikifying appear to be the two most successful task types, measured by both numbers of users selecting articles in those task types and successfully editing the articles.

Clicks and conversions

The following is a heatmap of clicks on the different sections of the opentask list, and a table of unique users who passed through different stages of the funnel and the percentages that completed each stage (except task, edit article, save edit) in a 10-day period. The top two task types, wikifying and copyediting, received a fourth of all the clicks on the list. However, only copyediting was able to convert a significant number of users.

Heatmap of clicks

Complete table of unique users in the funnel, by task type

Task type	# of unique users who accepted at least once	# of users who clicked on the edit button	% converted	# of users who saved an edit	% converted
Improve prose	136	8	6%	0	0
Copyedit	412	41	10%	15	37%
Wikify	447	14	3%	4	29%
Discuss AfD	144	0	0	0	0
Translate	214	10	5%	0	0
Merge	167	6	4%	1	17%
Split	108	5	5%	0	0
Verify	245	8	3%	0	0
Update	159	4	3%	0	0
Cleanup	279	5	2%	1	20%
Orphan	207	7	3%	0	0
Expand	136	4	3%	2	50%
Add image	187	13	7%	2	15%
NPOV	130	2	2%	0	0
Original research	252	9	4%	2	22%
Total	3239	136	n/a	27	n/a