Research:Community portal redesign

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
Research project
Community Portal redesign
Main contact Growth team
WMF contact Maryana Pinchuk
Start 2012-08
End 2012-10
Status completed Icon 100 percent.png
Open access This project has open access publications
WMF support
Wikimedia research projects Wikimedia research projects

The purpose of this experiment was to test various redesigns to the Community portal on English Wikipedia, in order to identify the elements that will make it a more useful space for new and/or existing editors.

Background[edit]

Questions asked by new users at the Teahouse on English Wikipedia; the fifth most common question is where to find task suggestions or projects

The Community portal was created in 2004 on the English Wikipedia, as an editor-focused (as opposed to reader-focused) offshoot of the main page, and was originally a simple task list and a few links to fundamental policies and guidelines.[1] Over time, as the number of editors grew and other pages like Village pumps and WikiProjects became more lively centers of community discussion, the Community portal became less important. However, the page is still linked from the sidebar and draws in around 10,000 pageviews a day.[2]

A qualitative assessment of current Community portal viewers demonstrated that many of these pageviews probably come from readers and very new users who are looking for help or information from experienced editors, but most experienced editors no longer watch the page.[3] As an additional datapoint, many of the new users who come to the Teahouse ask where to find task suggestions and/or general to-do lists and are guided to the Community portal's open task list.[4] (See fig. right)

Unfortunately, the bot that updated the open tasks template with new tasks stopped running in February 2012.[5]

These findings suggest that the Community portal might potentially fill a vital role for newer and less experienced users, but the current format is not serving this purpose effectively. Because the page is no longer used by most of the community of established editors, it may be an excellent space to iteratively test changes to design and layout and measure reader conversion and new editor engagement/retention.

Research questions[edit]

In this experiment, we are testing our hypotheses that:

  1. It is mainly new and less experienced users of Wikipedia who are landing on the Community portal, and improving the information that is presented there will educate and empower these users to potentially become established Wikipedians.
  2. Reducing and streamlining the information presented on the Community portal will make it a powerful resource for new and current Wikipedians alike.

Assumptions[edit]

  1. The Community portal could potentially be a useful starting place/landing page for newer and less experienced users, but it is currently not set up with any target audience in mind.
  2. The overwhelming number of links on the Community portal makes it unnavigable and unusable.
  3. The redundant and outdated links on the Community portal confuse and disorient newer and less experienced users.
  4. The lack of focus on any one audience (e.g., readers, new editors, established Wikipedians) makes the Community portal not useful for anyone.
  5. The lack of focus on any one aspect of community engagement (e.g., suggestions for tasks to do, projects to join, help spaces to visit, etc.) makes the Community portal not useful for any one of these calls to action.
  6. The design/layout of the Community portal does not effectively highlight the important information that is contained there and needs to be updated.

Methodology[edit]

  • Make weekly changes to the design/layout of the Community portal page, bringing one main page element into focus (e.g., task recommendation, Wikiprojects, help pages, adopt-a-user program...)
  • Measure clicks and impressions on the page and rate of successful conversions (e.g., tasks attempted/completed, WikiProjects joined, help pages used, new users adopted, etc.)
  • Identify successful changes and discuss incorporating them into a stable new version of the page

Control data-gathering[edit]

The Community portal as of July 24, 2012, before testing
The opentask list as of July 24, 2012
  • Measure current clickrate on page for one week to determine which elements are being used

Initial results[edit]

After three days of clicktracking, there were a total of 3212 links clicked (approx. 9% of the pageviews for that period, roughly 36,000). Below is some more fine-grained data:

Target
  • 87% (2802) of the clicks came from unregistered users
  • 13% (410) came from registered users
  • As expected, there was no one overwhelming area where users clicked most. Among the more popular links were the Signpost (7%), the featured WikiProjects and Portals (each 6%), and, oddly, the Community portal itself (5%) – possibly because users were expecting something different on the page and clicked on the sidebar link again
  • Just 3% of the clicks were to items in the Open tasks box, and only one of the articles clicked on was edited around the time of the experiment, meaning that none of these clicks were successful conversions
Referrer
  • 37% (3425) of visitors who clicked on a link in this period were coming from the article namespace on Wikipedia
  • 18% (1657) of visitors who clicked on a link were coming from the Main page
  • 13% (1217) of visitors who clicked on a link were coming from a Special page (preferences, recent changes, watchlist, etc.)

One week results[edit]

After a week of clicktracking, there were a total of 9156 clicks on links in the Community portal, representing 13% of the approximately 70,000 pageviews for the period of July 27-August 2.

Users
  • 86% (7847) of the clicks came from unregistered users
  • 14% (1299) of the clicks came from registered users
  • The majority of the registered editors (903, 70%) had made less than 1,000 edits to English Wikipedia
Links clicked

The majority of links clicked are located on the lefthand navigation toolbar. Some of the most popular included:

  • The Community portal itself - 10% (888)
  • Portals (e.g., Portal:Featured content) - 9% (818)
  • The Main page - 9% (805)
  • Special (e.g., Special:Random, Special:RecentChanges) - 9% (805)

Of the actual page content clicked, the most popular were:

  • The Signpost content - 7% (635)
  • Discussions (Requests for comment) - 6% (553)
  • Articles in the open tasks area - 5% (418)
  • WikiProjects - 4% (411)

Importantly, the general trend of clicks was highest at the top of the page and lowest at the bottom, and higher on the left than the righthand side, which is consistent with most eye-tracking studies of screen reading.

Conclusions[edit]

These quantitative findings support the qualitative feedback initially received: most visitors to the Community portal who click on any of its page elements are unregistered or relatively inexperienced editors. The haphazardness of elements clicked, and especially the very high number of clicks on the Community portal and other sidebar links, suggest that these users are not currently finding the page useful, and that there is much need for improved layout and design. The higher volume of clicks on links located in the top lefthand quadrant of the page suggests that more salient information should be placed there, so any topical redesign should be focused primarily in that area.

First iteration: tasking (8/6-813)[edit]

First iteration of opentask redesign, August 6, 2012
See also: Research:Community portal redesign/Opentask

Hypothesis: new users who come to this page might be looking for things to do/ways to help (per the Teahouse survey), but that information is not prominently displayed. This redesign will test the effect of placing a task list in the area where more visitors to the page are likely to see/interact with it.

Changes made:

  • the header bars were removed/collapsed as much as possible
  • the open tasks list was moved to the top lefthand corner of the page
  • the task list was split into easier new-user-directed tasks (top) and more difficult experienced-user-directed tasks (bottom)

Initial results[edit]

The following was observed after a day of clicktracking:

  • The overall number of clicks on the page increased (from an average of ~1308 a day during the control period to 1524 a day during the test period)
  • The clickrate on open task articles went up from 3% of all clicks during the control period to 8% during the test period – a statistically significant increase
  • However, the rate of task completion was also the same: no open task articles were edited during the experimental period
  • Clicks on the open tasks list occurred mostly on articles in the first two task categories ("Add wikilinks" and "Copyedit", 18 and 20 of 123 open task clicks, respectively), followed in frequency by the two bottom task categories, "Requested articles" (12 of 123) and "Requested images" (10 of 123)

Second iteration: reordered task list (8/17-8/23)[edit]

To test for interest-dependence in tasking, the list of tasks was reordered to move high-click topics to the middle and low-click topics to the top - if clicks on tasks by topic remain the same, this will suggest that task list order does not play a significant confounding role in what kinds of tasks users are choosing to click on.

Results
Task # of total clicks # of unique clicks # of returned clicks
Wikify 49 92% 8%
Copyedit 60 82% 18%
Remove original research 37 95% 2%
De-orphan 183 88% 12%
Verify 41 90% 10%
Merge 61 90% 10%
Split 91 93% 7%
Expand 166 90% 10%
Make neutral 36 94% 6%
Update 78 92% 8%
Discuss AfD 178 93% 7%
Clean up 55 87% 13%
Add image 7 100% 0%
Translate 104 95% 5%
Create 60 88% 12%
Total 1206

After one day of clicktracking, it appeared that people were still almost twice as likely to click on the two top-performing tasks, wikifying and copyediting, than on other tasks on the list, despite the fact that these two tasks no longer appeared in the eye-catching top area.

However, after 7 days, these results were reversed. Orphan and stub expansion, which were at the top of the new reordered list, became the most popular click area. The third most popular was AfD discussion, which was at the bottom:

Conclusion

It appears there is a weak relationship between task type and clicks, with people generally preferring wikifying and copyediting (two relatively simple and straight-forward tasks). However, there is a strong relationship between order of tasks and clicks—with items at the very top of the list receiving the most attention.

Third iteration: tasking (9/2-9/8)[edit]

Redesigned page and task list, September 2012
Collaboration-focused redesign, October 2012

Because clicks on articles went up significantly, but task completions did not, more design work on the task list seems promising.

Changes made:

  • the task list was redesigned to place easier tasks in the high-priority page area (top left)
  • more whitespace was added, tasks were placed on a grid, heavy formatting was removed
  • prominent links to tutorial pages were added next to each task for receiving instruction on how to complete the tasks correctly
Results
Task # of total clicks # of unique clicks # of returned clicks
Wikify 435 90% 10%
Copyedit 345 85% 15%
Remove original research 186 86% 14%
De-orphan 148 89% 11%
Verify 101 92% 8%
Merge 74 89% 11%
Split 80 90% 10%
Expand 94 93% 7%
Make neutral 101 90% 10%
Update 86 95% 5%
Discuss AfD 72 90% 10%
Clean up 128 96% 4%
Add image 0 0 0
Translate 92 92% 8%
Create 183 85% 15%
Total 2125

High-level data tables[edit]

Control period
August 7-13 (1 week)
  • Pageviews: ~69,000
  • Clicks (clicks on at least one outgoing link, did not close browser or click back/home button on browser): 10,522
  • Clicks per pageview: 15%
  •  % of unique user clicks: 69%

Breakdown by user type and return/unique:

User type # total clicks  % per pageviews  % per total clicks # unique visitor clicks  % unique per user type # returned visitor clicks  % returned per user type  % returned per pageviews
Anonymous 9038 13% 86% 6478 72% 4044 45% 6%
Registered 0 edits 355 1% 3% 265 74% 90 25% 0.1%
Registered 1-10 edits 236 0.3% 2% 126 53% 110 47% 0.2%
Registered > 10 edits 889 1% 8% 446 50% 443 50% 0.6%
Referrer page # of referrals  % per total clicks # unique referrals  % per total clicks
Articles 3537 34% 2827 27%
Main Page 2037 19% 1451 14%
Special:Search 970 9% 625 6%
Test 1 (reorganizing page, task list at top)
August 17-23 (1 week)
  • Pageviews: ~56,000
  • Clicks (clicks on at least one outgoing link, did not close browser or click back/home button on browser): 13,072
  • Clicks per pageview: 23%
  •  % of unique user clicks: 71%

Breakdown by user type and return/unique:

User type # total clicks  % per pageviews  % per total clicks # unique visitor clicks  % unique per user type # returned visitor clicks  % returned per user type  % returned per pageviews
Anonymous 11515 21% 88% 8343 72% 3172 28% 6%
Registered 0 edits 447 8% 3% 293 65% 154 34% 0.3%
Registered 1-10 edits 245 0.4% 2% 129 53% 116 47% 0.2%
Registered > 10 edits 892 2% 7% 491 55% 401 45% 0.7%
Referrer page # of referrals  % per total clicks # unique referrals  % per total clicks
Articles 3844 29% 2980 23%
Main Page 3666 28% 2781 21%
Special:Search 1363 10 % 965 7%
Test 2 (redesigned task list)
September 3-8 (1 week)
  • Pageviews: ~52,500
  • Clicks (clicks on at least one outgoing link, did not close browser or click back/home button on browser): 13,105
  • Clicks per pageview: 25%
  •  % of unique user clicks: 68%

Breakdown by user type and return/unique:

User type # total clicks  % per pageviews  % per total clicks # unique visitor clicks  % unique per user type # returned visitor clicks  % returned per user type  % returned per pageviews
Anonymous 11753 22% 90% 8105 69% 3648 31% 7%
Registered 0 edits 401 1% 3% 300 75% 101 25% 0.6%
Registered 1-10 edits 203 0.4% 2% 116 57% 87 43% 0.2%
Registered > 10 edits 797 2% 6% 406 51% 391 49% 1%
Referrer page # of referrals  % per total clicks # unique referrals  % per total clicks
Articles 4069 31% 3232 25%
Main Page 2105 16% 1518 12%
Special:Search 2526 19% 1732 13%
Test 3 (redesigned task list)
September 9-15 (1 week)
  • Pageviews: ~55,000
  • Clicks (clicks on at least one outgoing link, did not close browser or click back/home button on browser): 11981
  • Clicks per pageview: 22%
  •  % of unique user clicks: 68%

Breakdown by user type and return/unique:

Possible next steps[edit]

Task suggestions/recommendations
to make the open task list more prominent and useful

Potential changes:

  1. Move "Help out" section above the fold and move "Open tasks" to the left (higher visibility and importance) Yes check.svg Done
  2. Streamline and simplify the task recommendation list to make it more user-friendly (fewer categories and tasks, more white space between links) Yes check.svg Done
  3. Test random articles (current) against targeted articles (low quality/high importance articles, similar to the "high priority" algorithm used by Suggestbot) Symbol wait.svg In progress...
Featured WikiProjects/collaborations
to surface current collaborations of note

Potential changes:

  1. Remove Signpost and CENT
  2. Replace the Community bulletin board with a reduced number of featured projects/collaborations, presented in a more newbie-friendly way
  3. Add images
  4. Reload with a different project/projects with each pageload
Help/mentoring
to surface useful help and mentoring spaces in an non-overwhelming way

Potential changes:

  1. Remove all sidebars, navbars, boxes, etc., except for "Guidelines and resources"
  2. Reduce number of links to help spaces to just three or four most essential/active ones (e.g., Teahouse, Help desk, Reference desk)
  3. Feature three or four experienced users seeking adoptees

Notes[edit]