Research:Wikipedia Primary School SSAJRP programme/Evaluation
| Wikipedia Primary School SSAJRP programme | Timelines & progress |
Method[edit]
Overview[edit]
Project evaluation main goals
- Analyzing the state of the art of topics related to Wikipedia Primary School Project
- Verifying the project impact in Wikipedia content
- Evaluating significant variables of interest for other research projects
Evaluation steps
- Current state, Hypothesis, 2014
- Intermediate state, Comparison, 2015
- Final state, Conclusions, 2016
Hypothesis and questions:
- If an article is impossible to find it is useless. How easy is it to find an article?
- Single article
- Number of templates and categories
- Number of visits (per categories and portals)
- Featured in homepage as an article and as a fact
- Links to the article and interlinks Portals and Wikiprojects Google Rank
- Correlation with other articles
- Total amount of articles in the same categories
- Single article
- How to “calculate” the quality of an article through quantitative data? Is it possible to define variables in order to extract significant qualitative information?
- Positive variables: Number of references and interlinks, Mentions, Portals and Wikiprojects, ecc.
- Negative variables; Issues Stub, ecc.
- Article: References and external links, Issues Templates, Edits
- Direct evaluations: Features in homepage, Article quality, Monitoring, Mentions
- Indirect evaluations: Links to the article, Number of visits, Page rank
- Assigning points according to direct and indirect evaluations in order to generate a quality rank.
- Usually people write Wikipedia articles according to their interests. Can we drive interest?
- Articles size (and variation)
- Editor task force and casual contributors
- Talk page
- New users
Process
- Define the list of 100 articles to evaluate
- Extract data to visualize
- Concept of the data visualization
- First draft
- Review and second iteration
- Final data visualization
Preliminary analysis[edit]
Concept[edit]
The preliminary analysis aims to give a general view on the current status of the selected articles. The articles under analysis are 171.
The main goals of the intermediate analysis are the following:
- Understand the current state of the selected articles
- Define and visualize a set of parameters that can lead to the understanding of the ‘quality’ of the selected articles
- Explore the relations among articles
Tools[edit]
In order to do the visual evaluation, a scraper has been implemented. It’s name is Wikimole and it is available on GitHub for further implementation.
The following is the process Wikimole uses:
- Wikimole gets data by using Wikipedia Api, a jQuery page scraper and some other external data sources such as Google Page rank and the Wikipedia article traffic statistics.
- After an external section of data filtering and cleaning, Wikimole visualizes data by using Gephi and D3.js.
Intermediate analysis[edit]
Concept[edit]
The intermediate analysis provides an overview of the state of the selected articles at six months from the previous one.
The articles under analysis are now 176. Among them 36 have been reviewed thanks to the involvement of the Wikipedia Community, 25 articles have been reviewed by expert reviewers.
The main goals of the intermediate analysis are the following:
- Understand the impact of Wikipedia Primary School Project in the selected articles
- Provide actionable information on how to improve the selected articles
- Investigate the user interest in editing Wikipedia articles and how it changes over time
Two new features the data visualizations show:
- The benchmarks regarding the previous analysis
- The articles reviewed by the Wikipedia Community and by expert reviewers.
Results[edit]
Phase 0[edit]
Report of the initial status of the selected articles.
-
Network of incoming links -
Network of outgoing links -
Incoming and outgoing links -
Pageviews -
Features of the articles -
Edits
Phase 1[edit]
Report of the provisional status of the selected articles considering the previous visual analysis. All datasets are open and available online.
-
Network of incoming links -
Network of outgoing links -
Incoming and outgoing links -
Incoming and outgoing links (link from and to articles excluded) -
Features of the articles -
Features: comparison before-after of articles improved through the Wikipedia School Project -
Features: comparison before-after of articles improved through assessment -
Features: comparison before-after of articles improved by Wikipedia community -
Features: comparison before-after of articles improved through edit-a-thon -
Features: comparison before-after of articles improved by experts -
Features: comparison before-after of articles improved through reassessment
Phase 2[edit]
Report of the final status of the selected articles considering the previous visual analysis. All datasets and protocols are open and available online.
-
Articles network in 2015 -
Articles network in 2017 -
Network of incoming links -
Balance between incoming and outgoing links -
Balance between incoming and outgoing links (except links to articles) -
Pageviews -
Comparison of articles network in August 2015 and August 2017
Findings[edit]
- The cluster of articles related to South Africa has been enhanced. That is due to the larger number of mutual links between articles belonging to South Africa topic. Some articles that links to the selected articles are the following:
- Incoming links has slightly decreased. Outgoing links are increased. The two new articles created under the framework of WIkipedia Primary School project (Kaditshwene and Makhonjwa Mountains) have respectively 13 and 25 incoming links, 46 and 58 outgoing links.
- Significant reduction of issues related to articles under examination. Despite reviews, some issue on articles under examination still remain. Among them the following articles have issues:
- The number of features of the articles (within WIkipedia Primary School project we consider features the following: references, notes, images, see also) has grown. It emerges the trend to increase the features in the lacking articles and to filter out the relevant features in articles with a rich body. Articles with few features, among those under examination, are:
- Between 2014 and 2015 the total amount of editors has slightly decreased (from 845 to 773, -8.5%). The number of edits has slightly increased (from 1463 to 1481, +1,2%). It suggests a collective greater effort in contributing to the develop of Wikipedia articles. Among articles with none or very few edits in 2015 we found:
Documentation[edit]
-
Evaluation plan, September 2014. -
Preliminary analysis, August 2015. -
Intermediate analysis methodology, March 2016. -
Intermediate analysis, March 2016 -
Final analysis, August 2017
Open datasets[edit]
All datasets are open and available online.
Related articles[edit]
Credits[edit]
Giovanni Profeta is leading the evaluation of the project with a specific focus on information design.