Research:Understanding Curious and Critical Readers/Detailed analysis of knowledge networks of wikipedia readers

Previous studies have developed a network approach to curiosity. Looking at pages visited by Wikipedia readers, they identified different styles of curiosity (most notably they identified two different types denoted hunter and busybody). However, this relied on a small/homogeneous sample of participants in a laboratory-type setting where they were given specific instructions to browse Wikipedia.

In this work, we aim to generalize the application of this framework to a larger population. For this we consider knowledge networks from readers of Wikipedia improving upon the previous studies in several ways: i) it yields a much larger sample; ii) it yields a much more diverse sample (different countries, different Wikipedia-language version, more representative demographics); iii) observations from logs without instructions for participants. Furthermore, it allows us to obtain new quantitative evidence for the characterization of curiosity styles of readers beyond the hunter- and busybody types.

[Note]

• this is still work in progress
• we will share the link to the pre-print or published manuscript here

Results

We aim to generalize previous findings about curiosity inferred from knowledge networks of readers. For this, we compare the knowledge networks from two different sources

• The KNOT dataset comprises 149 participants of a lab-based study in which they were instructed to browse English Wikipedia for 15 minutes each day over a period of 3 weeks.
• the WAPP dataset comprises several millions of navigation sequences of readers of English Wikipedia using the Wikipedia app over a period of one month.

Propensity-score matching

Looking at the summary statistics, we find substantial differences between the KNOT and the WAPP datasets. Most notably, on average, readers in the WAPP dataset visit fewer pages. In addition, they also visit fewer unique pages, visit on fewer days, and reach a lower fraction of pages through internal hyperlinks.

Summary statistics of KNOT and WAPP datasets.

In order to avoid that differences in these summary statistics affect comparison of the knowledge networks, we generate a biased sample of the WAPP dataset. Specifically, we use propensity score matching on the number of pageviews per reader using the package psmpy. While this leads to a smaller subsample, the summary statistics between KNOT and WAPP become more similar. In fact, the distributions of the number of visited pages per reader are almost identical. Furthermore, the differences in the distributions of the other summary statistics also become less pronounced. Following these insights, for the WAPP dataset we will always consider biased subsamples from this approach unless otherwise stated.

Network metrics for comparison

Next, we systematically compare the knowledge networks generated from the two datasets. In previous studies, curiosity in knowledge networks was  characterized using mainly the clustering coefficient and the characteristic path length. Here, we expand the characterization of the knowledge networks by calculating a set of 8 network metrics: degree, clustering coefficient, characteristic path length, global efficiency, core-ness, modularity, number of groups, minimum description length.

Distribution of the network metrics in knowledge networks from Wikipedia and KNOT. Solid (PDF from kernel density estimation), dotted (normalized histograms)

Looking at the marginal distributions for each metric, we can compare the similarity between the population of knowledge networks from the KNOT and WAPP datasets. Overall, we find that distributions for the KNOT and WAPP data are qualitatively similar across all network metrics. For example, for modularity both distributions show a bimodal structure with peaks at approximately similar scales. This provides evidence that curiosity styles uncovered in previous studies using the KNOT data generalize to broader population of Wikipedia readers in the WAPP data.

Quantifying similarity

KS-distance for the distribution of knowledge networks (for each network metric) between the Wikipedia reference dataset and other datasets.

In order to quantify the similarity of the population of knowledge networks in the WAPP and KNOT datasets, we calculate a distance d (with values from 0...1) between the respective distributions. Specifically, we use the Kolmogorov-Smirnov distance for this, a commonly used test-statistic. For better interpretability of the scales, we perform similar comparisons with reference knowledge networks generated from other dataset and null models.

The averaged distance between WAPP and KNOT is around d=0.3. Comparing knowledge networks from WAPP with those from targeted navigation experiments (such as Wikispeedia) yields much larger distances (d~0.7). In contrast, comparing WAPP datasets from different points in time, the distance is much smaller (d~0.5).

Nevertheless, the knowledge networks in WAPP do show a larger variation when stratifying by the country from where readers access. The knowledge networks in WAPP are similar when comparing to those when stratifying by different countries. While for most countries (such as Canada, United States, Australia, and Great Britain) distances are similarly small (d~0.05), other countries such as Germany yield distances that are comparable to the distance between WAPP and KNOT (d~0.2).

In order to account for the fact that readers from countries such as Germany will access preferably the German Wikipedia (instead of the English Wikipedia), we create WAPP datasets for 14 different language versions of Wikipedia (similar to the ones in previous studies on readers). Comparing the knowledge networks for WAPP datasets from different languages, we find that distances between English and most languages (German, Russian, Japanese, Spanish, Dutch, Hebrew, Ukrainian, Chinese) are of comparable magnitude to the distance between WAPP and KNOT (0.1< d <0.3). Some languages (Arabic, Hungarian, Bengali, Romanian, Hindi) show slightly larger distances (0.3< d < 0.4). However, the latter can be partly explained by the fact that the underlying networks of hyperlinks between articles are much smaller in size.

In order to assess to which degree similar knowledge networks could result from chance alone, we compare the WAPP data with knowledge networks from different null models. Overall, the distances to the null models are much larger compared to the KNOT data. Whereas a random walk on the hyperlink network yields the comparatively smallest distance (d~0.4), random networks yield very large distances (d~0.8).

Overall, these results shows that knowledge networks from WAPP data are very similar to those from KNOT data in earlier studies. This is not a trivial finding because other reference or null model knowledge networks yield much larger differences. In turn, this also implies that curiosity styles uncovered from knowledge networks in the KNOT data can be generalized to the population at large in the WAPP data.

We demonstrate this specifically with the hunter- and busybody-types, the two main curiosity types identified in this approach, which are characterized through different knowledge networks. Specifically, we define a scalar hunter-busybody-score aggregating different network metrics, which captures whether a knowledge network is more hunter- or more busybody-like. We find that WAPP data and KNOT data show similar populations in terms of their hunter-busybody score. Figure: t.b.a.