Research:Timestamp position modification
 | This page in a nutshell: The purpose of this experiment is to test whether displaying in a more prominent way the timestamp of an article's last revision produces a significant increase in clickthrough rate to the revision history of the article. |
Background[edit]
Currently, the only way to see the time of the last change to a wiki page is to either:
- Click on the View history tab
- Scroll to the bottom of the page and view the timestamp in UTC time [1]
This experiment will test the effects of exposing the revision history of Wikipedia articles by moving the timestamp to a more prominent position on an article, making it human-readable, and actively linking it to the article's history.
Our hypothesis is that this will increase user knowledge of how Wikipedia works, increase reader trust in articles, and encourage updating of articles which have not been edited for months or years by highlighting their outdated state.
Justification[edit]
- Impact: We expect this experiment to have a minor, indirect impact on editing. However, it will impact many thousands of both editors and readers, primarily by increasing the visibility of Wikipedia's transparent revision history.
- Community: There has not been any major clamoring for this change, though at least one community member has suggested this adjustment.
- Workload: This experiment will leverage iCloud integration development, and design work already completed at WMF - so will not be resource-intensive to undertake.
Research questions[edit]
- RQ1. Does a more prominently visible timestamp of the last revision produce a significant increase in click-through rates to the history of an article?
- RQ2. Does the addition of the timestamp list "cannibalize" clicks on the history tab?
- RQ3. Does the addition of a timestamp produce any learning effect for users (increase the clicks on the history tab, after learning how to get to the revision history page via the modified link)?
Overall metrics[edit]
- Timestamp link CTR
- Clicks on the TPM link divided by number of impressions (experimental condition only)
- History tab CTR
- Clicks on the history tab divided by number of impressions (experimental condition and control)
- Combined CTR
- Aggregate clicks leading to the revision history divided by number of impressions (experimental condition and control)
- Edit CTR
- Clicks on the edit tab, section edit link or edit CTA (experimental condition and control)
- Edit conversions
- Rate of successfully converted edits (edit save events divided by impressions of invitation to edit) (experimental condition and control)
Iterations[edit]
Experiment #1[edit]
Methods[edit]
Our first experiment will use a random sample of articles to activate the modified timestamp on. Articles will be sampled from the English Wikipedia and exclude redirects as well as articles that are part of the current Article Feedback tests. The experiment is currently enabled on 20,770 articles meeting the above criteria. 50% of the visitors to the articles in the sample are bucketed in the experimental condition, and 50% in a control group with no modification.
The time frame of the experiment is as follows:
Start Time: 2012-06-22 00:00:00
End Time: 2012-06-27 20:00:00
Analysis requirements[edit]
The clicktracking specifications are on this page. The following English Wikipedia pages should be excluded from the analysis, due to the fact that they were actively linked to by volunteers or staff in order to test the feature:
- Ryu
- Goldberg Variations
- Das Käthchen von Heilbronn
- World of Warcraft: Tides of Darkness
Notes on process[edit]
We ended up missing a page and rev_id pair from the click logs. Fortunately, in this iteration of the experiment, it only impacts some secondary research question and also provides us with some process feedback on how we can improve our integration as a team. In future, the following will help to address these issues:
- Codify experimental hand off of E3 resources in the future
- More rigorous formulation of timeline and sanity checks
Technical requirements[edit]
See documentation on MediaWiki.org
Results[edit]
The tables below show the click-through rates for the different conditions of this experiment:
| Condition | Logged-in | Impressions | History tab clicks | Timestamp link clicks | History tab CTR | Timestamp link CTR | Combined CTR | |
|---|---|---|---|---|---|---|---|---|
| A | Expr1 | No | 2,366,430 | 1,473 | 606 | .0006225 | .0002561 | .0008786 |
| B | Yes | 23,041 | 537 | 76 | .02331 | .003298 | .02661 | |
| C | - | 2,389,471 | 2,010 | 682 | .0008412 | .0002854 | .0011266 | |
| D | Ctrl1 | No | 2,370,015 | 944 | - | .0003983 | - | - |
| E | Yes | 22,943 | 430 | - | .01874 | - | - | |
| F | - | 2,392,958 | 1,374 | - | .0005742 | - | - |
Table 2 shows the results about unique visitors, filtering out repeat clicks by the same user. Here, if a user in a given condition clicked on both the timestamp link and the history tab this was counted as a single click-through event:
| Condition | Logged-in | Unique visitors | Unique CTR | |
|---|---|---|---|---|
| G | Expr1 | No | 1,820,643 | .0007036 |
| H | Yes | 15,039 | .02474 | |
| I | - | 1,835,682 | .0009005 | |
| J | Ctrl1 | No | 1,821,155 | .0004453 |
| K | Yes | 14,908 | .02059 | |
| L | - | 1,836,063 | .0006089 |
To measure the significance of these results we used a binomial test where clicks correspond to Bernoulli trials of the experiment. In table 3 the results are compared with references to subgroups in table 1 and table 2.
| Metrics compared | Leading group | % Increase | p-value | |
|---|---|---|---|---|
| A - D | Combined CTR, exp group, not logged-in (A) vs History tab CTR, control, not logged-in (D) | A | 120.6% | <2e-16 *** |
| A - D | History tab CTR, exp group, not logged-in (A) vs History tab CTR, control, not logged-in (D) | A | 56.3% | <2e-16 *** |
| B - E | Combined CTR, exp group, logged-in (B) vs History tab CTR, control, logged-in (E) | B | 42.0% | <2e-16 *** |
| B - E | History tab CTR, exp group, logged-in (B) vs History tab CTR, control, logged-in (E) | B | 24.4% | <2e-16 *** |
| C - F | History tab CTR, exp group, all users (C) vs History tab CTR, control, all users (F) | C | 46.5% | <2e-16 *** |
| C - F | Combined CTR, exp group, all users (C) vs History tab CTR, control, all users (F) | C | 96.2% | <2e-16 *** |
| G - J | Unique target completion rate, exp group, not logged-in (G) vs Unique target completion rate, control, not logged-in (J) | G | 58.0% | <2e-16 *** |
| H - K | Unique target completion rate, exp group, logged-in (H) vs Unique target completion rate, control, logged-in (K) | H | 20.2% | <2e-16 *** |
| I - L | Unique target completion rate, exp group, all users (I) vs Unique target completion rate, control, all users (L) | I | 47.9% | <2e-16 *** |
| Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 | ||||
Discussion[edit]
The timestamp position modification produced an overall positive effect on the visibility of article revision history. In particular:
RQ1: Does a more prominently visible timestamp of the last revision produce a significant increase in click-through rates to the history of an article?
The presence of the new timestamp appears to produce twice the amount of clicks on the history by readers or anonymous editors. In particular, when comparing the Combined CTR for all users in the experimental condition and the History CTR for all users in the control group, the total click-through rate for the experimental group shows an increase of 96.2% compared to the control. While this was the largest difference observed, the results are significant for each user category, in favor of the experimental group which received the "Last Modified" link. This suggests that readers are in fact aware of the view history tab and what it stands for, and that the more prominent timestamp drove them to inspect the history of an article.
RQ2: Does the addition of the timestamp list "cannibalize" clicks on the history tab?
We observed no evidence of cannibalization on the history tab for the experimental group. On the contrary, there was evidence of increased activity on the history tab, an increase of the click-through rate of 46.5% for the experimental group when combing both anonymous and registered users. Interestingly, the click-through rate for the last modified link was much lower than the history tab click through for both groups.
Finally, when normalizing the groups for unique users by filtering out repeat clicks (table 2), we still observed a significant increase for the experimental group. This is supportive that the last modified link overall contributes positively to the likelihood that any given user will visit article history.
Future Iterations[edit]
This first experiment showed that a simple modification of the timestamp of the latest revision can drive a larger attention of both readers and registered editors to an article's revision history. The next iterations of this experiment will tackle the following questions:
- to measure the effects of a higher visibility of article revision histories on reader engagement, we intend to iterate on this experiment by including a call to action upon clicking the last modified link;
- we will also measure the time difference distribution for the click-through of the last modified link and the time of the last revision to see if recency has any effect on the click-through rate;
- finally, we aim to measure whether initially clicking on the last modified link leads to a later increase of visits to article history, thereby suggesting an explicit "learning effect" (RQ3).
