Research:Asking anonymous editors to register/Study 1
Experimental CTAs
[edit]In this study, we will compare two experimental CTAs to a control.
- A control, in which an anonymous (IP) editor see no differences -- represents the current user experience where no calls to action appear before a user registers.
- A pre-edit, Call to action which is triggered by an anonymous editor clicking any of the edit buttons on an article
- A post-edit, Call to action which is triggered by an anonymous editor saving an edit
-
German Wikipedia, pre-edit condition -
German Wikipedia, post-edit condition
Hypotheses
[edit]Rationale: We suspect that the prominence of guider-based UI cues, as well as a rationale designed to align with users' concerns, will increase the rate at which anonymous editors decide to register an account. All calls to register on Wikipedia are passive for anonymous editors, in the form of static links to account creation in the site navigation and in the edit form. We suspect that these passive "create account" links (in the personal toolbar or in static MediaWiki messages mid-edit) are relatively hidden to a new users' eyes. We've observed that tooltip-based guided tours increases completion rates within a funnel.[1]
Hypothesis 1: More anonymous editors in the experimental conditions will register accounts.
Rationale: Further, we suspect that the pre-edit CTA will encourage more users to register because it places the call to action between a user and their goal -- to make an edit to a page. Further, the flow of try action and be prompted to log in is a common pattern in web applications and is already used by mw:Extension:MobileFrontend to surface logged-in only functionality like watchlists.
Hypothesis 2: More anonymous editors in the pre-edit conditions will register accounts than in the post-edit condition.
Rationale: People are motivated by contribute to the outcomes of groups with whom they identify[2]. Encouraging anonymous editors to obtain a persistent identity within the group of Wikipedians could motivate these users to make more contributions.
Hypothesis 3: Anonymous editors in the experimental conditions will be more productive.
Rationale: Anonymous editors who see the pre-edit CTA may not understand that they can still make edits anonymously. These users may become intimidated and decide not to make the edits they had planned. Further, those anonymous editors who understand the pre-edit CTA and take the time to register an account will have less time to complete the edits they started.
Hypothesis 4: A smaller proportion of anonymous editors in the pre-edit experimental condition will complete edits.
Rationale: On English Wikipedia at least, anonymous editors are reverted more than twice as often as registered users.[3] We suspect that this is due in part to the lack of a persistent identity associated with anonymous edits. Behavioral economic research suggests that obtaining a persistent identity (e.g. registering an account) within an open community could lead users toward increased feelings of accountability[4]. As editors consider their future reputation, they may think more carefully about how the edits they make reflect on them and we suspect that this will lead anonymous editors to act less disruptively. Further, registered editors may apply greater (and potentially excessive) scrutiny to anonymous editors. Wikipedians may be more inclined to improve an edit rather than revert it if they perceive the original editor to be a part of the community.
Hypothesis 5: Anonymous editors in the experimental conditions will be reverted and blocked less often.
Experimental design
[edit]
A change was deployed to English, German, French and Italian Wikipedias that enabled bucketing anonymous users into the experimental buckets at approximately 2014-05-19 18:08:00 UTC. Experimental CTAs were shown to users for approximately 7 days -- until 2014-05-26 18:08:00 UTC.
Results
[edit]In order to test Hyp 1 and Hyp 2, we needed a way to compare the registration rates between experimental conditions and the control. As discussed in #Methods, we developed the notion of the "pure anon editors" as the type of editor the experiment was targeting. By comparing the proportion of these pure anons who register across conditions, we should be able to reason about changes in the rate.
Registration rate show the different proportion of pure anons who register by experimental bucket. Surprisingly we see that different wikis can have drastically different registration rates. English Wikipedia has baseline (control) rate of about 1.2% while German Wikipedia has a baseline of 0.5%. In other words, pure anons registered accounts in English Wikipedia at twice the rate that they did in German over the same time period. There were also similar differences in underlying registration rates for Italian and French Wikipedias – 0.61% and 0.75% respectively.
Next, we look at the effects of the intervention. In both experimental conditions and across all wikis, we see a significant increase in the rate at which pure anons register accounts. Further, the pre-edit condition had a substantially larger effect. In all wikis, the pure anons in the pre-edit condition were more than twice as likely to register an account than those in the control. Together, these results firmly support Hyp 1 and Hyp 2.
| wiki | Baseline % (est. n) | Pre-edit % (delta) | Post-edit % (delta) | Pre-edit factor % (est. n) | Post-edit factor % (est. n) |
|---|---|---|---|---|---|
| dewiki | 0.524 (620) | 1.339 (+0.815) | 0.61 (+0.086) | 255.5 (1584) | 116.5 (723) |
| enwiki | 1.184 (6874) | 3.165 (+1.980) | 1.402 (+0.218) | 267.2 (18367) | 118.4 (8138) |
| frwiki | 0.75 (665) | 2.462 (+1.712) | 1.163 (+0.412) | 328.1 (2182) | 155 (1031) |
| itwiki | 0.611 (347) | 2.028 (+1.417) | 1.016 (+0.405) | 332.1 (1151) | 166.3 (576) |
Rates and estimates. Rates plotted in #Registration rate are plotted with estimates of the raw count of registrations after re-extrapolating the experimental conditions.
In order to look for evidence of boosted productivity, we explored the editing patterns of the entire group of pure anons -- whether or not they ended up registering an account or not -- since any change in productivity may affect those editors who decide not to register. Because of the large amount of skew to the distribution of edits per editor, we use the geometric mean in order to model the expected number of edits per editor.
Figure #All editing activity shows the expected number of revisions saved per pure anon. While it appears that pure anons in the post-edit condition made about the same number of edits as pure anons in the control, pure anons in the pre-edit condition made substantially ~ 25% fewer edits.
To ensure that this observed decrease in edits reflects actual content work, we replicated the analysis with edits to non-articles filtered out. Figure #article editing activity shows a similar trend of no significant difference for post-edit, but a substantial decay for pre-edit.
Finally, to ensure that the lost edits weren't merely edits that wouldn't have been reverted anyway, all reverted revisions were removed from the dataset which left behind only R:productive edits. #Productive article editing activity shows that this final control made no substantial difference to the trend of lost edits in the case of the pre-edit condition. In the next section, we'll explore the source of these lost edits.
The above results include eligible users in the experiment whether they went on to register accounts or not. Since looking at these cohorts of "pure anons" shows a decrease in productivity in the pre-edit condition, we wanted to know why. Did we decrease productivity by attracting new registered users who were less likely to edit? Or did we discourage users who chose not to register, and who wanted to edit as an IP? To answer this, we looked at the sub-cohorts of users who chose to accept the CTA and register an account.
We observed that users in the post-edit condition who went on to register were significantly more likely to reach active editor status, compared to the control group (p=0.035). We also observed that users in the pre-edit condition who chose to register were significantly more likely to make one or more edits (p<0.001).
In short, the users who accepted the CTA and registered an account were in fact more likely to edit than their counterparts in the control group. Consequently, we assume that the drop in overall productivity was due to discouraging users who chose not to register accounts.
| Wiki | Bucket | Registered users reaching 1+ edits | Registered users reaching 5+ edits | Total registered users |
|---|---|---|---|---|
| dewiki | control | 166 | 11 | 415 |
| dewiki | pre-edit | 183 | 27 | 413 |
| dewiki | post-edit | 296 | 31 | 660 |
| enwiki | control | 1840 | 283 | 6171 |
| enwiki | pre-edit | 1808 | 319 | 5922 |
| enwiki | post-edit | 3637 | 451 | 8768 |
| frwiki | control | 201 | 33 | 619 |
| frwiki | pre-edit | 235 | 32 | 702 |
| frwiki | post-edit | 492 | 43 | 1037 |
| itwiki | control | 89 | 15 | 232 |
| itwiki | pre-edit | 102 | 15 | 285 |
| itwiki | post-edit | 266 | 40 | 465 |
| Bucket | Registered users reaching 1+ edits | Registered users reaching 5+ edits | Total registered users |
|---|---|---|---|
| control | 2296 | 342 | 7437 |
| pre-edit | 4691 | 565 | 10930 |
| post-edit | 2328 | 393 | 7322 |
- Results of a chi-squared test
- Pre-edit vs. control, 1+ edits:
prop.test(c(2296, 4691), c(7437, 10930))(x^2=271.95, p<0.001) - Pre-edit vs. control, 5+ edits:
prop.test(c(342, 565), c(7437, 10930))(x^2=2.95, p=0.086) - Post-edit vs. control, 1+ edits:
prop.test(c(2296, 2328), c(7437, 7322))(x^2=1.42, p=0.234) - Post-edit vs. control, 1+ edits:
prop.test(c(342, 393), c(7437, 7322))(x^2=4.45, p=0.035)
In order to look for evidence of hypothesis Hyp 4, we explored the rate at which pure anons who started an edit flow (clicked edit) completed an edit at some point during the experiment. Figure #Edit completion rates plots the proportion of pure anons who completed an editor per condition. As in the previous section, we see insignificant difference between the post-edit condition and the control -- which makes sense since the intervention in post-edit would not appear until after an edit had been completed.
However, we also see the substantial drop in the proportion of pure anons who both click edit and complete an edit. As when we examined Hyp 3 we wanted to make sure that the observed drop in completion rates were not simply a result of filtering non-productive editors. Figure #Productive editor rates plots the rate at which pure anons who clicked edit saved a productive edit. Again we see a similar trend where pure anons in the pre-edit condition are substantially less likely to save a productive edit than pure anons in the control condition.
Together, these results support Hyp 4.
In order to look for evidence of changes in the amount of burden due to the presence of the experimental CTAs, we measured the revert rates and block rates (still TODO as of 15:19, 2 July 2014 (UTC)) of pure anons (whether the registered an account or not). In Hyp 5, we hypothesized that requesting that users register would result in less burden for Wikipedians. If this hypothesis is supported, we should see decreased in the rate of reverts per editor and the rate of blocks per editor.
One of the difficulties in operationalizing block rate as a fair measure of the amount of expected reverting is that some editors will do more than others. If we draw a proportion of reverted edits from the entire pool edits made by editors in a particular condition, editors who make more edits will have an over-weighted effect on the result of the calculation. Similarly, if we generate a mean of the individual editor rates, editors who only make one edit (and therefore have an individuate revert rate of either 1 or 0) will dominate the statistic. In order to control for both of these effects and get a sense for the overall rate of reverts in the experimental conditions, we sample exactly one article edit from all editing pure anons and generated a pooled proportion of reverted revisions.
#Reverted sampled revision prop plots the sampled revert proportion. Across all wikis, pure anons in the pre-edit condition who successfully edited an article were significantly less likely to have their revisions reverted. In the post-edit condition, the results are less consistent. Pure anons in the post-edit condition were significantly more likely to be reverted than editors in the control on German Wikipedia while the same group on French Wikipedia were significantly less likely to be reverted than editors in the control.
While it seems clear that the reduced revert rate of the pre-edit condition could be explained by the reduced probability of completing an edit reported in the previous section, the higher revert rate in the control and post-edit conditions did not account for the higher expected # of productive edits seen in Hyp 3 boosted productivity.
TODO: block rates
Funnels
[edit]Pre-edit
[edit]
Post-edit
[edit]
Footnotes
[edit]- ↑ Our third A/B test of the Onboarding process tested an experience with guiders vs. one without
- ↑ Karau, S. J., Markus, M. J., & Williams, K. D. (2000). On the elusive search for motivation gains in groups: Insights from the collective effort model. Zeitschrift fur Sozialpsychologie, 31(4), 179-190.
- ↑ Wikipedia Statistics - Edit and Revert Trends: English
External links
[edit]- Analysis code repository: https://github.com/halfak/Anonymous-phenomena