Research:Editor milestones

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
Research project
Editor milestones
Main contact Growth team
WMF contact Maryana Pinchuk, Steven Walling
Start 2012-04
End 2012-07
Status completed Icon 100 percent.png
Open access This project has open access publications
WMF support
Wikimedia research projects Wikimedia research projects

Background and rationale[edit]

A number of studies have suggested that barnstars and other onwiki rewards motivate Wikipedians to contribute. However, due to the size of the editing community on English Wikipedia, many editors' good work goes unnoticed and therefore unrewarded. To learn more about the population of editors who are on the verge of becoming "established" Wikipedians, and to test the effects of personalized recognition on editing behavior, in April 2012 we began tracking and thanking all users who reach their 1,000th edit to articles on English Wikipedia.

Methodology[edit]

Using a slave database of the English Wikipedia, we began logging the number of times that users on English Wikipedia made their 1,000th edit to the article namespace (including deleted revisions). We then built an analytics dashboard that automatically updates with the username and timestamp of each 1,000 edit event.

We chose 1,000 edits somewhat arbitrarily, with the rationale that...

  • it is a significant contribution to the encyclopedia, and therefore inherently praiseworthy
  • it is highly unlikely to be a milestone reached by vandals, spammers, or other overtly disruptive editors
  • it suggests a possible "power-user" or very highly active Wikipedian in the making

Starting on April 5, 2012, we began thanking each person who reached this milestone with a personalized barnstar, using WikiLove.

First iteration[edit]

1K editors by year of first edit, out of a sample of 158

Currently, about 5-6 Wikipedians cross the 1,000-edit threshold each day. In our first month of performing this experiment, we thanked about 160 editors and analyzed their contribution history to find when they began editing and what subjects they edited most frequently. We then analyzed their editing activity in the 3-day period after receiving a barnstar for reaching their 1,000-edit milestone.

Age of editors (yearly cohorts)[edit]

In addition to what they were currently editing, we also looked at what year these Wikipedians made their first contributions. What we found was that these editors roughly fell into two camps: slow and steady editors who began contributing five or six years ago and accumulated their 1,000 edits over a long period of time, and new users who were among the most active Wikipedians in their first few months of editing and quickly ratcheted up their 1,000 edits.

As demonstrated by the chart to the right, the new Wikipedians outnumber the old – 59% made their first edit in 2009 or later, out of 158 editors to date.

Editing patterns - Pre-Barnstar[edit]

Primary topics and editing activity of current 1K editors

In order to write a personal thank-you message to each editor, we looked at their user pages and contribution histories to determine what subjects they had been most actively working on. What we found was that many of these Wikipedians were intensely focused on one topic or editing activity. The chart on the right illustrates the most common topics and the percentage of users whose edits mostly fall into each topic. These topics were:

  • Sports (e.g., football teams, car races, rugby players)
  • Music (.e.g., songs, albums, composers)
  • Milhist (e.g., battles, military technology)
  • Aviation (e.g., airports, air lines)
  • Video games
  • TV/film/radio
  • Bios (e.g., biographies of people, living or dead, not necessarily connected by any particular topic)
  • Local culture (e.g., notable places, events, and cultural phenomena in the city/country where the editor is from)
  • World culture (e.g., notable places, events, and cultural phenomena in different parts of the world, not where the editor is from)
  • Math/science (e.g., theorems, astronomy)
  • Technical gnoming (e.g., copyediting, disambiguating/repairing links, adding navboxes)
  • Lit/comics (e.g., novels, manga, comic books)
  • Vandal-fighting/patrolling
  • Other/varied (e.g., a narrow topic not covered above, such as solar power, or diverse editing activity in many subject areas)

Somewhat surprisingly, by far the most common topic of editing for this sample of users was local culture. Whether they self-identified as being from Turkey, Bosnia-Herzegovina, China, the U.S., or the Philippines, these editors tended to create and/or improve articles about their home town, state, or country.

The other common area of interest for this sample of users was popular culture – in the form of television, film, sports, and music. It is also interesting to note that some of the topics covered by the more active and successful WikiProjects, such as military history, aviation, and biology, were either very rare or not found at all in this sample.

Editing Patterns - Post-Barnstar[edit]

  • Mean # of edits in 3-day period after 1k event per user: 49.85
  • Mean # of blocks in 3-day period after 1k event per user: 0.01 (1%)
  • Mean # of warnings in 3-day period after 1k event per user: 0.24 (24%; however, a few users were warned several times)

From the low rate of blocks and warnings and the relatively large number of edits in the 48 hours after these editors are given a barnstar, we can assume this class of editors continues to be of high quality. However, in order to determine whether a barnstar was the cause of an increase or decrease in activity, a control group must be developed.

Results[edit]

Here's a supplemental analysis comparing the prevalence of edits over three-day periods, both before and immediately after the barnstar was delivered:

http://meta.wikimedia.org/wiki/User:Renklauf/Barnstarred_Editors_Analyses

Second iteration[edit]

Because we are finding patterns in editor behavior that may suggest that thanking editors (via barnstar) for reaching milestones may have a significant impact on their editing, we will iterate on this research by randomly assigning users to a control (no barnstar) and test (barnstar) group. We will then analyze their contributions to determine the following:

Short-term editing effect[edit]

Control group
  • Average number of contributions per user in the 0-3 day period before 1k edit timestamp
  • Average number of contributions per user in the 0-3 day period after 1k edit timestamp
  • Averages for new editors (2009-2012) vs old editors (2002-2008)
Test group
  • Average number of contributions per user in the 0-3 day period before barnstar timestamp
  • Average number of contributions per user in the 0-3 day period after barnstar timestamp
  • Average number of contributions per user in the 0-3 day period before 1k edit timestamp
  • Average number of contributions per user in the 0-3 day period after 1k edit timestamp (previous month's cohort was 49.85)
  • Averages for new editors (2009-2012) vs old editors (2002-2008)

Long-term editing effect[edit]

Control group
  • Average number of contributions per user in the 0-30 day period before 1k edit timestamp
  • Average number of contributions per user in the 0-30 day period after 1k edit timestamp
Test group
  • Average number of contributions per user in the 0-30 day period before barnstar timestamp
  • Average number of contributions per user in the 0-30 day period after barnstar timestamp
  • Average number of contributions per user in the 0-30 day period before 1k edit timestamp
  • Average number of contributions per user in the 0-30 day period after 1k edit timestamp

Results[edit]

Short-term editing effect[edit]

This experiment concerns the measurement of revision activity in all namespaces in the context of the following events:

  • 1K Event - the time at which an editor reaches their 1000th edit
  • Barnstar Event - the time at which the 1K editor received a Barnstar from WMF recognizing their accomplishment of 1K edits


Below is the list of control users and the timestamp of their 1K event:

List of 1K Users - Control (1K timestamp)

ViriiK  2012-05-15 02:58:25
Treydavis3      2012-05-15 06:00:16
Listmeister     2012-05-15 21:12:27
Mike Schwartz   2012-05-16 06:58:17
Horhey420       2012-05-17 19:15:05
Adjwilley       2012-05-18 04:58:54
Loginnigol      2012-05-18 11:39:58
Olaff   2012-05-18 16:47:09
Shanghainese.ua 2012-05-18 20:13:15
Smokeyfire      2012-05-19 00:53:09
Wikimace08      2012-05-19 21:35:27
Acsian88        2012-05-20 02:31:29
Jackl   2012-05-20 08:12:00
DonBarton       2012-05-20 19:57:03
Abh87   2012-05-20 21:03:52
AzseicsoK       2012-05-21 01:20:31
Bazuz   2012-05-21 12:08:57
Inscription     2012-05-21 16:41:46
Ttenchantr      2012-05-21 18:32:56
Dzlinker        2012-05-21 22:48:32
Kwesiidun91     2012-05-22 02:30:50
Dougsim 2012-05-22 12:39:56
Gwen-chan       2012-05-22 15:37:50
Room429 2012-05-22 18:01:19
Ekin    2012-05-23 12:09:20
JoelWhy 2012-05-23 22:00:59
M0rphzone       2012-05-24 01:46:09
KarlcmanRockets 2012-05-24 11:48:10
Ich     2012-05-24 16:52:41
Tyrannus Mundi  2012-05-25 18:40:27
Callanecc       2012-05-26 02:21:46
BrightStarSky   2012-05-26 05:22:40
Allkayloh       2012-05-26 13:46:11
Paulika1995     2012-05-27 11:43:46
Trafalk09       2012-05-27 20:52:31
TRLIJC19        2012-05-28 04:17:44
PaulWicks       2012-05-28 12:34:14
Beefyt  2012-05-29 06:02:32
Draganmkd       2012-05-29 11:50:16
MCTales 2012-05-29 23:43:51
Tyrogthekreeper 2012-05-30 09:20:44
JmKissme        2012-05-30 16:41:03
Soulboost       2012-05-31 00:15:19
Gbawden 2012-05-31 13:06:14
Meagles111      2012-06-01 03:33:38
EllsworthSK     2012-06-01 20:39:15
SecretStoryStyle        2012-06-01 22:15:47
MaxPayne888     2012-06-03 03:04:23
Healy10 2012-06-03 14:20:28
Mhygelle        2012-06-03 18:12:41
Sorie   2012-06-03 22:32:59
Preacher lad    2012-06-04 08:25:58
Dark Silver Crow        2012-06-04 15:49:10
Zhinz   2012-06-05 02:35:28
Karin127        2012-06-05 10:17:46
Hornymanatee    2012-06-06 10:31:02
FoCuSandLeArN   2012-06-06 15:13:28
Anarchangel     2012-06-06 23:33:58
Anir1uph        2012-06-07 17:26:18
M samadi        2012-06-08 11:00:34
Spparky 2012-06-08 13:42:23
Writerchic99    2012-06-08 21:35:45
Rudyindarto     2012-06-09 07:17:40
Bar Code Symmetry       2012-06-09 15:49:14
Tchaliburton    2012-06-09 22:30:18
Loupiotte       2012-06-10 00:47:59
Solarra 2012-06-10 16:03:34
Samil20 2012-06-11 05:41:41
Lovysinghal     2012-06-12 08:59:57
Treki   2012-06-12 13:25:16


Below is the list of test group editors and the timestamp of their 1K event:

List of 1K Users - Test (1K timestamp)

3210    2012-05-15 02:24:56
Werothegreat    2012-05-15 15:33:56
Tholme  2012-05-15 20:31:59
Yutsi   2012-05-15 22:06:04
Andy4789        2012-05-17 21:00:10
Redhanker       2012-05-18 07:16:18
Nikpapag        2012-05-18 12:29:32
Yachty4000      2012-05-18 20:18:23
Anderch 2012-05-19 09:57:31
BuzyBody        2012-05-20 00:17:06
Thepidding      2012-05-20 03:42:15
Danrok  2012-05-20 13:54:27
Rui78901        2012-05-20 21:01:46
Peetlesnumber1  2012-05-20 21:35:15
Riley Huntley   2012-05-21 03:35:55
Alborzagros     2012-05-21 12:27:12
Yahnatan        2012-05-22 00:38:41
Jonadin93       2012-05-22 02:58:11
Jprg1966        2012-05-22 14:51:18
Editfromwithout 2012-05-22 22:41:12
Coal town guy   2012-05-23 14:05:36
Sir Wrestler    2012-05-23 23:07:22
Treyvo  2012-05-24 04:20:56
JZCL    2012-05-24 16:43:38
Idiotchalk      2012-05-24 17:18:49
JeanneMish      2012-05-25 13:23:52
Entity of the Void      2012-05-25 23:40:58
EricEnfermero   2012-05-26 02:55:56
Xtian06 2012-05-26 12:23:46
Gylatshalit     2012-05-26 19:19:51
EagerToddler39  2012-05-27 03:53:25
Devite  2012-05-27 15:56:02
Ericleb01       2012-05-28 01:14:50
Jpaulm  2012-05-28 22:45:17
Jonathan A Jones        2012-05-29 07:15:54
Phileasson      2012-05-29 11:59:18
Dorje108        2012-05-30 02:56:49
Welshsocialist  2012-05-30 22:21:45
Tthaas  2012-05-31 00:16:20
TheSpecialUser  2012-05-31 16:35:17
Ashok2102       2012-05-31 16:49:22
David1217       2012-06-01 03:14:30
DeXXus  2012-06-01 07:56:18
Jayven09maddie  2012-06-01 18:28:31
Russ Jericho    2012-06-01 20:43:16
Witger  2012-06-02 09:29:45
Rytyho usa      2012-06-02 22:42:41
Narcis90        2012-06-03 13:15:09
SolomanMcKenzie 2012-06-03 16:29:07
Cloptonson      2012-06-03 20:24:21
Fylbecatulous   2012-06-04 02:08:04
TuneyLoon       2012-06-04 20:17:31
Jfd34   2012-06-05 03:31:44
RJR3333 2012-06-05 21:31:08
Xxhopingtearsxx 2012-06-06 22:14:34
Boseritwik      2012-06-07 05:35:46
Total-MAdMaN    2012-06-07 22:41:52
FallenAngelII   2012-06-08 00:12:20
Sidsahu 2012-06-08 12:50:07
Carcamagnu      2012-06-09 06:41:29
Run34   2012-06-09 20:07:35
CodeCat 2012-06-09 22:57:26
TheTimesAreAChanging    2012-06-10 22:29:30
VarietyPerson   2012-06-11 14:24:18
Bearpatch       2012-06-12 00:47:29


Below is the list of test group editors and the timestamp of their Barnstar event:

List of 1K Users - Test (1K timestamp)

Werothegreat    2012-05-16 16:33:00
3210    2012-05-16 17:01:00
Yutsi   2012-05-16 17:08:00
Tholme  2012-05-16 18:00:00
Andy4789        2012-05-18 18:07:00
Redhanker       2012-05-18 18:10:00
Nikpapag        2012-05-18 18:17:00
Alborzagros     2012-05-21 19:15:00
Riley Huntley   2012-05-21 05:40:00
Peetlesnumber1  2012-05-21 19:28:00
Rui78901        2012-05-21 19:59:00
Danrok  2012-05-21 20:02:00
Thepidding      2012-05-21 20:05:00
BuzyBody        2012-05-21 20:10:00
Anderch 2012-05-21 20:15:00
Yachty4000      2012-05-21 20:18:00
Yahnatan        2012-05-22 15:58:00
Jonadin93       2012-05-22 16:02:00
Jprg1966        2012-05-22 06:08:00
Idiotchalk      2012-05-24 22:30:00
JZCL    2012-05-24 22:32:00
Treyvo  2012-05-24 22:35:00
Sir Wrestler    2012-05-24 22:37:00
Coal town guy   2012-05-24 22:41:00
Editfromwithout 2012-05-24 22:44:00
Tow     2012-05-30 18:46:00
Cyan Gardevoir  2012-05-30 18:48:00
JeanneMish      2012-05-30 18:53:00
Entity of the Void      2012-05-30 19:02:00
EricEnfermero   2012-05-30 19:06:00
Xtian06 2012-05-30 19:19:00
Gylatshalit     2012-05-30 19:22:00
EagerToddler39  2012-05-30 19:58:00
Devite  2012-05-30 20:18:00
Ericleb01       2012-05-30 20:23:00
Jpaulm  2012-05-30 20:49:00
Jonathan A Jones        2012-05-30 23:19:00
Phileasson      2012-05-30 23:41:00
Dorje108        2012-05-30 23:44:00
Welshsocialist  2012-05-31 21:54:00
Tthaas  2012-05-31 22:11:00
TheSpecialUser  2012-05-31 22:15:00
Ashok2102       2012-06-01 22:20:00
David1217       2012-06-01 22:26:00
DeXXus  2012-06-01 22:30:00
Jayven09maddie  2012-06-01 22:34:00
Russ Jericho    2012-06-01 22:43:00
Witger  2012-06-04 19:12:00
Rytyho usa      2012-06-04 19:16:00
Narcis90        2012-06-04 19:19:00
SolomanMcKenzie 2012-06-12 17:45:00
Cloptonson      2012-06-04 19:29:00
Fylbecatulous   2012-06-04 19:48:00
TuneyLoon       2012-06-05 18:50:00
Jfd34   2012-06-05 18:59:00
RJR3333 2012-06-07 18:42:00
Xxhopingtearsxx 2012-06-07 18:48:00
Boseritwik      2012-06-07 18:51:00
Total-MAdMaN    2012-06-08 18:47:00
FallenAngelII   2012-06-08 18:52:00
Sidsahu 2012-06-08 18:55:00
Carcamagnu      2012-06-11 20:03:00
Mogism  2012-06-11 20:06:00
Run34   2012-06-11 20:08:00
CodeCat 2012-06-11 20:13:00
TheTimesAreAChanging    2012-06-11 22:25:00
VarietyPerson   2012-06-11 22:28:00
Bearpatch       2012-06-12 23:50:00


Editor Response: behaviour after event[edit]

These are preliminary results showing a comparison of the revision activity in the 0-3 day period after (1) the test 1K event and control 1K event and (2) the test barnstar event and control 1K event. The results are inconclusive.


T-Test results - Control 1K vs Test 1K

data:  metrics_1K_test_all$revisions_after_0_3 and metrics_1K_control_all$revisions_after_0_3 
t = 0.0575, df = 132.695, p-value = 0.9542
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -44.85078  47.53869 
sample estimates:
mean of x mean of y 
 69.81538  68.47143 


T-Test results - Control 1K vs Test Barnstar

data:  metrics_bStar_test_all$revisions_after_0_3 and metrics_1K_control_all$revisions_after_0_3 
t = -0.4677, df = 121.68, p-value = 0.6409
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -51.10443  31.57334 
sample estimates:
mean of x mean of y 
 58.70588  68.47143 


Editor Response: behaviour before and after event[edit]

In addition to measuring the revision activity of the editors in this experiment in 0-3 day period after the events the revision activity in the 0-3 day period before the event was also measured:


Edit Diff = edits_0-3_days_after - edits_0-3_days_before


Below are the results of the T-tests over this metric, which yielded no significant results between the groups.


T-Test results - Edit Diff - Control 1K vs Test 1K

data:  metrics_1K_test_all$edits_diff and metrics_1K_control_all$edits_diff 
t = 1.2269, df = 101.934, p-value = 0.2227
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -17.85703  75.77571 
sample estimates:
 mean of x  mean of y 
  7.430769 -21.528571 


T-Test results - Edit Diff - Control 1K vs Test Barnstar

data:  metrics_bstar_test_all$edits_diff and metrics_1K_control_all$edits_diff 
t = 0.2081, df = 96.076, p-value = 0.8356
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -41.04367  50.65964 
sample estimates:
mean of x mean of y 
-16.72059 -21.52857 


Long-term editing effect[edit]

The results when observing the 30 day period after the barnstarring yielded the same results as the short-term effect. That is, no significant difference in editing behavior was observed among the two groups.


T-Test results - Edit Diff 30 - Control 1K vs Test Barnstar

        Welch Two Sample t-test

data:  metrics_1K_test_all$edits_30 and metrics_1K_control_all$edits_30 
t = -0.1107, df = 124.506, p-value = 0.912
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -260.2132  232.6440 
sample estimates:
mean of x mean of y 
 421.4154  435.2000 


T-Test results - Edit Diff 30 - Control 1K vs Test Barnstar

        Welch Two Sample t-test

data:  metrics_bstar_test_all$edits_30 and metrics_1K_control_all$edits_30 
t = 0.5423, df = 133.347, p-value = 0.5885
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -217.9296  382.5590 
sample estimates:
mean of x mean of y 
 517.5147  435.2000 

Raw data[edit]

Full data set is available here.

Conclusions[edit]

Being awarded a barnstar for making 1,000 edit to the article namespace does not appear to have a significant effect on a English Wikipedians' short-term or long-term editing activity. More work may be necessary to determine whether a different barnstar metric (e.g., more or less than 1,000 edits, a certain number of edits in a successive editing session, etc.) or a different awarding treatment (e.g., community-awarded barnstars, rather than staff-awarded) produces an effect.

Discussion and Follow-up[edit]

Impact on other experiments

These results may inform post-registration activities we experiment with in our account creation UX work. If editors likely to be very productive in the encyclopedia edit either base on their geographic location, cultural heritage, and hobbies, then suggesting this may be fruitful. It may also tie into the work proposed on structured profiles that can be used to suggest articles to edit and WikiProjects to join.

Other research questions
  1. Previous research efforts looked at dates 87-90 pre-milestone. We should look at the average contributions per user in the 0-3 day period 90 days before the barnstar, to account for any possible seasonal effects.
  2. Review whether the findings from Aaron Shaw's and Mako's study on barnstars holds true. (They found that a burst of activity could usually be observed in editors' contribution histories immediately before receiving a barnstar, followed by a decline in activity.)
  3. Does receiving personalized recognition instill a stronger community sense in new Wikipedians? Analyze userpage creation and edits in the non-article space for test and control groups.
  4. How does the status of the giver affect the outcome of the barnstar? Perform another month-long A/B test, asking members of the community to award barnstars to the test group and continue staff barnstarring as a control.
  5. How has the editing pattern of 1k editors changed in the past year? Pull a list of editors who achieved the 1k milestone 1 year ago and compare to control group. Analyze whether the cohort graph looks the same but shifted back one year (e.g., did most people who reached 1k edits last year begin editing in 2008?)
Other A/B test ideas
  • Test sending the barnstar from a volunteer account rather than a WMF one. (Shaw and Mako's study suggested that the sender of a barnstar matters as much as the barnstar itself, so a thank-you message from another editor may have a different effect than one from a staff member.)
  • Test removing mention of 1,000 edits and only including a personalized thank-you message. (Does priming with edit count create editcountitis, does it lead to more/less contributions than no priming?)
  • Test simple talkpage message vs WikiLove. (Any kind of talk page activity might lead to a short-term bump in contributions, so we should see whether it matters what the message is.)

See also[edit]