Research:Editor milestones

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
Contact
Maryana Pinchuk
Steven Walling
Duration:  2012-04 — 2012-07
VisualEditor - Icon - Check.svg
This page documents a completed research project.


Background and rationale[edit]

A number of studies have suggested that barnstars and other onwiki rewards motivate Wikipedians to contribute. However, due to the size of the editing community on English Wikipedia, many editors' good work goes unnoticed and therefore unrewarded. To learn more about the population of editors who are on the verge of becoming "established" Wikipedians, and to test the effects of personalized recognition on editing behavior, in April 2012 we began tracking and thanking all users who reach their 1,000th edit to articles on English Wikipedia.

Methodology[edit]

Using a slave database of the English Wikipedia, we began logging the number of times that users on English Wikipedia made their 1,000th edit to the article namespace (including deleted revisions). We then built an analytics dashboard that automatically updates with the username and timestamp of each 1,000 edit event.

We chose 1,000 edits somewhat arbitrarily, with the rationale that...

  • it is a significant contribution to the encyclopedia, and therefore inherently praiseworthy
  • it is highly unlikely to be a milestone reached by vandals, spammers, or other overtly disruptive editors
  • it suggests a possible "power-user" or very highly active Wikipedian in the making

Starting on April 5, 2012, we began thanking each person who reached this milestone with a personalized barnstar, using WikiLove.

First iteration[edit]

1K editors by year of first edit, out of a sample of 158

Currently, about 5-6 Wikipedians cross the 1,000-edit threshold each day. In our first month of performing this experiment, we thanked about 160 editors and analyzed their contribution history to find when they began editing and what subjects they edited most frequently. We then analyzed their editing activity in the 3-day period after receiving a barnstar for reaching their 1,000-edit milestone.

Age of editors (yearly cohorts)[edit]

In addition to what they were currently editing, we also looked at what year these Wikipedians made their first contributions. What we found was that these editors roughly fell into two camps: slow and steady editors who began contributing five or six years ago and accumulated their 1,000 edits over a long period of time, and new users who were among the most active Wikipedians in their first few months of editing and quickly ratcheted up their 1,000 edits.

As demonstrated by the chart to the right, the new Wikipedians outnumber the old – 59% made their first edit in 2009 or later, out of 158 editors to date.

Editing patterns - Pre-Barnstar[edit]

Primary topics and editing activity of current 1K editors

In order to write a personal thank-you message to each editor, we looked at their user pages and contribution histories to determine what subjects they had been most actively working on. What we found was that many of these Wikipedians were intensely focused on one topic or editing activity. The chart on the right illustrates the most common topics and the percentage of users whose edits mostly fall into each topic. These topics were:

  • Sports (e.g., football teams, car races, rugby players)
  • Music (.e.g., songs, albums, composers)
  • Milhist (e.g., battles, military technology)
  • Aviation (e.g., airports, air lines)
  • Video games
  • TV/film/radio
  • Bios (e.g., biographies of people, living or dead, not necessarily connected by any particular topic)
  • Local culture (e.g., notable places, events, and cultural phenomena in the city/country where the editor is from)
  • World culture (e.g., notable places, events, and cultural phenomena in different parts of the world, not where the editor is from)
  • Math/science (e.g., theorems, astronomy)
  • Technical gnoming (e.g., copyediting, disambiguating/repairing links, adding navboxes)
  • Lit/comics (e.g., novels, manga, comic books)
  • Vandal-fighting/patrolling
  • Other/varied (e.g., a narrow topic not covered above, such as solar power, or diverse editing activity in many subject areas)

Somewhat surprisingly, by far the most common topic of editing for this sample of users was local culture. Whether they self-identified as being from Turkey, Bosnia-Herzegovina, China, the U.S., or the Philippines, these editors tended to create and/or improve articles about their home town, state, or country.

The other common area of interest for this sample of users was popular culture – in the form of television, film, sports, and music. It is also interesting to note that some of the topics covered by the more active and successful WikiProjects, such as military history, aviation, and biology, were either very rare or not found at all in this sample.

Editing Patterns - Post-Barnstar[edit]

  • Mean # of edits in 3-day period after 1k event per user: 49.85
  • Mean # of blocks in 3-day period after 1k event per user: 0.01 (1%)
  • Mean # of warnings in 3-day period after 1k event per user: 0.24 (24%; however, a few users were warned several times)

From the low rate of blocks and warnings and the relatively large number of edits in the 48 hours after these editors are given a barnstar, we can assume this class of editors continues to be of high quality. However, in order to determine whether a barnstar was the cause of an increase or decrease in activity, a control group must be developed.

Results[edit]

Here's a supplemental analysis comparing the prevalence of edits over three-day periods, both before and immediately after the barnstar was delivered:

http://meta.wikimedia.org/wiki/User:Renklauf/Barnstarred_Editors_Analyses

Second iteration[edit]

Because we are finding patterns in editor behavior that may suggest that thanking editors (via barnstar) for reaching milestones may have a significant impact on their editing, we will iterate on this research by randomly assigning users to a control (no barnstar) and test (barnstar) group. We will then analyze their contributions to determine the following:

Short-term editing effect[edit]

Control group
  • Average number of contributions per user in the 0-3 day period before 1k edit timestamp
  • Average number of contributions per user in the 0-3 day period after 1k edit timestamp
  • Averages for new editors (2009-2012) vs old editors (2002-2008)
Test group
  • Average number of contributions per user in the 0-3 day period before barnstar timestamp
  • Average number of contributions per user in the 0-3 day period after barnstar timestamp
  • Average number of contributions per user in the 0-3 day period before 1k edit timestamp
  • Average number of contributions per user in the 0-3 day period after 1k edit timestamp (previous month's cohort was 49.85)
  • Averages for new editors (2009-2012) vs old editors (2002-2008)

Long-term editing effect[edit]

Control group
  • Average number of contributions per user in the 0-30 day period before 1k edit timestamp
  • Average number of contributions per user in the 0-30 day period after 1k edit timestamp
Test group
  • Average number of contributions per user in the 0-30 day period before barnstar timestamp
  • Average number of contributions per user in the 0-30 day period after barnstar timestamp
  • Average number of contributions per user in the 0-30 day period before 1k edit timestamp
  • Average number of contributions per user in the 0-30 day period after 1k edit timestamp

Results[edit]

Short-term editing effect[edit]

This experiment concerns the measurement of revision activity in all namespaces in the context of the following events:

  • 1K Event - the time at which an editor reaches their 1000th edit
  • Barnstar Event - the time at which the 1K editor received a Barnstar from WMF recognizing their accomplishment of 1K edits


Below is the list of control users and the timestamp of their 1K event:

List of 1K Users - Control (1K timestamp)

ViriiK	2012-05-15 02:58:25
Treydavis3	2012-05-15 06:00:16
Listmeister	2012-05-15 21:12:27
Mike Schwartz	2012-05-16 06:58:17
Horhey420	2012-05-17 19:15:05
Adjwilley	2012-05-18 04:58:54
Loginnigol	2012-05-18 11:39:58
Olaff	2012-05-18 16:47:09
Shanghainese.ua	2012-05-18 20:13:15
Smokeyfire	2012-05-19 00:53:09
Wikimace08	2012-05-19 21:35:27
Acsian88	2012-05-20 02:31:29
Jackl	2012-05-20 08:12:00
DonBarton	2012-05-20 19:57:03
Abh87	2012-05-20 21:03:52
AzseicsoK	2012-05-21 01:20:31
Bazuz	2012-05-21 12:08:57
Inscription	2012-05-21 16:41:46
Ttenchantr	2012-05-21 18:32:56
Dzlinker	2012-05-21 22:48:32
Kwesiidun91	2012-05-22 02:30:50
Dougsim	2012-05-22 12:39:56
Gwen-chan	2012-05-22 15:37:50
Room429	2012-05-22 18:01:19
Ekin	2012-05-23 12:09:20
JoelWhy	2012-05-23 22:00:59
M0rphzone	2012-05-24 01:46:09
KarlcmanRockets	2012-05-24 11:48:10
Ich	2012-05-24 16:52:41
Tyrannus Mundi	2012-05-25 18:40:27
Callanecc	2012-05-26 02:21:46
BrightStarSky	2012-05-26 05:22:40
Allkayloh	2012-05-26 13:46:11
Paulika1995	2012-05-27 11:43:46
Trafalk09	2012-05-27 20:52:31
TRLIJC19	2012-05-28 04:17:44
PaulWicks	2012-05-28 12:34:14
Beefyt	2012-05-29 06:02:32
Draganmkd	2012-05-29 11:50:16
MCTales	2012-05-29 23:43:51
Tyrogthekreeper	2012-05-30 09:20:44
JmKissme	2012-05-30 16:41:03
Soulboost	2012-05-31 00:15:19
Gbawden	2012-05-31 13:06:14
Meagles111	2012-06-01 03:33:38
EllsworthSK	2012-06-01 20:39:15
SecretStoryStyle	2012-06-01 22:15:47
MaxPayne888	2012-06-03 03:04:23
Healy10	2012-06-03 14:20:28
Mhygelle	2012-06-03 18:12:41
Sorie	2012-06-03 22:32:59
Preacher lad	2012-06-04 08:25:58
Dark Silver Crow	2012-06-04 15:49:10
Zhinz	2012-06-05 02:35:28
Karin127	2012-06-05 10:17:46
Hornymanatee	2012-06-06 10:31:02
FoCuSandLeArN	2012-06-06 15:13:28
Anarchangel	2012-06-06 23:33:58
Anir1uph	2012-06-07 17:26:18
M samadi	2012-06-08 11:00:34
Spparky	2012-06-08 13:42:23
Writerchic99	2012-06-08 21:35:45
Rudyindarto	2012-06-09 07:17:40
Bar Code Symmetry	2012-06-09 15:49:14
Tchaliburton	2012-06-09 22:30:18
Loupiotte	2012-06-10 00:47:59
Solarra	2012-06-10 16:03:34
Samil20	2012-06-11 05:41:41
Lovysinghal	2012-06-12 08:59:57
Treki	2012-06-12 13:25:16


Below is the list of test group editors and the timestamp of their 1K event:

List of 1K Users - Test (1K timestamp)

3210	2012-05-15 02:24:56
Werothegreat	2012-05-15 15:33:56
Tholme	2012-05-15 20:31:59
Yutsi	2012-05-15 22:06:04
Andy4789	2012-05-17 21:00:10
Redhanker	2012-05-18 07:16:18
Nikpapag	2012-05-18 12:29:32
Yachty4000	2012-05-18 20:18:23
Anderch	2012-05-19 09:57:31
BuzyBody	2012-05-20 00:17:06
Thepidding	2012-05-20 03:42:15
Danrok	2012-05-20 13:54:27
Rui78901	2012-05-20 21:01:46
Peetlesnumber1	2012-05-20 21:35:15
Riley Huntley	2012-05-21 03:35:55
Alborzagros	2012-05-21 12:27:12
Yahnatan	2012-05-22 00:38:41
Jonadin93	2012-05-22 02:58:11
Jprg1966	2012-05-22 14:51:18
Editfromwithout	2012-05-22 22:41:12
Coal town guy	2012-05-23 14:05:36
Sir Wrestler	2012-05-23 23:07:22
Treyvo	2012-05-24 04:20:56
JZCL	2012-05-24 16:43:38
Idiotchalk	2012-05-24 17:18:49
JeanneMish	2012-05-25 13:23:52
Entity of the Void	2012-05-25 23:40:58
EricEnfermero	2012-05-26 02:55:56
Xtian06	2012-05-26 12:23:46
Gylatshalit	2012-05-26 19:19:51
EagerToddler39	2012-05-27 03:53:25
Devite	2012-05-27 15:56:02
Ericleb01	2012-05-28 01:14:50
Jpaulm	2012-05-28 22:45:17
Jonathan A Jones	2012-05-29 07:15:54
Phileasson	2012-05-29 11:59:18
Dorje108	2012-05-30 02:56:49
Welshsocialist	2012-05-30 22:21:45
Tthaas	2012-05-31 00:16:20
TheSpecialUser	2012-05-31 16:35:17
Ashok2102	2012-05-31 16:49:22
David1217	2012-06-01 03:14:30
DeXXus	2012-06-01 07:56:18
Jayven09maddie	2012-06-01 18:28:31
Russ Jericho	2012-06-01 20:43:16
Witger	2012-06-02 09:29:45
Rytyho usa	2012-06-02 22:42:41
Narcis90	2012-06-03 13:15:09
SolomanMcKenzie	2012-06-03 16:29:07
Cloptonson	2012-06-03 20:24:21
Fylbecatulous	2012-06-04 02:08:04
TuneyLoon	2012-06-04 20:17:31
Jfd34	2012-06-05 03:31:44
RJR3333	2012-06-05 21:31:08
Xxhopingtearsxx	2012-06-06 22:14:34
Boseritwik	2012-06-07 05:35:46
Total-MAdMaN	2012-06-07 22:41:52
FallenAngelII	2012-06-08 00:12:20
Sidsahu	2012-06-08 12:50:07
Carcamagnu	2012-06-09 06:41:29
Run34	2012-06-09 20:07:35
CodeCat	2012-06-09 22:57:26
TheTimesAreAChanging	2012-06-10 22:29:30
VarietyPerson	2012-06-11 14:24:18
Bearpatch	2012-06-12 00:47:29


Below is the list of test group editors and the timestamp of their Barnstar event:

List of 1K Users - Test (1K timestamp)

Werothegreat	2012-05-16 16:33:00
3210	2012-05-16 17:01:00
Yutsi	2012-05-16 17:08:00
Tholme	2012-05-16 18:00:00
Andy4789	2012-05-18 18:07:00
Redhanker	2012-05-18 18:10:00
Nikpapag	2012-05-18 18:17:00
Alborzagros	2012-05-21 19:15:00
Riley Huntley	2012-05-21 05:40:00
Peetlesnumber1	2012-05-21 19:28:00
Rui78901	2012-05-21 19:59:00
Danrok	2012-05-21 20:02:00
Thepidding	2012-05-21 20:05:00
BuzyBody	2012-05-21 20:10:00
Anderch	2012-05-21 20:15:00
Yachty4000	2012-05-21 20:18:00
Yahnatan	2012-05-22 15:58:00
Jonadin93	2012-05-22 16:02:00
Jprg1966	2012-05-22 06:08:00
Idiotchalk	2012-05-24 22:30:00
JZCL	2012-05-24 22:32:00
Treyvo	2012-05-24 22:35:00
Sir Wrestler	2012-05-24 22:37:00
Coal town guy	2012-05-24 22:41:00
Editfromwithout	2012-05-24 22:44:00
Tow	2012-05-30 18:46:00
Cyan Gardevoir	2012-05-30 18:48:00
JeanneMish	2012-05-30 18:53:00
Entity of the Void	2012-05-30 19:02:00
EricEnfermero	2012-05-30 19:06:00
Xtian06	2012-05-30 19:19:00
Gylatshalit	2012-05-30 19:22:00
EagerToddler39	2012-05-30 19:58:00
Devite	2012-05-30 20:18:00
Ericleb01	2012-05-30 20:23:00
Jpaulm	2012-05-30 20:49:00
Jonathan A Jones	2012-05-30 23:19:00
Phileasson	2012-05-30 23:41:00
Dorje108	2012-05-30 23:44:00
Welshsocialist	2012-05-31 21:54:00
Tthaas	2012-05-31 22:11:00
TheSpecialUser	2012-05-31 22:15:00
Ashok2102	2012-06-01 22:20:00
David1217	2012-06-01 22:26:00
DeXXus	2012-06-01 22:30:00
Jayven09maddie	2012-06-01 22:34:00
Russ Jericho	2012-06-01 22:43:00
Witger	2012-06-04 19:12:00
Rytyho usa	2012-06-04 19:16:00
Narcis90	2012-06-04 19:19:00
SolomanMcKenzie	2012-06-12 17:45:00
Cloptonson	2012-06-04 19:29:00
Fylbecatulous	2012-06-04 19:48:00
TuneyLoon	2012-06-05 18:50:00
Jfd34	2012-06-05 18:59:00
RJR3333	2012-06-07 18:42:00
Xxhopingtearsxx	2012-06-07 18:48:00
Boseritwik	2012-06-07 18:51:00
Total-MAdMaN	2012-06-08 18:47:00
FallenAngelII	2012-06-08 18:52:00
Sidsahu	2012-06-08 18:55:00
Carcamagnu	2012-06-11 20:03:00
Mogism	2012-06-11 20:06:00
Run34	2012-06-11 20:08:00
CodeCat	2012-06-11 20:13:00
TheTimesAreAChanging	2012-06-11 22:25:00
VarietyPerson	2012-06-11 22:28:00
Bearpatch	2012-06-12 23:50:00


Editor Response: behaviour after event[edit]

These are preliminary results showing a comparison of the revision activity in the 0-3 day period after (1) the test 1K event and control 1K event and (2) the test barnstar event and control 1K event. The results are inconclusive.


T-Test results - Control 1K vs Test 1K

data:  metrics_1K_test_all$revisions_after_0_3 and metrics_1K_control_all$revisions_after_0_3 
t = 0.0575, df = 132.695, p-value = 0.9542
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -44.85078  47.53869 
sample estimates:
mean of x mean of y 
 69.81538  68.47143 


T-Test results - Control 1K vs Test Barnstar

data:  metrics_bStar_test_all$revisions_after_0_3 and metrics_1K_control_all$revisions_after_0_3 
t = -0.4677, df = 121.68, p-value = 0.6409
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -51.10443  31.57334 
sample estimates:
mean of x mean of y 
 58.70588  68.47143 


Editor Response: behaviour before and after event[edit]

In addition to measuring the revision activity of the editors in this experiment in 0-3 day period after the events the revision activity in the 0-3 day period before the event was also measured:


Edit Diff = edits_0-3_days_after - edits_0-3_days_before


Below are the results of the T-tests over this metric, which yielded no significant results between the groups.


T-Test results - Edit Diff - Control 1K vs Test 1K

data:  metrics_1K_test_all$edits_diff and metrics_1K_control_all$edits_diff 
t = 1.2269, df = 101.934, p-value = 0.2227
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -17.85703  75.77571 
sample estimates:
 mean of x  mean of y 
  7.430769 -21.528571 


T-Test results - Edit Diff - Control 1K vs Test Barnstar

data:  metrics_bstar_test_all$edits_diff and metrics_1K_control_all$edits_diff 
t = 0.2081, df = 96.076, p-value = 0.8356
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -41.04367  50.65964 
sample estimates:
mean of x mean of y 
-16.72059 -21.52857 


Long-term editing effect[edit]

The results when observing the 30 day period after the barnstarring yielded the same results as the short-term effect. That is, no significant difference in editing behavior was observed among the two groups.


T-Test results - Edit Diff 30 - Control 1K vs Test Barnstar

	Welch Two Sample t-test

data:  metrics_1K_test_all$edits_30 and metrics_1K_control_all$edits_30 
t = -0.1107, df = 124.506, p-value = 0.912
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -260.2132  232.6440 
sample estimates:
mean of x mean of y 
 421.4154  435.2000 


T-Test results - Edit Diff 30 - Control 1K vs Test Barnstar

	Welch Two Sample t-test

data:  metrics_bstar_test_all$edits_30 and metrics_1K_control_all$edits_30 
t = 0.5423, df = 133.347, p-value = 0.5885
alternative hypothesis: true difference in means is not equal to 0 
95 percent confidence interval:
 -217.9296  382.5590 
sample estimates:
mean of x mean of y 
 517.5147  435.2000 

Raw data[edit]

Full data set is available here.

Conclusions[edit]

Being awarded a barnstar for making 1,000 edit to the article namespace does not appear to have a significant effect on a English Wikipedians' short-term or long-term editing activity. More work may be necessary to determine whether a different barnstar metric (e.g., more or less than 1,000 edits, a certain number of edits in a successive editing session, etc.) or a different awarding treatment (e.g., community-awarded barnstars, rather than staff-awarded) produces an effect.

Discussion and Follow-up[edit]

Impact on other experiments

These results may inform post-registration activities we experiment with in our account creation UX work. If editors likely to be very productive in the encyclopedia edit either base on their geographic location, cultural heritage, and hobbies, then suggesting this may be fruitful. It may also tie into the work proposed on structured profiles that can be used to suggest articles to edit and WikiProjects to join.

Other research questions
  1. Previous research efforts looked at dates 87-90 pre-milestone. We should look at the average contributions per user in the 0-3 day period 90 days before the barnstar, to account for any possible seasonal effects.
  2. Review whether the findings from Aaron Shaw's and Mako's study on barnstars holds true. (They found that a burst of activity could usually be observed in editors' contribution histories immediately before receiving a barnstar, followed by a decline in activity.)
  3. Does receiving personalized recognition instill a stronger community sense in new Wikipedians? Analyze userpage creation and edits in the non-article space for test and control groups.
  4. How does the status of the giver affect the outcome of the barnstar? Perform another month-long A/B test, asking members of the community to award barnstars to the test group and continue staff barnstarring as a control.
  5. How has the editing pattern of 1k editors changed in the past year? Pull a list of editors who achieved the 1k milestone 1 year ago and compare to control group. Analyze whether the cohort graph looks the same but shifted back one year (e.g., did most people who reached 1k edits last year begin editing in 2008?)
Other A/B test ideas
  • Test sending the barnstar from a volunteer account rather than a WMF one. (Shaw and Mako's study suggested that the sender of a barnstar matters as much as the barnstar itself, so a thank-you message from another editor may have a different effect than one from a staff member.)
  • Test removing mention of 1,000 edits and only including a personalized thank-you message. (Does priming with edit count create editcountitis, does it lead to more/less contributions than no priming?)
  • Test simple talkpage message vs WikiLove. (Any kind of talk page activity might lead to a short-term bump in contributions, so we should see whether it matters what the message is.)

See also[edit]