Research:Autoconfirmed article creation trial

From Meta, a Wikimedia project coordination wiki
Jump to: navigation, search
Light-Bulb icon by Till Teenck.svg

This page documents a proposed research project.
Information may be incomplete and may change before the project starts.

The goal of this study is to run an experiment on English Wikipedia where we examine the effects of disabling article creation for non-autoconfirmed newly registered editors.

Contents

Research Questions[edit]

We are interested in understanding the effects this change in permissions will have on newly registered accounts, the English Wikipedia's quality assurance processes (in particular New pages patrol), and the quality of Wikipedia's articles. These three main themes are also reflected in the following three research questions:

RQ-New accounts
How does requiring autoconfirmed status to create new articles affect newly registered accounts?
RQ-Quality Assurance
How does requiring autoconfirmed status affect Wikipedia's quality assurance processes?
RQ-Content quality
How does requiring autoconfirmed status affect the quality of Wikipedia's articles?

Hypotheses[edit]

RQ-New accounts[edit]

H1: Number of accounts registered per day will not be affected.[edit]

Because the system is designed so that restrictions and limitations are communicated and placed at the time of action, rather than described up front, we should see little to no change in the number of accounts that are registered.

H2: Proportion of newly registered accounts with non-zero edits in the first 30 days is reduced.[edit]

Some proportion of new accounts start out by creating a new article, a path that will no longer be available. Instead, they are required to make other types of edits to reach autoconfirmed status if they wish to create articles. This can be regarded as an increased barrier to entry similar to those studied by Drenner et al,[1] who found that requiring more effort from new users resulted in a lower completion rate. We expect some of the newly registered accounts to be "single purpose accounts", ones who are registered only in order to be able to create an article. Because these accounts are unable to create articles, we expect those to no longer make any edits. In line with Drenner et al's work, we expect some of the remaining accounts to not continue their work and leave, which will also contribute to a reduction in the proportion of accounts with non-zero edits.

H3: Proportion of accounts reaching autoconfirmed status within the first 30 days since account creation is unchanged.[edit]

We have so far not found any existing research on the proportion of users who reach autoconfirmed status within a specific time period. Studies on Wikipedia contributors that mention autoconfirmed status have looked at for example spam attack vectors[2] and page protection.[3] Two studies of roles in Wikipedia[4][5] did not distinguish between accounts with and without autoconfirmed status, instead combining both into a "new user" group.

There is presently no requirement to make a certain number of edits/contributions in order to unlock privileges on the English Wikipedia, and the site is not gamified in the way that, for example, Stack Overflow is.[6] We do not know to what extent newly registered contributors seek to unlock these privileges. If a large proportion of newly registered accounts make edits that are subsequently deleted (e.g. because they created a non-encyclopedic article) and that this behaviour leads to their account not reaching autoconfirmed status, it means that those who reach said status can be regarded as productive members of the community. Secondly, H2 proposes that the proportion of accounts with non-zero edits is reduced, suggesting that these newly registered accounts will never reach autoconfirmed status. In combination, there are reasons to hypothesize that the proportion of accounts reaching autoconfirmed status within 30 days of registering will be unchanged.

H4: The median time to reach autoconfirmed status within the first 30 days is unchanged.[edit]

Similarly, we hypothesise that the time it will take for a newly registered account to reach autoconfirmed status does not change.

H5: The proportion of surviving new editors who make an edit in their fifth week is unchanged.[edit]

When it comes to measuring editor retention, the WMF research staff has at least two definitions: Returning new editor, and Surviving new editor. The difference between the two is what they measure (edit sessions and edits, respectively), and what kind of timespan is defined. Given the other kinds of measures and hypothesis we have the measure of "returning new editor" is most likely covered elsewhere, meaning we are mainly concerned about surviving new editors. Due to the time constraints of the trial, we propose measuring retention by measuring surviving new editors who make at least one edit in the first week, who also make at least one edit in the fifth week (as that week crosses the 30 day threshold after registration). From the previous hypotheses, it follows that the proportion of surviving new editors should not change.

Further segmentation[edit]

We would like to analyze surviving new editors further by segmenting this population into those who start out by creating a new article, and those who start out by editing existing articles. When looking at historical data it would also be useful to split the former group depending on whether their article survives or not. Taken together, these measures should tell us more about the extent to which newly registered accounts stick around, and what the effect of new article creation is on contributions and retention.

H6: The diversity of participation done by accounts that reach autoconfirmed status in the first 30 days is unchanged.[edit]

There are no restrictions on where to make the edits needed to reach autoconfirmed status. A newly registered account might make five copy edits to articles, a couple of edits to create a user page to describe themselves, and some edits to discussion pages (e.g. the Teahouse) in order to reach that limit, while another account might make ten articles edits. Earlier we did argue that the newly registered accounts that reach autoconfirmed status will not change significantly, meaning that the diversity (e.g. number of different pages edited) of work done by these accounts will also not change.

H7: The average number of edits in the first 30 days since registering is reduced.[edit]

We hypothesized that the proportion of accounts with non-zero number of edits will be reduced because some of these accounts will be abandoned. This will likely also affect the average number of edits made in the first 30 days since registering. Note that we in this case will also count edits to pages that are subsequently deleted (e.g. non-encyclopedic articles).

H8: The number of requests for "confirmed" status is increased.[edit]

The English Wikipedia has a page where users can request "confirmed" status: Requests for permissions/Confirmed A newly registered account might want to go there to request said status in order to not have to meet the requirements for autoconfirmed status. At the same time, someone who is new to Wikipedia might not know that this possibility exists. We hypothesized earlier that some newly registered accounts have a single purpose, and we might expect some proportion of these to seek out other venues than reaching autoconfirmed status, particularly if the venue is perceived as low cost. This can lead to a small but significant increase in the number of requests for "confirmed" status.

RQ-Quality Assurance[edit]

H9: Number of patrol actions will decrease.[edit]

The number of articles created per day will be reduced since non-autoconfirmed users can no longer create them. This leads to a reduction in the influx of articles into the New Page Patrol queue, and a subsequent reduction in the number of patrol actions.

Related measure[edit]

The ratio of patrol actions to created articles is unchanged. In other words, patrollers notice that there is less work to do and adjust their efforts accordingly.

H10: Number of active patrollers will decrease.[edit]

With a lower influx of articles there is less work to do, leading some patrollers to stop patrolling. Given the lower workload, it is unlikely that new patrollers will be recruited.

Related measure[edit]

The ratio of active patrollers to created articles is unchanged. This follows logically from H10.

H11: The distribution of patrolling activity evens out.[edit]

Because of the reduction in the number of articles that enters the patrol queue, there is less need for the most active patrollers to do as much "heavy lifting". This in turn means the patrol work is distributed more evenly across the active patrollers.

H12: The number of Time-Consuming Judgement Calls will decrease.[edit]

The analysis of New Pages Patrol by the Wikimedia Foundation describes the job of patrolling certain types of newly created articles as "Time-Consuming Judgement Calls" (TCJC). These articles are characterized as follows:

  • They're probably notable but badly written
  • They're well-written but have questionable notability
  • They're a weird mix of both (because life is complicated)

From the WMF's analysis we know that 15% of the articles that are not quickly patrolled are created by non-autoconfirmed users. Because these users are no longer able to create new articles, we can hypothesize that the number of TCJCs will be reduced. We expect the influx of TCJCs created by autoconfirmed users to remain stable.

H13: The size of the backlog of articles in the New Page Patrol queue will remain stable.[edit]

It is not certain whether the the reduction in the workload of New Page Patrollers will free up resources to go through and process the backlog of articles that still need patrolling, or if the patrollers will focus on other areas of the encyclopedia. Because we hypothesized that the number of patrol actions and patrollers would decrease in H9 and H10, also relative to the number of created articles, it would follow that the size of the backlog remains stable.

H14: The survival rate of newly created articles by autoconfirmed users will remain stable.[edit]

When it comes to article survival, we are obviously only concerned with the survival of articles created by autoconfirmed users. As we earlier hypothesized that these types of users will continue to be productive members of the community, we also hypothesize that the survival rate of their articles will remain stable. We adapt the definition of survival used by Schneider et al,[7] meaning an article survives if it is not deleted during the first 30 days of its life.

H15: The rate of article growth will be reduced.[edit]

The previously mentioned statistics on article creation and survival[8][9] indicated that around 50–100 surviving articles are created each day by non-autoconfirmed users. As before, we have hypothesized that it is unlikely that newly registered accounts will do the work necessary to reach autoconfirmed status, meaning that these articles will no longer be created and reduce the rate of article growth on the English Wikipedia.

Related measures[edit]

We are also interested in understanding more about how articles come to be. To what extent are they created as drafts in the user namespace and then moved into the article namespace? Secondly, are creations and moves done by recently autoconfirmed accounts? The proportion of articles created as moves might increase due to it becoming an alternative approach to article creation. Since we hypothesized that the proportion of accounts reaching autoconfirmed status in 30 days will remain unchanged, we would also hypothesize that creations and moves by recently autoconfirmed accounts would remain stable.

H16: The rate of new submissions at AfC will increase.[edit]

We expect some proportion of newly registered accounts to make a failed attempt at creating an article, and upon doing so will choose to use the Article Wizard instead, meaning their proposed article will be added to the review queue at Articles for Creation (AfC). This should result in an increase in the rate of proposed articles reaching AfC.

H17: The backlog of articles in the AfC queue will increase faster than expected.[edit]

The process of handling AfC submissions is similar to NPP in that it requires human intervention. Since we proposed that the NPP backlog will decrease due to the lower rate of influx of articles there, we must similarly propose that the backlog of articles at AfC will increase.

H18: The reasons for deleting articles will remain stable.[edit]

When it comes to reasons for why articles get deleted, we again refer to our previous hypothesis that newly created accounts that reach autoconfirmed status will be productive members of the community. This means that article creations will remain stable and the reasons for why articles get deleted will not change significantly.

H19: The reasons for deleting non-article pages will change towards those previously used for deletion of articles created by non-autoconfirmed users.[edit]

When it comes to reasons for why non-article pages get deleted, we expect to see a change as newly registered accounts can create pages outside the main (article) namespace (e.g. AfC as mentioned above, or their user page or sandbox). Some of this content will likely be candidates for speedy deletion due to copyright infringement (G12) or "Blatant misuse of Wikipedia as a web host" (U5). In other words, we expect to see a change in the reasons for deletions of non-article pages.

RQ-Content quality[edit]

H20: The quality of articles entering the NPP queue will increase.[edit]

The quality of articles entering the NPP queue will change due to the removal of articles created by non-autoconfirmed users. Those articles that enter the queue that are originally created by non-autoconfirmed users will have gone through the AfC process, meaning they should be of higher quality than earlier, leading to an increase in the average quality of articles in the NPP queue.

H21: The quality of newly created articles after 30 days will be unchanged.[edit]

Most of the English Wikipedia's articles are lower quality (Stub- and Start-class) with low readership,[10] and over time the articles that are created are increasingly on niche topics.[11] We therefore expect the quality of newly created articles after 30 days to remain unchanged.

H22: The quality of articles entering the AfC queue will be unchanged.[edit]

Little is currently known about the quality of content entering the AfC queue. Research on AfC has instead focused on whether the process leads to greater success in publishing articles or retaining contributors.[7][12] We have hypothesized that the influx of new drafts into AfC will increase somewhat due to newly registered accounts not being able to create articles in the main namespace, but we do not know to what extent those who choose to do so are those who create higher quality content to begin with. Given this uncertainty about the current state of articles entering AfC as well as the creators of these drafts, we hypothesize that the quality of articles entering AfC will not change.

Methods[edit]

Our methods of gathering data makes extensive use of the replicated databases available through for example Wikimedia Toolforge and Quarry. If you are unfamiliar with the schema of these databases you can learn more through the database layout manual. Some of the data is from the Data Lake. We build a pipeline that enables us to easily gather historic data as well as being updated during the trial.

H1: Number of accounts registered[edit]

We utilize the logging table to gather this data because it provides two benefits over the user table: most importantly, it has indexes that allow us to gather data across a specific time span; secondly, we can identify how the account was created. A drawback of this method is that the account registration timestamp in the logging table might differ from the one in the user table, typically by about one second. Because this hypothesis is interested in the number of accounts registered, which is usually on the order of "thousands", an error of a second in the timestamp should affect a small number (ideally zero) of accounts per day, thereby enabling us to ignore it.

There are different types of ways accounts can be created, ref Manual:Log actions. Since account creations started being logged in 2006 there have been five different types in use:

newusers
A new account is created.
create
The "normal" user registration, someone without an account registers one.
create2
A logged-in user creates an account for someone else.
autocreate
An account is automatically created, for example because someone with an account on a different language edition visits another Wikipedia.
byemail
An account is created for someone else by a logged-in user, and the created account gets the password through email.

The Python code for capturing this data is in newaccounts.py.

H2: Accounts with non-zero edits[edit]

In order to determine the proportion of accounts that have made at least one edit in the first 30 days, we need to know two things: the account's identifier and when they registered. We utilize the logging table similarly as we did for H1, because we're interested in data across a given timespan, and record the user ID of the registered account, the time of registration (as logged in the logging table), and how the account was created (per the types listed above).

Once we have a list of accounts registered, we can look their edits up in the revision and archive tables in order to account for edits to deleted pages. Because we can get data for three hypotheses at the same time, our code gathers data for H2, H6, and H7, as that is most efficient. The Python code for this is in registrations.py and first30.py.

H5: Surviving new editors[edit]

A surviving new editor is defined as one who makes an edit in both the first and fifth week after registration. We use the same approach to finding edits as for H2, but alter it so that it uses the appropriate timespans. In order to learn what the actual reduction in proportion of surviving new editors is, we store edit counts in both weeks for accounts that made an edit in the first week. The Python code is found in survival.py.

Data is also gathered about whether an account started out by creating an article. This is done by utilizing the dataset gathered for understanding article creations to answer H9's related measure, as that dataset contains all article creations. We look up a user's first edit (taking deleted pages into account as we did for H2) and check if that edit corresponds to an article creation event. If so, we use the logging table to identify whether the article was deleted within 30 days of the creation event. The relevant Python code is found in articlesurvival.py.

H9: Number of patrol actions[edit]

Since the introduction of the PageTriage extension, patrol actions are logged in the logging table. We use this to count the number of patrol actions using the SQL query below.

This data is gathered using the ReportUpdater tool.

In order to calculate the related measure, we gathered data on article creations from the data lake using a modified version of the query used in T149021]. The modifications involve not removing current redirects because a created article might be replaced by a redirect later, improving the detection of pages created as redirects per Tbayer's notes, and removing pages created by moves because those also create redirects.

We also gather data on articles published through a move from User, Draft, or Wikipedia talk namespaces. This is a two-step process that first finds all revisions where the edit comment indicates a page was moved from either of those three namespaces, and secondly identifies whether the namespace it was moved to was main. The Python code for this is in moves.py.

H10: Number of active patrollers[edit]

We modify the query from Research:New page reviewer impact analysis that counts the number of patrollers so that it counts them by day.

Similarly as for H9, this data is gathered using the ReportUpdater tool. We gathered data on the related measure in the same way as described for H9 above.

H11: Distribution of patroller activity[edit]

We modify the SQL query used in H10 so that it counts the number of patrol actions performed by each patroller on a given day. The Python code for this is in patrollers.py, the SQL query is shown below.

Results[edit]

todo

References[edit]

  1. Sara Drenner, Shilad Sen, and Loren Terveen. 2008. Crafting the initial user experience to achieve community goals. In Proceedings of RecSys. DOI
  2. Andrew G. West, Jian Chang, Krishna Venkatasubramanian, Oleg Sokolsky, and Insup Lee, "Link Spamming Wikipedia for Profit", 8th Annual Collaboration, Electronic Messaging, Anti-Abuse, and Spam Conference, 152–161. September 2011. DOI
  3. Benjamin Mako Hill and Aaron Shaw. 2015. Page Protection: Another Missing Dimension of Wikipedia Research. In Proceedings of OpenSym DOI
  4. Arazy, Ofer, Oded Nov, and Felipe Ortega. "The [Wikipedia] World is Not Flat: on the organizational structure of online production communities." (2014).
  5. Ofer Arazy, Felipe Ortega, Oded Nov, Lisa Yeo, and Adam Balila. 2015. Functional Roles and Career Paths in Wikipedia. In Proceedings of CSCW DOI
  6. Ashton Anderson, Daniel Huttenlocher, Jon Kleinberg, and Jure Leskovec. 2013. Steering user behavior with badges. In Proceedings of WWW DOI
  7. a b Jodi Schneider, Bluma S. Gelley, and Aaron Halfaker. 2014. Accept, decline, postpone: How newcomer productivity is reduced in English Wikipedia by pre-publication review. In Proceedings of OpenSym DOI
  8. User:Scottywong/Article_creation_stats
  9. User:MusikAnimal_(WMF)/NPP_analysis
  10. Warncke-Wang, M., Ranjan, V., Terveen, L., and Hecht, B. "Misalignment Between Supply and Demand of Quality Content in Peer Production Communities", ICWSM 2015. pdf See also: Signpost/Research Newsletter coverage
  11. Shyong (Tony) K. Lam and John Riedl. 2009. Is Wikipedia growing a longer tail?. In Proceedings of GROUP DOI
  12. Research:Wikipedia article creation

See also[edit]