Talk:IP Editing: Privacy Enhancement and Abuse Mitigation/IP Editing Restriction Study/Portuguese Wikipedia

Pipelines

The long-time editors I've asked generally edited Wikipedia for months or even more than a year before creating their account. In this respect, IP editors form a pipeline to the registered community. The effects of shutting off the pipeline might not be seen for a year or more. How long do you expect to follow this experiment? Whatamidoing (WMF) (talk) 19:39, 3 September 2021 (UTC)

Hello @Whatamidoing (WMF)! There could likely be long-term effects, as you have pointed out. We're looking at capturing data for 1-2 years and even more. STei (WMF) (talk) 21:37, 13 September 2021 (UTC)

I'd like to echo this sentiment. At the very least, any kind of conclusion reached so far should be contextualized or qualified. That is, a declarative sentence like this: "We found no significant negative impact in the analysis conducted thus far" should be more precise in what period was measured, at what time it was measured, and any notable comments about the limitations of the methodology. It currently reads like an ironclad conclusion, and as a researcher and academic, it really gives me pause. -- Fuzheado (talk) 16:19, 10 September 2021 (UTC)

Agreed. Nemo 17:21, 10 September 2021 (UTC)

Hi there @Fuzheado and Nemo, in case you missed it this section of the report will help with all things on context and period of measurement. I will also sum up here to save you time. IP editing was turned off on October 4th 2020. Per the background section, we're concluding based on metrics gathered from October 2020 to June 2021. That should amount to Q2, Q3, and Q4 of the Foundation's 2020/2021. Ping me if you need anything else. STei (WMF) (talk) 21:54, 13 September 2021 (UTC)

On the issue of limitations of methodology, that are about five points in the report that data shortcomings or lack of clarity were mentioned. I am also pinging my colleague Jennifer to help with further commentary if there's any to add. – STei (WMF) (talk) 22:08, 13 September 2021 (UTC)

Kudos to the Analytics team for, erm, analysing this. (And to the Antabuse team for sponsoring it, if that's how it works.) I've been wondering whether any positive and negative effects of ptwiki's change have become discernible. Surely I'm not alone in that. I'd love to see a re-analysis in say a year's and two years' time with some longer-term data. ⁓ Pelagic (talk) 23:00, 10 September 2021 (UTC)

Hello Pelagic, we're looking at capturing data for 1-2 years and even more. – STei (WMF) (talk) 15:27, 20 September 2021 (UTC)

Thanks, @STei (WMF), good to hear that you do have plans for that. Pelagic (talk) 14:02, 27 September 2021 (UTC)

Comparison graphs

In IP Editing: Privacy Enhancement and Abuse Mitigation/Impact report for Login Required Experiment on Portuguese Wikipedia#Net non-reverted content edits, I think it would be helpful to have graphs that visually show the changes at the Portuguese-language Wikipedia against the similar numbers for other Wikipedias. If everyone else is growing while you stagnate, that's not necessarily good. Whatamidoing (WMF) (talk) 19:40, 3 September 2021 (UTC)

Apples, oranges and herrings

Just above that, at § Net non-reverted edits, it says "In Q2, the monthly average of all wikipedias projects increased by 13.5%." Is that 13.5% total edits, or NNREs? I hope they're not comparing apples and oranges there! Pelagic (talk) 20:53, 10 September 2021 (UTC)

13.5 % is for NNREs. Thanks for catching that. -- NKohli (WMF) (talk) 19:39, 7 October 2021 (UTC)

Also, a global edit growth of >10% per month would be flabbergasting, if sustained. How much would that amount to over a year? (1.135)¹² = 4.57. Has anyone investigated where all those edits came from? And if it's just an unsustained glitch followed by a corresponding fall, then why is it relevant to this analysis? Is it a red herring thrown in to downplay the growth in NNREs? Pelagic (talk) 20:53, 10 September 2021 (UTC)

Aha, at mw:Product Analytics/Data Products/ptwiki intervention impact report the context is that it's “year over year” not one month to the next. It would still be nice to see NNRE comparisons vs. other wikis. Pelagic (talk) 04:57, 11 September 2021 (UTC)

@Pelagic one other issue with the graphs is that previous year had multiple lockdowns which drastically changes user behavior and making the comparison much more unreliable. Amir (talk) 20:48, 17 September 2021 (UTC)

@Amir that is very true. COVID has impacted editing patterns. It is hard to discount those changes in the data. Which is why we are proposing to do other similar experiments on other projects to compare findings. -- NKohli (WMF) (talk) 07:02, 25 September 2021 (UTC)

Other research

I'm sure there has been several researches in this topic before. On impact of automated reverting of vandalism, impact of pending change on all of the wiki, and impact of restricting IP editing. All unanimously came to conclusion that making editing harder for IPs would harm the wiki's new user retention in long-term. It is interesting to see this research disagreeing with all of that. I'm not debating your result, It can be simply that users' behavior have changed since most websites require an account for really long time now.

I can find these researches if you want but finding them shouldn't be too hard. Amir (talk) 22:54, 3 September 2021 (UTC)

This study looked at a few months. If long-term is several years, then this study doesn't know what will happen.

OTOH, I think you are correct that people's expectations have changed compared to 20 years ago. Whatamidoing (WMF) (talk) 16:22, 5 September 2021 (UTC)

Agree with @Ladsgroup on this – the stark difference in the conclusions from examining pt.wp in this report versus the previous scholarship from very storied and respected researchers needs to be greatly highlighted and analyzed.

Also, in chat discussions with pt.wp editors before IP editing was turned off, we talked about the effectiveness on en.wp of anti-vandal bots (i.e. ClueBot NG) that essentially allowed en.wp to stay completely open to IP editing. Editors on pt.wp noted the lack of a similar performing bot on their wiki, and how this might have affected community voting sentiment. I'd like to see any kind of research take these other factors into account, in that wiki-to-wiki comparisons have multiple variables that can easily lead to improper conclusions. - Fuzheado (talk) 16:28, 10 September 2021 (UTC)

@Fuzheado indeed. I can add that each wiki utilizes different tools to combat vandalism. In my home wiki (fawiki), AbuseFilter is heavily used and has rule that if an IP adds a swearword (and some other conditions), it also blocks the IP for three days. That helps with the flood of vandalism a lot.

IMHO, just blank banning all IP edits is lazy. There are lots of tools you can use. Amir (talk) 18:29, 10 September 2021 (UTC)

@Ladsgroup: Let it be lazy, then. We are all volunteers, and we do what we like, when we please. If you want people that prefer to operate antivandalism tools instead of doing something else, then pay them or find any way of attracting them to do what you think that must be done (or do it yourself). And you are being blind to the HUGE problem of security that IP editing is in all the projects - starting with the newspeak misnomer of "anonymous editing" to IP editing. Ask mainland China what they think of such "anonymity".--- Darwin ^Ahoy! 12:19, 14 September 2021 (UTC)

Let's then be super lazy and block all editions unless you can show proof of identity and preferably any documentation backing up your expertise on the subject. On a more serious note, no one from the WMF (and they are not volunteers, but rather salaried employees) has even bothered replying to my comment here. Hence, I assume they'll only do a blanket ban on IP editing for 8 months and generate some YOY graphs using Microsoft Excel. That's not an experiment. Ñ (talk) 08:07, 15 September 2021 (UTC)

You may be interested in reading about the concept of a w:Natural experiment. Whatamidoing (WMF) (talk) 16:11, 15 September 2021 (UTC)

Thank you. If that Wikipedia article is supposed to somehow justify the design of this endeavour, I would strongly recommend the higher ups to hire someone with ample expertise of Experimental Design. I would also recommend them to read about w:Causal inference. Ñ (talk) 19:17, 15 September 2021 (UTC)

@Ñ Read about what you've just done here. - Darwin ^Ahoy! 12:07, 18 September 2021 (UTC)

@DarwIn: are you talking about my suggestion of being super lazy? My statement was tongue-in-cheek. I am completely serious, however, and I'll encourage all the interested parties to think about the value of this endeavour. As I said in one of my first interventions: if you'll do it, do it -- but design a proper experiment (or at least a quasi-experiment). Ñ (talk) 16:48, 18 September 2021 (UTC)

I sometimes find that humor doesn't 'translate' well across cultures. What I think is a funny comment might sound like an insult to others. Just looking at the names on this page, I see people from at least seven different countries, and at least five different native languages. It might be more practical to avoid tongue-in-cheek statements.

It sounds like you are very concerned about this issue. Can you tell me more about what you think could go wrong? For example, do you worry that we'll accidentally kill the communities by requiring registration? (I do, especially for smaller ones.) Whatamidoing (WMF) (talk) 21:19, 20 September 2021 (UTC)

Unless one is a shareholder in a corporation and one risks to lose one's own money and livelihood, 'worry' is not part of the equation Whatamidoing (WMF) - at least the WMF never worries about losing or wasting money, after all, donations are easy-come-easy-go, and they waste a lot of it. The results of a structured trial, like ACTRIAL, are what count. The only thing that could go wrong would be if the WMF were not willing to cooperate. Running a trial is something the communities can do locally by using filters, and they won't wait for 'permission' from the devs at Phab or the salaried employees of local chapters. Most important however, is that a trial is prepared and conducted professionally and scientifically like it was for ACTRIAL as NKohli (WMF) as remembers. And that is worth investing a little bit of money into to have it done properly. Kudpung (talk) 01:19, 27 September 2021 (UTC)

I don't think that even the most professionally and scientifically designed trial can tell you whether a change is desirable. It might be able to tell you certain facts, such as whether current admins prefer the current or prior state, or whether the growth of article volume or quality has changed, but not whether something is "good". Part of designing a trial is to figure out which outcomes you think you should study. For example, if you worry that cutting off IP editing will result in a long-term reduction in future registrations, then you would want to design a trial that looks at long-term registration trends. If you worry that not cutting off IP editing will result in near-term burnout for admins, then you would want to design a trial that looks at admins' satisfaction. All of the hopes and worries should be reflected in the trial design. Whatamidoing (WMF) (talk) 17:58, 27 September 2021 (UTC)

┌─────────────────────────────────┘
Whatamidoing (WMF), that does not really address the points I was making. ACTRIAL - with which I am sure you are still familiar - proved that objective and professionally organised research and trials work very well. Discussing admin burn-out is not relevant here. Admin 'burn out' is a totally subjective construct; there is however a limit to what the volunteers are prepared to do for free for the 500 employees who in fact are only there to represent our interests and maintain the software and servers. The volunteers signed up for building an encyclopedia, not a socio-political movement. It's not rocket science to establish the criteria on which such a trial be based, but although I have over 40 years of academic experience designing trials, it's not my job to tell the WMF what they should be. Employees Nettrom and DannyH (WMF) came up with an excellent plan for ACTRIAL. Founding principles laid down 20 years ago may well not be entirely appropriate for today's Wikipedia and what counts now is pragmatism over ideology; what concerns us here among other issues is that self-appointed volunteers at Phab are acting as 'keepers of the toilet keys' and dictating policies which are not within their remit. IP masking yet needs to be proven as a legal requirement, what is needed now are better tools for controlling article quality or banning IP editing altogether. (FYI DarwIn). Kudpung (talk) 01:43, 28 September 2021 (UTC)

Anonymous literally means ‘without name’. In that sense it is the right term for someone who has't registered – or hasn't logged in to – a named account. It's a good counterpoint to pseudonymous, which is what most of us are. (The other cases are real-life name with no proof of identity, or some kind of verified-identity scheme.) Unfortunately, anonymous has taken on some connotations of protected or untraceable, which we all know it isn't, unless you can find a proxy/tunnel exit node that we're not blocking. But that's more related to MediaWiki IP Masking, and our problematic No-Proxy approach. I suspect for IP Vandalism it's the low friction or effort which is the main factor. Pelagic (talk) 21:53, 26 September 2021 (UTC)

I have a lot of difficulty in considering "anonymous" something that is often easily traceable to a single person (sometimes even by anyone looking at it - I used to connect from IPs that were named as something like "terminal04.deic.fc.ul.pt", giving away the exact place where I was seated in the faculty, as the terminals were numbered). It may not say your name, but be pointing at you very clearly. So, "anonymous", it is not. - Darwin ^Ahoy! 19:08, 27 September 2021 (UTC)

Who does the WMF think it is?

Allow Portuguese Wikipedia to continue with their current practice for now, if the volunteers want to continue. 'Allow'? If a community wishes to change its rules, it will. It won't wait for the WMF to give permission. The WMF tried to bully the en.Wiki fpr 6 whole years over ACTRIAL until the volunteers got wise that the WMF are not the bosses. Please change the wording. Kudpung (talk) 09:51, 7 September 2021 (UTC)

Whatamidoing (WMF), you seem to be more informed than most here. Could you consider rephrasing the page so that it reflects facts and reality? Kudpung (talk) 15:41, 10 September 2021 (UTC)

The community has not approved this change. Nemo 17:21, 10 September 2021 (UTC)

I say leave in "allow". It illustrates the attitude or misconception of the authors management, and may serve as a stimulus for further discussion. ⁓ Pelagic (talk) 22:06, 10 September 2021 (UTC)

@Nemo bis: The pt.wiki community certainly has.--- Darwin ^Ahoy! 12:10, 14 September 2021 (UTC)

Modified my comment above to reflect info from Danny @ 21:57, 20 Sep, below. Pelagic (talk) 21:01, 26 September 2021 (UTC)

Correct me if I'm wrong, but the w:pt community implemented this themselves using edit filters and javascript. (See also User:Stjn's post at Login Required Experiment#Technical implementation.) Ptwiki went to Phabricator with a community mandate, asking for a config change, and got no help from the sysadmins. I'm not sure whom else they petitioned, perhaps somebody could elaborate on that. Foundation management must have been aware what was afoot, but failed to support the community. However, once the community was forced into a hacky DIY solution, the only way the W?F could disallow it would have been to impose another Superprotect. ⁓ Pelagic (talk) 22:45, 10 September 2021 (UTC)

@Pelagic: After all the drama on phabricator, we decided to ignore it and go forward with solutions designed by ourselves that would accomplish what the community had decided, which we did. After that there was still a denounce to legal about the filter we were using potentially allowing for the match between IP and account, with some imagination (something that happens all the time without any need for imagination and special access to special tools when accidentally someone logs out and edit), and we had a WMF lawyer (very politely) expressing those concerns. It was quickly settled, in any case.--- Darwin ^Ahoy! 12:10, 14 September 2021 (UTC)

Kudpung, Nemo bis, Pelagic: You're right, "allow" is the wrong word here. This report was inspired by questions that we got from WMF leadership and the Board, essentially asking: This has never happened before, so should we take some kind of action to discourage it, or talk with the volunteers in the hope of changing their decision? I'm sure we can all list examples of WMF staff taking that attitude, and it doesn't lead to good results. We decided that — rather than attempt to interfere, without listening and understanding — we should think of this as an experiment/pilot, which we can assist by doing some data analysis. We also talked with volunteers on Portuguese WP, to make sure that we heard their views on how it was going, and those voices are included in this report. Our conclusion, based on the data and the volunteers' responses, was that we shouldn't try to do anything to sway this one way or the other, and instead to encourage a couple more Wikipedias to try it out, so we could learn more. That got expressed as "allow this to continue", which I agree doesn't really capture the intent. — DannyH (WMF) (talk) 21:57, 20 September 2021 (UTC)

DannyH (WMF) thank you for this. It is indeed very reminiscent of ACTRIAL where you personally stepped in to reverse the years-long arbitrary refusal of developers at Bugzilla (the former Phab) to accept the overwhelming consensus at several RfC on en.Wiki to prevent brand new users from creating new pages. Your offer to provide help went far beyond what we had hoped for and was an essential asset to the trial. Kudpung (talk) 00:18, 21 September 2021 (UTC)

@Kudpung Why do developers end up having so much undue power (or the perception of it, at least)? Is there not any way to prevent that? - Darwin ^Ahoy! 00:57, 21 September 2021 (UTC)

@DarwIn: I've answered on your talk page. Kudpung (talk) 02:05, 21 September 2021 (UTC)

DannyH (WMF), I think it's great that you are doing the analysis and talking to the pt community now! On enwiki, we were aware of the discussions at ptwiki well before they were concluded and even later enacted. Did WMF leadership consider (and decide against) talking to the volunteers at the early stages? Or was it not until ptwiki flipped the switch that leadership had an "OMG, they're actually serious about this" moment? Pelagic (talk) 21:01, 26 September 2021 (UTC)

Pelagic: I don't actually know the history of who knew about it and when. People on the Community Relations team and in Trust & Safety usually have a good grasp of important things happening around the wikis. It got surfaced to me personally at the "this is a thing that's happened, what should we do about it?" stage. — DannyH (WMF) (talk) 21:04, 27 September 2021 (UTC)

Editors reduction

I was looking at the data on this and this caught my eyes: https://stats.wikimedia.org/#/pt.wikipedia.org/contributing/editors/normal%7Cline%7C2019-08-01~2021-10-01%7C~total%7Cmonthly

There is a massive reduction in unique editors (was cut to one third!) but I assume it's partially expected because a unique device can have multiple IPs in time period of a month and when they are forced to create a user, they stick to one.

But this also explains the increase in user retention and basically saying it's artificially inflated because they got just mostly transferred from a regular IP editor to a user editor. What percentage of them gave up editing altogether (instead of creating an account) is not measured here and should be measured before making a decision. This can be measured by looking at data on unique devices which is private (but I have access to them) and requires some data magic.

Another point I mentioned is that the captcha for user creation is not a11y-friendly at all: T6845 We are effectively perpetuating an ablest bias in our users by requiring creation of accounts for editing. Amir (talk) 16:56, 18 September 2021 (UTC)

Thank you for catching that! @JWang (WMF) flagging this for your attention.

Re: Captcha - Thanks for brining that to our attention. I understand that it is a major concern. Going through the ticket briefly, it seems that we are far from a consensus on the best path forward. It is also clearly a rather large project that no team has tried to tackle so far, with a lot of open questions. Given the scale of this project, it is unlikely that we will arrive at a resolution for it anytime soon. However I think it would be valuable this information with the ptwiki and fawiki communities and make them aware that this could be happening. @STei (WMF) bringing this to your attention. -- NKohli (WMF) (talk) 07:19, 25 September 2021 (UTC)

Well noted NKohli (WMF) – STei (WMF) (talk) 14:49, 28 September 2021 (UTC)

After requiring registration for editing articles, the total number of editors at the Portuguese Wikipedia declined sharply, and the number of registered editors rose.

A belated follow-up on this, @Ladsgroup:

That graph scared me the first time I saw it, but it turns out that it's not as scary as it looks. It includes both registered accounts and IP addresses (which are IMO not really equivalent). The image on here shows the two kinds of users split.

Before the change, about 6,000 registered editors (yellow line) and and unknown number of users from about 23,000 IP addresses (red line) made edits each month. After the change, there were 9,000 registered editors plus edits from about 300 IP addresses each month (registration doesn't seem to be required for talk pages).

I am a little concerned that they had ~10,000 registered editors in 2021 and ~9,000 registered editors in 2023 – if that were a straight-line trend (which is IMO extremely unlikely), then there could be no editors left 15 years from now (I think that a decline followed by a stable plateau is more likely) – but I don't even know whether the trend is statistically significant. WhatamIdoing (talk) 17:54, 7 February 2024 (UTC)

The bottom line is: what quality are/were the IP edits? A few IP edits might be constructive, but it's a fair assumption that most are not. Subjectively: I have retired from editing enWiki, but I use the encyclopedia many times daily; I come across a great many weird minor edits (and some not so minor) and when I check the history they are almost always IP edits. With dynamic IPs the actual number of humans behind IP users could be significantly lower. It would need some sophisticated tools to identify and quantify all that and break it down into useful data. Is it worth it? The best test for enWiki wouould be to simply stop IP editing for a couple of months and see what happens. The encyclopedia is now so monolithic that no harm would be done by such an experiment. ACTRIAL proved that. Kudpung (talk) 06:04, 13 February 2024 (UTC)

I've wondered about flushing out some of the long-term IPs by very briefly (maybe in repeated bursts of a few hours) requiring registration. That would get people to log (back) in, and thus give us a clearer idea of how many IPs are long-term editors. One of the known results of the ptwiki change is that people who'd accidentally gotten logged out, and didn't care enough to bother logging back in, decided to log back in to their old accounts. However, this would have to be balanced against the previous user research, which showed that asking people to create an account resulted in lost edits. We have traditionally said that the content is more important than the contributor.

I don't think that a test of a couple of months can show us the years-long effects. About half of enwiki's existing high-volume "core community" editors started off editing as IPs. My own on-ramp to editing took somewhere between one and two years of very sporadic, often minor, edits before creating an account. If you ran a three-month test, you wouldn't see the effects on an editor like me at all. An editor who ends up discouraged from contributing, or who fails to register an account in the future, would not appear in the data.

I am concerned about the volume of edits at enwiki that do not create content. I noticed an article on my watchlist a while ago that had about 25 edits over about 25 months, since the last time I'd looked at it. Most of the edits were bots or semi-automated, and none added or removed even a single sentence. High-volume editors are often not people adding sentences and paragraphs and sources. They are often people making barely visible adjustments to the presentation of someone else's work (e.g., reference formatting – a desirable type of edit to make, but not the same as adding a fact and a good source). WhatamIdoing (talk) 19:35, 13 February 2024 (UTC)

New study on the costs of requiring accounts

I'd say it's now proven beyond doubt that requiring registration stifles long-term high quality contributions in a wiki: https://blog.communitydata.science/the-hidden-costs-of-requiring-accounts/

Thanks to this study we no longer need more "experiments" on Wikipedias, and we can start reversing the new restrictions where they were introduced, in order to restore the status quo and the founding principles. However, if WMF had sufficient resources to fund an experimental replication of the study on some Wikimedia wiki, it now has a script to follow. Nemo 06:16, 5 November 2021 (UTC)

This is how sensationalism distorts the reality: Benjamin Mako Hill may have a point but comparing Fandom with Wikipedia is even less useful than trying to compare apples with oranges. The 20-year old Founding principles were based on ideology. Today's Wikipedias need a pragmatic approach that addresses today's needs. The en.Wiki ACTRIAL proved exactly this, and for it, the WMF made adequate funds available for a fully scientific study and analysis.(@DarwIn:) Kudpung (talk) 08:26, 5 November 2021 (UTC)

A/B hypothesis Testing

We need to do [testing] which is common practise on web sites such as Etsy] [a methodology used within Mediawiki] and is a highly referenced way to get accurate facts. A/B would allow us to check whether IP edits are due to user experience related [lazy registration issues], or a false perception that IP editing is private, or something else.

At the moment there are 4 options - status quo, stop IP edits, other login options, and modifying the [[1]] to softly [[2]] towards non IP editing. Each of these options have implications in terms of new editor retention, but we need to be clear to ourselves that they also affect editor burn out rate/loss of existing NPP editors/admins.

In our case. A/B testing involves doing a series of experiments on what occurs after an unregistered editor chooses to edit. The process would involve automatically assigning a sample of similar first time editing users to an experiment or as a control. The results are then used to design new experiments.

Similar editors are found by using a combination of some/all of IP address (which gives us location, VPN , and time zone), similar local times (eg 3 AM), and their chosen articles belonging to the same projecte(Marvel) .Experiments are designed using a combination of allowing/not allowing/or advising a of a delay access to IP editing, giving different user experiences (messaging position/colour/text/description, nags, nudges, delays , notices, success notices, ability to claim recent edits, wording, survey questions before or after, whether their edit was not vandalism but reverted, and features of the Article that might make editors choose IP editing I think the features of the article they choose to edit (big yellow tags, importance, size, in body tags, number of talk messages as a proxy for conflict) may cause the new editor to choose IP to avoid conflict or Experienced Editor criticism due to the perception Wikipedia Editing is high conflict.

We should also include a post publish notice on whether they would have preferred to use a mockup of social login screen. Social login is an alternative to IP and a perennial proposal, rejected in the past due to advertising concerns or privacy. But the world has changed. 80 % of logins now use social accounts according to Y-combinator forum post] and the would be editor is already using their product if they choose their login, and users only see the login option very occasionally. Modern Users don't want email accounts registration, as they don't want to maintain accounts, email addresses change,and they can forget passwords. It also reduces reputational risk if wikipedia is hacked (not for Wikipedia access but find the password used on username, as users often reuse the same username and password on multiple sites. It also increases the Wikipedia's reputations if we cam anonymous editors can vandalise any page.--Wakelamp (talk) 10:58, 9 November 2021 (UTC)

An FAQ section is now available

An FAQ section has been provided to help answer questions you had. Please check the project page. Any further comments/questions are welcome. –– STei (WMF) (talk) 16:06, 22 February 2022 (UTC)