User:Isarra/Releases Investigation

Possible grant proposal: Investigation and support for better MediaWiki releases for third-party users?

This would be a research and software development grant for whenever they do those next.

Currently:

• Need to talk to Release Engineering: responsible for tarball, but they're also responsible for everything from wikimedia deployments across multiple phases to CI to dev tools and support. What are their priorities? How do they feel about tarball releases, what would they like to do better? Are we way off base with all of this, or can we turn this into something they do want?
• Aside from announcements and the specific assembly, it appears a lot of the release handling and fixes are mostly done by a group of staff from other teams operating more in a volunteer capacity, as well as various third-party contributors. Are they happy with this setup? What do they need to be able to handle this more effectively? Should this specifically get more direct support, and would it be justified within the overall aims of the WMF? Would it fit in the current structure?

Problems

WMF developers don't take the tarball/version releases seriously

WMF development isn't targeting the version releases that third-parties use at all. WMF projects are on a completely different kind of release model, running off master, so for Wikimedia development we mostly just need to make sure things work on master, and changes that depend on each other across components/repositories are simply marked 'depends-on' so they get merged at the same time.

Resulting issues include:

• Make breaking changes right before cutoffs (sometimes which also get reverted before the next version)
• Requirements for renames, deprecations, usage shifts, etc are mostly based on needs for WMF approach. This results in compatibility problems across versions, for other users. When renaming a function in core, it is considered good practice to replace all use of that function with new name across repositories. Unfortunately this breaks repositories for third-party components which are often targetting the older versions of MediaWiki that their maintainers actually use, as the new name does not exist on those.
• Others?

To investigate: whether or not using the release cycle as unifying targets, with recommendations for planning ahead and timing for deployments, could help with inter-team coordination and expectations as well as reduce overall effort of effective deployments within the WMF ecosystem specifically? How do we show this? Or if not, how do we effectively rule it out?

Specific processes are documented inconsistently, could also likely be greatly improved

• What processes affect release integrity/efficiency? Is the documentation complete for the current processes?
• Current release process gets news out when it happens, but lacks heads up ahead of time.

Would recommendations what to do now when announcements happen help? Defining specific targets for 'release is now branched, do breaking changes from now to X, or hold off (a month?) until after the next deployment' could help. How would this impact the development process for impacted components? How would this impact planning, and perhaps improve overall efficiency in terms of producing the code itself?

Tools for testing and building could likely be greatly improved

Folks are doing the best they can, but often with an unnecessary scramble to even just not do a release that's broken within itself, let alone with previous versions or expectations, or non-packaged components. What is the infrastructure for creating and updating and cross-testing releases? How much is done manually? How consistently are bugs reported, and resolved?

Apparently unclear guidelines as to what to backport to previous releases, when to make breaking changes, what is appropriate to change at all when it does not change functionality

Or are guidelines simply not followed? How much of this does the stable interface policy cover? What likely needs changing, especially around LTS vs regular releases?

For instance:

• Large-scope breaking changes should probably ooccur on LTS releases. Fixes for these changes that are found and included in later releases should also be backported. But what else is worth backporting? If such changes are deemed worth doing later, should they be done anyway, and also be backported to LTS?
• Should simple breaking changes, such as name changes like DB_MASTER to DB_PRIMARY, only occur on LTS releases or be supported in either form across the LTS cycle, until the next? What are the current guidelines for these, where it is purely for consistency or clarity, but function is unchanged? What are the requirements to make them worth doing? How do these impact the difficulty of supporting different versions when working on external components, extensions, third-party projects, etc? Is this part of why third parties are often so out of date?
• Do we need to reassess the deprecation policy to cover renames or transitions across minor versions between LTS releases in order to better allow third-party developers to support multiple versions?

Third parties are often very out of date, and we don't even know why

Why? What are the blockers?

• Skins are one, and have historically been very difficult to keep updated, and required a strong understanding of core to do so effectively. Has this improved? Are the current changes making this better or worse?
• How are extensions affected? What extensions are mostly affected? What winds up being blockers in particular? (issues with actor migration, MCR, etc: how these impacted often key components such as SMW or DPL?
• Where is wikiHow at and how does this affect their outlook, with their many custom tools for content and user management?
• What has gone wrong and what should be worked on, with these still, or with other similar things in the future?
• Services? Extension:Math? Restbase? Why is parsoid so easy to set up, but math of all things seemingly impossible, except when it randomly decides to work in testing? Is parsoid sorted out anyway at this point (packaged?), what were the learnings from the team behind it, and could they be applied to others? Who would actually do it? Who is currently responsible for this?

We used to have (more) people explicitly responsible for releases, certain aspects of third-party support, maintaining an overall mw community, intra-organisational project health

• Who does these now?
• What has changed? What is the impact of this change?
• What are we missing, either that we had and now don't, or that we never had to begin with?
• What is actually just fine and happily chugging along in the background, and we don't even realise what they're doing for us all?

We are not supporting marginalised groups effectively

Third-party users of the MediaWiki software come in all sizes and have all manner of goals. Some include relatively marginalised groups. Given the somewhat large overhead evidently required to keep up with the current release cycle and constant breaking changes, how are these impacted? Based in WMF goals, what are the relevant goals that are not at all being met with this? How does this apply to the 2030 strategy?

Specific issues:

• The only groups with clout are those with the resources to do everything themselves. Corporate, large organisations with their own staff to work on this, organisations with entire communities of developers of their own like SMW.
• Related third-party conferences are often prohibitively expensive, and assume corporate backing. Even Wikimania is prohibitively expensive especially for volunteers from marginalised backgrounds to attend without aid; what mw-specific organisations and events can they even take part in? Hackathons generally were a europe-specific thing.
• How are third-party users who do not have these kinds of resources doing? What are their needs which are not being met? How many still exist at all, and how do we even find them?

We're kind of really blind about third-party users

And they've kind of given up on upstream for the most part as well?

• What's up with wikiapiary? What's even going on in the wider mw world these days?
• If we check the box on install to send usage info upstream, where does it actually go, and does anyone actually use this data? What data is this? Do we have any info on how many folks use it relative to don't (for plublic wikis)?
• If we actually compare the overall health of the mw development community using mw.org, phab, etc, how does this compare to the community previously? How do we meaningfully compare this? What are the implications, what do we then need to ask?

People previously maintaining things on gerrit are moving off to github, gitlab

How can we better support projects that are on gerrit/in phab so they don't get too annoyed and/or get an actual benefit from being here?

• If projects are here, we can do easier code searches for needs to meet/things to not break, etc - but are folks actually doing these? Are they following up adequately? Or are they just ignoring what they do find?
• How do we search for code that will be impacted outside of gerrit, for what's published elsewhere? How do we convince people that supporting these use cases actually is beneficial overall (essentially, would it potentially be feasible to actually prove long-term scaling benefits of a wider consideration in designing architectures, and establish such as normal best practices? How would one go about this for a followup project or such?)

Installation and especially update process for third parties can be, uh, hard

Besides even just making sure all non-wmf components are actually compatible with the new version first, the process is just annoying. Either you use a tarball, and then you have to somehow... tarball over it? Or you use git, and then you need to find the right branches, not overwrite changes you made as well, etc. And often you wind up using different versions of skins and extensions and even vendor than were actually packaged, which then means you need to NOT overwrite any of that, either.

At least most extensions play nice with being included as links instead of directly placed in the core/extensions/ directory, now, right? Or does update.php also still fail on those, so then you gotta actually move them into place so the maintenance scripts or whatever go off right...

• What all makes it hard these days?
• How are other folks doing this in general? How are we going to do it?!

Method

Basically, we'll just write a big-arse report, and everything we do will be 'research' to include. This is possibly how things work in academia, definitely how things work when you've suffered head trauma and literally can't remember jack shit unless it's written down. And we really need to write everything down anyway.

Specific smaller reports/documentation to work on

• Overview documentation of what we're dealing with currently, and determine some conclusions: what are our recommendations based on findings? What is good, what is bad, overall?
• Documentation everything we find as we go in various specific categories
• Documentation on-wiki documentation on wikitech and mw.org, adding to/updating as needed

Not explicitly documentation things we'll do

• Develop stuff! Work on things needed by third parties, document where it goes wrong, what the difficulties are, blockers. Document to how things change through the release cycle. Skins, extensions, maybe some core components especially. Also CI stuff!
• Pay attention to extant wmf mw stuff as things happen over time! Read mailing lists, see what's going on. Find the relevant policies for deprecations, renames, etc and see if people seem to be following, if they appear to be helping, or doing enough. Follow random teams if their work seems to be impacting anything we're working on, and document how, in a totally non-creepy way. Document findings of interactions. Were they amenable to working with our needs, or accepting our patches? Did we wind up telling them over and over that what they're doing won't even meet THEIR needs, let alone anyone else's, and they just kept ignoring us, so we finally just went 'screw it' and made our own extension instead? Did they just not have time for us at all due to being completely overworked to begin with, where even reviewing patches would pose too much of a risk of throwing them off schedule?
• Follow up with other teams! Have their experiences been like ours? How do different teams within WMF development coordinate and collaborate? What were their experiences, what have been the results? How can this be improved overall?
• Follow the release cycle specifically. What seems to be impacted by it, where and when do we run into problems with the things we're working on/maintaining? What should perhaps be impacted by it but isn't? Does it match our expectations, internal expectations, third party expectations? What does everyone expect to begin with?
• Planning! We all have development plans, right? How do these relate to the release cycle, or could they be related to it? If we're not planning, why not? What are the impacts of the lack of plans? Desktop improvements could be an interesting case study, what's their plan? For extensions we wind up working on, what are our plans? Do we actually document them first, or do we just delve right in too? How does this differ between big and small things?
• Probably specifically hire someone to do a large thing so we have a comparison. Get Skizzerz to do that skin templating extension, for instance. Watch him very, very closely to see what he does.

Goals

Make things better! In general, by the end, this should be true:

• We all know more what's going on in general, because relevant onwiki documentation is maybe improved across the board!
• Hopefully release engineering is happy with us because we helped with their goals, and not pissed because we wound up accidentally breaking all of gerrit or something?
• We'll publish a big-arse report of findings and recommendations, and how unexpected things can really benefit us all, where folks can use it to improve things, but if not, it will set a clear record of what's currently going on, and how it's changed over time to get here and remain a useful record in the future. Possibly also broken into smaller reports to directly send to relevant folks 'to do with what you will, and we totally get it if it really isn't feasible any time soon, but just so you have it or stuff'.
• Key extensions for third parties are updated a bit more so at least those aren't still blockers and maybe they'll stop using 1.27 already?!
• More and better modern skins are available for third-parties to use, and properly updated and whatnot. Probably a few more skins in the tarball, too, based on findings of what are particularly common ones?
• Improved installation and update process for third-parties, maybe?

What we need

• A whole bunch of money
• Skizzerz to agree to doing all the work making his thing now so we can stare at him the entire time and take notes
• Some guidance how to not get too much in anyone's way especially within the wmf itself
• A couple more people? Maybe actually pay ashley this time? Also a manager might be nice for a change.
• To figure out exactly how this fits in with grantmaking/2030 strategy priorities, not just wmf in general
• Probably need to actually set up our llc first, huh. And it probably can't actually be uncyc... can it be uncyc?
• To go find out what release engineering even makes of this first?