Upgrade procedure

Background

During the release of MediaWiki 1.3 on the various Wikimedia projects, many users felt that the new version had been inadequately tested and discussed to go into active service. The new features in the release were, in general, widely welcomed, but it was felt that more warning and chance for comment should have been given before activating the system on actively used sites. Indeed, the roll-out seemed to proceed to larger projects from an initial subset irrespective of the number of issues found in the latter.

This page was therefore created to formulate a more controlled approach to rolling out future version upgrades, to allow wide-spread testing of the new functionality without sacrificing the stability of active projects.

Key proposals

The key points that need to be considered before the next major release are:

• Beta testing: Encourage greater participation in testing during a well-defined beta phase, using one or more test installations.
• Conversion testing: Perform a dry-run of converting an existing installation using a real database, and incorporate this into the beta testing scheme.
• Stability criteria: Define in advance what level of apparent stability is required before the software can leave beta status.

Beta testing

One of the main responses of the developers to complaints about the 1.3 roll-out was that the new version had in fact been available for testing at http://test.wikipedia.org for some time, but that it was not receiving sufficient attention to discover the remaining bugs. This is in many ways a communications problem, since most potential testers were not aware of the status - especially since the test wiki is permanently accessible, and had been running rapidly changing versions of the 1.3 branch for some months.

To combat this, branches for major new releases of the software should have a well-defined status: alpha, beta, release.

1. For much of the time, the test wiki(s) are in an alpha state - probably stable enough to use, but with no guarantees that they will have the same functionality or issues from one day to the next. In this state, a minimal number of testers is needed, as many of the problems they will point out will simply be "in progress".
2. Once the software reaches a more stable state, completely new features should be held back while existing ones are debugged (this is analogous to the approach taken by the Mozilla project). During this beta stage, the test version should be actively publicised among users of the current software - messages posted to announcements pages, the en:Village pump and its equivalents in other languages (obviously, this requires some co-ordination of translations, but this should not be insurmountable). During this period one or more test servers should be made available, with clear instructions on where bugs should be sent and how they should be labelled (to avoid confusion with the currently "active" branch).
3. Finally, once all major issues discovered on the test installation(s) have been solved, a plan for rolling out the now stable software should be formed and announced. The roll-out and the announcing should then proceed in phases, rather than all projects at once. That way if major issues are discovered in the "stable" release, further installations can be put on hold until these are solved, while at the same time smaller projects that do not keep track of all developments, will still get a warning when it's actually their turn.

Key areas of testing

• Localisation (hard to test on a single test server)
• Accessibility (requires large variety of systems/users)
• User acceptance (developers and users must ensure that they communicate their intentions w.r.t. new features before those features become "set in stone")

Conversion testing

During the beta phase, it is important to test not just how the software will work in itself, but also how it will interact with existing content. This interaction can take the form of pages functioning differently after the upgrade, or even parts of the database being overwritten during install. Thus it is important to perform a "dry run" of an upgrade on a live database, and allow inspection of the results. The obvious way to do this is to create a copy of the database from one of the smaller projects, upgrade it, and then publicise the location of the resulting wiki as part of the beta testing process. This would allow side-by-side before and after comparisons of the two versions, and detect any problems brought about by the installation process itself.

Where changes to the software require manual alterations to content (e.g. the switch from HTML to wiki-markup in customised software messages) instructions should be given in advance to the appropriate users/administrators/stewards on how to implement these changes, so that a live site is not left "partially converted" by installation of the new software.

Stability criteria

Given the highly public nature of the Wikimedia projects, it is important that at no time are they "too unstable" - in terms of unexpected behavious in every-day functioning. A decent beta testing period, including a "dry run" upgrade, should prevent this, but is reliant on some a priori definition of "too unstable".