Wikimedia Blog/Drafts/Bug fixes at WMF

Title

Buggin’ out! How the Wikimedia Foundation triages and fixes software bugs

Copy

Finding software problems

As part of the Wikimedia Foundation’s mandate to operate the Wikimedia sites, a small technical team is responsible for maintaining the MediaWiki core platform, along with its features and extensions. Unusual among many of its tech peers, the small engineering team at Wikimedia Foundation is helped in this process by a contingent of dedicated volunteers who aid in finding, reporting and fixing these unintended software problems, or bugs. The developers who maintain key software in the stack, improving and enhancing the functionality of MediaWiki, include people outside and inside the Wikimedia Foundation, working side by side -- and similarly, volunteers and staff work together to detect and diagnose problems to fix.

Bugzilla is the website where any user or developer can report a software bug or, if they have an idea, request a feature enhancement be made to MediaWiki. Users of Bugzilla can also search current reports to add new comments or information to previous reports about a similar bug.

Andre Klapper, Bug Wrangler

Wikimedia’s Bug Wrangler, Andre Klapper, takes a look at each bug report. He said the first step in resolving bugs is getting each Bugzilla report to the developer who handles that area of software code. Therefore, Klapper said, it is important for him to figure out the correct product, such as MediaWiki or MediaWiki extensions, and then the related subcomponent that each bug falls into. This helps to put the reports into the appropriate basket, so they find their way to the right person to be addressed. As Bug Wrangler, it's also his job to make sure each report has enough information for the developers to fix a bug. All of this falls into the "bug triaging" process.

“I’m responsible for triaging, and there are some great community members who help me with that,” said Klapper. “When I see problems that are urgent or really critical, I escalate by making developers explicitly aware of such problems. I also try to keep an eye on the many forums (such as Village Pumps) and places where users report problems, and make sure that software bugs end up as a report in Bugzilla, so developers can find them.”

In describing the open platform used by the Wikimedia Foundation, Klapper added, “Basically anybody can define or correct the priority of bug reports and feature requests. Often it’s members of development teams or other volunteers who triage, or I set [priorities] in order to help developers see what’s the most important stuff to work on.”

Klapper, who joined the Wikimedia Foundation in 2012, estimates that 70 percent of bugs are reported by Foundation staff and developers, 30 percent by users. The role of Bug Wrangler was a natural fit for him because the “position described pretty well what I’ve worked on before and what I’m specializing in” at GNOME and the now defunct Maemo/MeeGo.

Bugzilla has taken its share of knocks from critics, said Klapper (who counts himself among them), because it requires a separate registration, the user interface is complex and it lacks dashboards. He said the developers at bugzilla.org are working on these issues for the next Bugzilla version, and in the meantime, he has started a weekly blog entry called “Bugzilla Tips,” where he gives advice for using the tool.

“I try to make Bugzilla work better for everybody and, hence, ask teams what they are missing, or try to help establish good workflows,” he said. “For example, I introduced a new Bugzilla front page on bugzilla.wikimedia.org recently that provides quicker access to the main tasks.”

Releasing new software on the production servers

Greg Grossmeier, Release Manager, Platform

The roughly 100 developers and engineers at the Wikimedia Foundation balance the need to address current software problems with new software development and deployment, such as mobile apps and the VisualEditor, which launched in beta on English Wikipedia this month. Greg Grossmeier, Release Manager in Platform, manages the process of deployment, which means getting this new software onto the production servers (on the live site that users experience).

Since Grossmeier’s arrival at the Wikimedia Foundation, his team has cut the length of time between software development to deployment from two weeks to one week, partially through enhanced testing before deployment.

“It actually was an interesting cultural change and process change for getting that software out there faster because there’s all kinds of considerations to deal with, like testing,” he said. “It involved a lot of support from our Engineering Community Team, including Andre, and also our QA department, who help do automatic software testing and also some manual software testing.”

Bugzilla reports addressed by the Wikimedia Foundation also come from third-party users who run MediaWiki software and contribute to the software. If a bug does not affect the Wikimedia Foundation's users, Grossmeier and Klapper don't give it as high a priority. “But a large portion of the time, honestly, it usually is something that affects us,” said Grossmeier. "So at that point, it is prioritized across what is being done right now, consistent with the timeline for everything that we’re working on.”

Not all new features that are deployed started their journey by bubbling up to the top of the priority list. The musical scores feature, deployed earlier this month, became a reality partly because volunteers drove this functionality forward.

“That’s been a request that we’ve had since, I think, 2004, and we just, in this last month, deployed it,” Grossmeier pointed out. “We finally were able to roll out that functionality, and so that was what, nine years.”

Assuring quality and transparency

What makes the Wikimedia Foundation so unique is the openness of the software development process. "Every aspect of development, from the planning, to the source code, to the defects is available for public scrutiny," said the Wikimedia Foundation's Quality Assurance Lead Engineer, Chris McMahon. He's been using wikis and open source browser testing since they were invented and he joined the Wikimedia Foundation in February 2012 to work on quality assurance issues. He considers this job the highlight of his career.

Chris McMahon, QA Lead Engineer

"Paradoxically, I think this openness might actually promote a higher tolerance for issues and problems," McMahon said. He noted that a large part of his time in QA is spent detecting "regression problems, where a change we make today has an adverse effect on existing software features." In the past, testing was mainly done in production and defects were reported after deployment. Today the developers and engineers are working to move testing the software "up the stack," identifying issues and problems before they affect users in a negative way.

McMahon doesn't care for the word "bug." "The word 'bug' is a 'defect,'" he said. "'Defect' is a 100 percent negative term, which doesn't really help the conversation much, nor does it accurately describe aspects of software function that may be negative in some sense, but positive in some other sense."

McMahon argued that Bugzilla is one of the "most flexible issue tracking systems ever invented." He listed numerous examples, such as project management, wish list, personal to-do lists, community engagement, asynchronous communication where e-mail is not appropriate, and its ability to scale to meet the demands of a really enormous site, like many Wikimedia projects.

Overall, McMahon said he thought Bugzilla is the best system to operate on such a large scale. "As Jerry Weinberg said once, 'Things are the way they are because they got that way.'"

To report bugs or get involved in the process to fix them, start an account on Bugzilla here. If you would like to get involved with the Wikimedia Foundation team, please see this opening for Platform Product Manager and consider applying.

Donna Peterson
Wikimedia Foundation Communications volunteer