Wikimedia CH/Software Development Guidelines

Add links
From Meta, a Wikimedia project coordination wiki

Software Development Guidelines[edit]

For Wikimedia CH

The Wikimedia movement consist in millions of software.

Wikimedia CH contributors can help, and this document can be useful to learn how.

Note: this is a stub, a work in progress. This is not an official document. Thank you for your contributions! ❤️

The Software Development Guidelines helps technical contributors to have successful software development on a Wikimedia project. It is especially important for Wikimedia CH technical contractors, including Innovative Projects.

Preamble[edit]

While Wikimedia CH is not a software company, it co-exist with Wikipedia that has billions of visits every month, and it is just a piece of the global Wikimedia infrastructure.

Your help will support Wikimedia volunteers. It can happen that the project leader is actually a volunteer. So, thank you for being even more proactive in communicating your progresses.

If a difficult bug stops you, don't worry! Share your challenges in the public. It's easier to fix a problem if somebody—in a public place—tried to describe and analyze part of it.

Onboarding[edit]

At the beginning of your work, to contribute as a Wikimedia developer, you will need to create some new accounts.

You will need a total of two accounts to be used in four separate websites:

  1. Wikimedia account
    1. Meta-wiki: global Wikimedia account (account registration)
      This registration provides a valid identity to login in meta.wikimedia.org, mediawiki.org, wikipedia.org, phabricator.wikimedia.org and additional collaborative websites for the general community.
    2. Wikimedia Phabricator: bug tracking (do your first login)
      Visit the page and use the button on the left to do the login (the one called "Log in or Register MediaWiki")
      If you are redirected to a login page in "mediawiki.org", just enter the credentials related to your global Wikimedia account.
  2. Developer account
    1. Wikitech: dedicated technical wiki (account registration)
      This separate registration provides access to important separated services, like wikitech.wikimedia.org, gitlab.wikimedia.org and other resources for the technical community.
    2. Wikimedia GitLab: collaborate on source code (do your first login)
      Just enter your credentials of Wikitech

Don't panic. It is a total of two credentials.

Take your time to customize your account preferences and become more familiar with these different powerful platforms.

Team[edit]

Once created the accounts as mentioned above, it is good practice to customize your own user pages in a way which is meaningful and helpful for others.

A good public user page is short, nice, useful and updated.
The first principle is that your public user page is not intended for self-promotional purposes, but to communicate your background, your purposes, and to declare your conflict of interests, your role, your paid assignments. The purpose of this is to help others understand where you can contribute on what area and why.

Here's a handy checklist to follow when starting one's assignment or when a role changes:

  1. https://meta.wikimedia.org/
    Login. Visit your user page (usually it's mentioned on the top-right corner)
    1. add info about your role, in English
      Example: "Hello! In YEAR I'm paid to work on project ABC for Wikimedia CH. My specific role is: developer"
    2. mention your nickname of Phabricator
      Example: "On Phabricator you can find me as @supermario-wmch"
    3. mention your nickname of Wikitech
      Example: "On Wikitech I'm this user: https://wikitech.wikimedia.org/wiki/User:SuperMario-WMCH"
  2. https://wikitech.wikimedia.org/
    1. Login. Visit your user page
    2. add a link pointing to your Meta-wiki user page
  3. https://gitlab.wikimedia.org/
    1. Login. Select your profile picture > Edit profile.
    2. add a link to your Meta-wiki user page

You have success if your user page in Meta-wiki helps others in finding you in all other services, and if people from somewhere-else (for example Wikitech) can easily find your Meta-wiki account.

Please help your coworkers in doing the same.

Bonus point: on your Meta-wiki user page you can mention each other, to help others find your coworkers and better understand your team divisions. Creating a template for your team would be an easy-manageable solution to keep everyone connected without the need of reviewing a lot of user pages for any change in the team.

Where do I start?[edit]

Now that you have the necessary accounts to act, where should you start contributing?

The first entry point to Wikimedia CH software should be Phabricator: this tool can be used for track projects, act on software and to foster communication among members.
There are several projects affiliated to WMCH:

By visiting the links above you can figure out what the current projects are and how the community looks like.
Please don't start to contribute just yet, complete the reading of the remainder of this document.

Software Guidelines[edit]

The Wikimedia technical community maintains hundreds of thousands of software projects.[1][2] Your help is important to follow basic best practices.

Operating System[edit]

An happy web application supports Debian GNU/Linux servers.

Software SHOULD be designed in a way that it executes successfully on Debian GNU/Linux stable.[3][4]

The reason is, among others, Debian stable is the only operating system supported by Wikimedia Cloud Services. This is true for both OpenStack and Kubernetes.[3]

To install your dependencies you SHOULD adopt these sources and in this order of preference:

  1. Debian apt with repository "main" or "security"
  2. Debian apt with repository "backports"
  3. Flatpak / snap
  4. Docker

To install your dependencies your SHOULD NOT use these sources:

  • manual download of ".deb" packages from the web
  • download of generic software from the web
  • execution of installation scripts like "wget" in pipe at sudo

Things that MUST NOT be used:

  • adoption of the Debian apt repository "non-free"
  • download of any software that is not released under a Free/Libre and Open Source license

Supported Software[edit]

Example of solutions that MUST NOT be used:

  • adoption of proprietary dependencies (both in content and in software)
  • adoption of proprietary user trackers (such as Google Analytics etc.) - instead you can adopt WMCH Matomo
  • adoption of external web resources (such as Google Font etc.) - instead you can serve them as local resources

Example of solutions that SHOULD NOT be used:

  • adoption of unstable / nightly software as base
  • adoption of esoteric programming languages

Since the operating system adopted in the Wikimedia movement is Debian stable, there are well-known stable versions that SHOULD be supported.

Your server-side software SHOULD NOT be incompatible with these versions:

Your client-side software SHOULD NOT be incompatible with these versions:

In addition to the above software versions, please note that further software versions, officially supported at least by Wikimedia Toolforge, are listed in the following page:

If these versions are too outdated for you, as already mentioned, check the next section for some acceptable workarounds.

Editors[edit]

In order to efficiently develop and write code, tools like the basic code editors or the full-fledged IDEs are an absolute necessity. There are hundreds of tools out there, but we can suggest a couple as industry-standard suggestions:

Software Versions Workarounds[edit]

The indicated versions could be obsolete for your needs. In that case you can follow one of these accepted workarounds:

These and similar workarounds allow flexibility with Debian stable compatibility, still with sources that are trusted and has security updates.

Infrastructure[edit]

Quick summary of the main available infrastructures to be know:

Wikimedia Foundation Infrastructure[edit]

Wikimedia Foundation already provides a structured, modern infrastructure, without costs to the end-users, able to minimize fragmentation, costs, complexity, gatekeeping. This infrastructure also provides effective user management, so, more efficient collaboration.

Here is an overview of the two separate types of access, and what they provide:

Meta-wiki provides a global account to access collaborative public access tools, including:

Wikipedia
the Free encyclopedia
Wikidata
The largest collaborative database of structured data
MediaWiki.org
website documenting MediaWiki (the software running Wikipedia)
Wikimedia Phabricator
collaborative bug tracker and more
and more

Wikitech provides a separate account, to access restricted technical tools, including:

Wikimedia GitLab
code hosting with git
Wikimedia Cloud services
virtual private servers (VPS) on OpenStack (PaaS)
Wikimedia Horizon
OpenStack administration panel
Wikimedia Toolforge
shared hosting based on Kubernetes (IaaS)
and more

Note that all of these services are hosted on machines physically controlled by the Wikimedia Foundation. It is not normally required in any way to adopt third party services like SaaSS. For example, when "GitLab" is mentioned, we normally mean https://gitlab.wikimedia.org/ and not https://gitlab.com/ etc.

Wikimedia CH Infrastructure[edit]

The infrastructure of Wikimedia CH is Switzerland-based and it's completely separated from the one of Wikimedia Foundation.

Infrastructure Comparison[edit]

This table helps in finding the infrastructure best suited for your project. In short:

#Wikimedia Foundation Infrastructure #Wikimedia CH Infrastructure
Easy to add and remove users and set access roles Yes No
Available to the entire Tech community Yes No
OpenStack and Kubernetes Yes No
May host data to be protected No Yes
Supports different GNU/Linux distros than Debian No Yes
Primary Data Center United States Switzerland

Hosting[edit]

This table summarizes suitable hosting solutions according to your need:

Need Implemented with Platform to be adopted Platform owner
shared hosting Kubernetes Wikimedia Toolforge #Wikimedia Foundation Infrastructure
VPS OpenStack Wikimedia Cloud
VPS hypervisor WMCH internal cloud #Wikimedia CH Infrastructure

To have a more comprehensive overview, check these resources:

Here some use-cases:

  • hosting a bot operating in read/write mode on wiki contents
    → Toolforge
  • hosting a web tool useful for Wikimedia purposes with stable versions of PHP/Python/Ruby on Debian:
    → Toolforge
  • hosting a web tool useful for Wikimedia purposes with recent software versions on Debian:
    → Cloud VPS
  • hosting a tool with custom software on custom GNU/Linux:
    → WMCH internal cloud

Coding Conventions[edit]

Any new project SHOULD mention its Coding Conventions to help newcomers in the onboarding. In this way, contributors don't need to inspect random portions of code to have a guess.

For MediaWiki extensions and gadgets, the project SHOULD adopt the following Coding Conventions:

If your project it's based on another software that already has different Coding Conventions, your project SHOULD adopt these.

A new project SHOULD adopt simple and libre tools to help newcomers in adopting the Coding Conventions successfully and without frictions. For example adopting PHP CodeSniffer etc.[5]

Documentation[edit]

People in the development team are the ultimate experts in their own creation. This is why we need your help to take care of the documentation.

The purpose of documentation is not to write it but to read it.

A good documentation helps in avoiding to abandon a project, or rewrite it from scratch, due to lack of shared knowledge.

The documentation must be released under a free license. Suggestions:

Sysadmin Documentation[edit]

We need your help to create a Sysadmin Documentation. This documentation will be useful to future GNU/Linux system engineers handling your service. The sysadmin documentation should be in English.

The goal is understanding how to (re)create a testing and a production environment and how to update, backup and restore.

Usually useful information to mention:

  • software dependencies (packages to be installed in a new minimal Debian GNU/Linux stable)
  • installation instructions (what apt command, etc.)
  • configuration instructions (which system configuration files should be changed, etc.)
  • references to other applications, repositories and documentation related to or useful for the deploy
  • log file paths (those relevant for investigating application issues)
  • Unix users in play (perhaps the app uses system users such as www-data or has custom users)
  • system daemons related to the application (e.g. own custom systemd units, mention other services in use such as mariadb, apache2, nginx, etc.)
  • listening ports (TCP/UDP) and from which app component
  • security instructions
    • which not paths should be externally exposed (example: ./my-temp etc.)
  • hardening instruction
    • paths that must be writable by the app (and thus assigned to a possible my-app user) during its normal operation (example: ./my-upload, ./my-tmp, etc.)
    • paths that not should be writable by the app (and therefore assignable to root) (example: all executable files, ./my-conf, etc.).
  • update notes
    • application update procedure
  • backup and restore procedure
    • where on the filesystem are the user data to be preserved (example: ./my-upload etc.)
    • which databases should be preserved
    • how to enable any maintenance mode

Not all of these points may be relevant. If you feel something is missing, better add it.

The documentation must not contain any secret or password.

The Sysadmin Documentation can be a section in the README file in the project repository.

Development Documentation[edit]

A short documentation in English describing the structure of the project is useful to help other technical people to approach, orient and contribute to the software.

Information that should be covered:

  • purpose of the project/challenges encountered
  • structure of the project (to navigate the directories)
  • how to test the code locally
  • how to configure the application (testing / production)
  • where to find application logs, how to examine

A section called "Development Documentation" in your README file of your project can be a good starting point to help other developers.

User Documentation[edit]

Any good software has good User Documentation. The User Documentation allows end-users to master the software.

Tips:

  • start the draft in English, especially if you plan to make it multi-language in the future
  • it is also okay to write it just in the language of your main destination community
  • start the draft without paying too much attention to formatting (example: OK an Etherpad, a wiki, a README, ...)
  • avoid proprietary tools from the beginning (example: avoid Google Docs)

Note: the user documentation is usually improved by non-technical users. So, it's probably better to adopt a wiki, than a README on git.

You can omit obvious details. For example you can omit steps already covered by in-application wizards, etc.

Server Inventory[edit]

Any server that is not inventoried sufficiently may risk elimination by the Wikimedia Foundation[6]

  • verify that adopted servers are well-known and mentioned in the documentation
  • check that any custom domain is mentioned as well
  • check if the documentation mentions the "The Cloud VPS Instance lifecycle" sufficiently

Maintenance[edit]

Until the end of their assignment your team should take care of the application maintenance. Examples:

  • take care of the initial setup
  • perform routine maintenance to keep it running
  • apply security updates on application dependencies
  • apply security operating system updates (when applicable)
  • verify that the #Backup procedures are working

Insights:

If the application is already in production:

  • schedule and communicate the intervention windows required to perform maintenance activities that may cause downtime (polite 6-hour notice)

Backup[edit]

At the beginning of the assignment, the team sets up a simple automatic backup plan, first of all storing a copy on the same server where the application is located, to implement a first on-site backup.

Purpose of on-site backup: to allow developers to restore data from a point in time before a (their?) mistake, independently and quickly.

Minimum and recommended on-site backup parameters:

  • Data to be saved: the minimum data needed to do a project restore
    • Examples you can include: databases, private application configurations, user uploads, application logs of the day, ...
    • Examples you can exclude: operating system files, files already published elsewhere (git), caches, logs already old, ...
  • Frequency: every night
  • Time of day: the night between 01:00 and 04:00 in UTC+1 (Switzerland, Geneve time)
  • Data retention: 24 hours (one copy only, each new backup overwrites the oldest)

Once implemented, the backup should be supervised at least until the end of the assignment.

Tip: it it's useful, a simple on-site backup can be done thanks to a simple crontab line, using simple tools like mysqldump and/or rsync etc.

The backup directory must not be accessible to the public or to users that are not trusted.

Suggested destination for your on-site backup in your VPS: 

/var/backups/wmch/$HOSTNAME/daily/files...

Suggested permissions: chown app:app with chmod o= ....

An on-site backup is not sufficient. It is just the first step to quickly setup an off-site backup.

Verify that both the backup procedure and its restore are well-defined in the #Sysadmin Documentation.

Code of Conduct[edit]

You (and your team) accept the following code of conducts:

In short: be nice with others.

Terms of Service[edit]

You (and your team) accept the Wikimedia Foundation Terms of Service:

Software License[edit]

Any new software created for Wikimedia CH and to be used in Wikimedia projects must be released with a Free/Libre and Open Source license.

If you work in a company, be sure that the person in charge of your company allows you to release such software. Get it written down. Details:

Offboarding[edit]

Prior to the conclusion of a team member's assignment, that person follow this checklist:

  1. update the Team Documentation to reflect the role change
  2. communicate the accounts that need to be deactivated
  3. communicate the list of personal information that should be removed (not guaranteed to be removed)
  4. contribute to the relevant beautiful #Documentation

Communication[edit]

A good Communication helps users to be aware of development directions. An optimal Communication helps the team in avoiding design mistakes.

Let's start by saying that this is not that easy, since some volunteers could create a controversy if they are not involved in early phases, while some other people just want to choose to be not involved.

A Communication compromise is necessary since the community is big. Some volunteers are conservative, since the Wikimedia platforms they use are almost assimilated as a working desk, and any change can waste their time and create frustration.

The development team, on the other hand, should be able to have creative and positive space to deliver something new.

Communicate Early[edit]

Don't wait the final project conclusion to communicate progresses. This is important also because many projects will be probably never declared as completed. You might be surprised how long your software might last (as example, let's mention Wikipedia).

Do you have a new project challenge? do you have a new progress? do you have a new important bug? That can be a good moment to propose a date for a quick meeting to show that. It is not required to plan a one-hour presentation each day or each week. However, it would be a mistake not to dedicate five-minute to share your screen sometime.

Online meetings should be open to the public, in order to allow some people at least to join and listen. For this reason, the video conferencing platform should be libre (example: Jitsi, BigBlueButton, etc.) otherwise, technical contributors may be excluded and there wouldn't be much benefit.

Contact / Questions[edit]

If something is not clear, please share your opinion in the talk page or contact us:

https://wikimedia.ch/en/contact/

Note[edit]

  1. ↑ https://phabricator.wikimedia.org/diffusion/
  2. ↑ https://gitlab.wikimedia.org/repos/?sort=stars_asc
  3. ↑ a b wikitech:Debian
  4. ↑ https://wiki.debian.org/DebianStable
  5. ↑ https://github.com/squizlabs/PHP_CodeSniffer
  6. ↑ (English) m:wikitech:Help:Cloud VPS Instances#The Cloud VPS Instance lifecycle but also from Wikimedia CH.
    "Instances will be removed for projects that have been determined inactive."


Retrieved from "https://meta.wikimedia.org/w/index.php?title=Wikimedia_CH/Software_Development_Guidelines&oldid=25436241"