Dicoado/Technical documentation

This is the public technical documentation page for Dicoado.

Visibility of this document

This page contains public information, safe to everyone.

In compliance with Wikimedia expectations and guidelines, we only use open source software, so, everyone in the world already know the structure of our system. Most information here can already be guess by malicious users, so, deleting this page, or omitting some sections, will not increase security in the system.

This is known as the Kerckhoffs's principle. In short, omitting technical details would just give a feeling of false security.

Thanks for any improvement in this page! \o/

Team

These are the authorized persons and their role and their #SSH accounts:

User:DSwissK
Main volunteer wiki administrator and main project contact.

vivian: personal Unix account, sudoers

www-dicoado: access for Filezilla, not sudoers
Nicolas Nallet
wikivalley-nicolas - working Unix account, sudoer
Sébastien Beyou
wikivalley-sebastien - working Unix account, sudoer
User:ValerioBoz-WMCH - staff system administrator (main on-request sysadmin contact)
valerio-bozzolan - volunteer and working Unix account, sudoer
User:Valerio Bozzolan - volunteer system administrator (when not in working hours)
valerio-bozzolan //
rififi - volunteer system administrator, historical great tech contributor
rififi - personal unix account, sudoers

Special thanks to DSwissK and rififi for all these great community efforts \o/

Server info

Name	`wmch-dicoado1`
Operating System	Debian GNU/Linux bullseye
LAN IP	192.168.128.35
WAN IP	89.207.237.145
RAM	12GB (probably over-scaled)
vCore(s)	4
Partition 1	`/` 16.7 GiB
Partition 2	`/var` 5.89 GiB
Partition 3	`/home` 64 GiB
Special mounts	bind: `/home/www-dicoado/web-resources` mounted on: `/var/www/dicoado`

When in doubt, check this page, visible to members of Wikimedia CH:

https://members.wikimedia.ch/wiki/Infrastructure/Servers

DNS

At the moment the DNS are is managed from an external provider (Infomaniak), by:

User:DSwissK (domain owner)
User:ValerioBoz-WMCH (for extra tech support)

To add additional DNS managers, contact the domain owner.

The above DNS information must be kept in sync with this internal WMCH documentation (that can be seen and updated by any WMCH member + Valerio):

https://members.wikimedia.ch/wiki/Infrastructure/Domains#Infomaniak_Dicoado

Maintained by: User:ValerioBoz-WMCH

Editable by: any WMCH member.

Visible to: any WMCH member.

VPN

SSH is blocked from external traffic.

In order to access the server via SSH, you need access in the "WMCH VPN" first.

Contact User:Ilario or User:ValerioBoz-WMCH or - if you are a member of WMCH - see this page:

https://members.wikimedia.ch/wiki/Infrastructure/VPN

SSH

First, be sure to have #VPN credentials.

Then, if you have no credentials, contact the #Team.

To access the server, use your username, and the server's IP in the #VPN's LAN:

ssh your-user@192.168.128.35

IMPORTANT: We strongly encourage all users to only adopt SSH-key-based authentication. Ideally all users should have no password and only use SSH keys. This allows a level of military security, without any possibility to brute-force. Help: wikitech:help:SSH

IMPORTANT: If you have failed a login, do not retry more than 3 times, or you will be banned by fail2ban.

Uploading files with Filezilla

There is only one authorized user allowed to use Filezilla using SFTP.

Here all the public parameters about that:

Protocol	SFTP
Server	192.168.128.35
Username	`www-dicoado`
Password	Indeed a wiki is not a good place to share a super-secret password. Never share passwords with anyone or with any unauthorized person or a kitten will die right now.

If this user is not allowed to connect, see #Team.

If the user is not allowed to write something, probably the #Hardening script needs a fix.

Log files

See last webserver error log in real-time:

sudo tail -f /var/log/apache2/error.log

See last visits in real-time:

sudo tail -f /var/log/apache2/dicoado.log

IMPORTANT: Never share log info to anyone but team members, since they may contain personal data.

Wiki (fr)

The website fr.dicoado.org has this document root:

/var/www/dicoado/fr/httpdocs

Webserver configurations:

/etc/apache2/sites-available/org-dicoado-fr-ssl.conf
/etc/apache2/my-include/dicoado/dicoado-mediawiki.conf

Homepage (static files)

The website dicoado.org has this document root:

/var/www/dicoado/www/httpdocs

Webserver configurations:

/etc/apache2/my-include/dicoado/dicoado-mediawiki.conf

Production vs Demo

Subject	Production	Demo
Home Path	https://dicoado.org/ `/var/www/dicoado/www`	https://demo.dicoado.org/ `/var/www/dicoado.demo/www`
Wiki Path DB wiki DB Cargo	https://fr.dicoado.org/ `/var/www/dicoado/fr/httpdocs` `nxxs_dicoado_fr` `nxxs_dicoado_cargo_fr`	https://demo.fr.dicoado.org/ `/var/www/dicoado.demo/fr/httpdocs` `nxxs_dicoado_fr_demo` `nxxs_dicoado_cargo_fr_demo`

Procedure to pull production into demo

copy production databases to demo
copy /var/www/dicoado/fr/ to /var/www/dicoado.demo/fr/

TODO:

Conclude this script:

/root/dicoado-demo-pull-production.sh

Procedure to publish demo in production

IMPORTANT:

NEVER COPY DEMO DATABASES IN PRODUCTION.

DEMO DATABASES ARE VERY PROBABLY ALWAYS OUTDATED.

Always copy just demo files in production.

Use this procedure if your files in demo are more updated than production.

This involves to run maintenance operations in the production database.

do a backup of the database in production
stop production services
set production in read-only and/or set production offline and/or maintenance page
copy /var/www/dicoado.demo to /var/www/dicoado
Suggested: rsync --archive --delete
run maintenance scripts in production
sudo -u www-data -s /bin/bash

cd /var/www/dicoado/fr/httpdocs/w/

php maintenance/update.php
on fail:
rollback databases and files from backup
on successs
start production services again

unset production from read-only and/or make it online and/or maintenance page

Maintenance

Maintenance scripts should be executed as the user www-data to avoid permissions issues, e.g.

sudo -u www-data php maintenance/showJobs.php

(Request the rights "sudo to www-data" if your Unix user don’t have it.)

At the moment there is not any particular maintenance mode.

Relevant pages:

mw:Manual:$wgReadOnly

MariaDB

The databases can be accessed and modified in a secure way from command line using standard ways.

Example to see database tables:

sudo mysql nxxs_dicoado_fr
SHOW TABLES;
exit

To backup databases:

sudo mysqldump nxxs_dicoado_fr       > nxxs_dicoado_fr.sql
sudo mysqldump nxxs_dicoado_cargo_fr > nxxs_dicoado_cargo_fr.sql

IMPORTANT: Never expose an SQL dump in a web directory and always restrict file permissions.

To restore a database:

sudo mysql YOUR_DATABASE_NAME < file.sql

IMPORTANT: This will nuke the already-existing database.

Permissions hardening

Ideally, to reduce attack surfaces, all files that are supposed to be read-only by the webserver, should be at least assigned to the user www-dicoado (so are writable during SFTP uploads). Example:

sudo chown www-dicoado: file.txt

Ideally, only files that should be writable by the webserver should be assigned to the webserver user www-data. Example:

sudo chown www-data: file.png

To simplify hardening, there is this script that restores the ideal situation:

sudo /root/dicoado-harden.sh

Be sure to harden the correct thing. Inside the script there is a configuration line. Change that accordingly to your purpose (hardening demo, or hardening production).

Cron jobs

The webserver user has some cron-jobs. To edit them:

sudo -u www-data crontab -e

Example (may be outdated):

# Hourly, at random minute (xx:17)
# Keep just last execution.
 17 *  *   *   *     /var/www/dicoado/fr/cron/hourly.sh

# Daily, at a random hour (23:22)
22 23  *   *   *     /var/www/dicoado/fr/cron/daily.sh

# Monthly, at a random day (2nd) and random time (23:58)
58 23  2   *   *     /var/www/dicoado/fr/cron/monthly.sh

Systemd service

A systemd service is running all the time, so that if you touch a template, it reflects changes immediately and not after an hour.

Example interaction:

systemctl status dicoado-wiki-fr

The related executable file is here:

/opt/wmch-scripts/mediawiki-job.php

Skin Source Code

DicoAdo is based on the skin Foreground but with local modifications.

See the source code of the DicoAdo's skin on Wikimedia GitLab:

https://gitlab.wikimedia.org/repos/wikimedia-ch/dicoado/dicoado-skin

SSL Certificates

DicoAdo relies on Let's Encrypt for certificates, using the Debian package certbot. We use the "webroot" renewal method.

Every certificate can be renewed with one of these (check your case):

certbot certonly --webroot --webroot-path /var/www/html -d YOURDOMAIN
certbot certonly --webroot --webroot-path /var/www/html -d YOURDOMAIN -d www.YOURDOMAIN

The webroot renewal method will create a temporary file here:

/var/www/html/.well-known/acme-challenge/<tempfile>

Your certificate will be released only if this URL matches:

http://YOURDOMAIN/.well-known/acme-challenge/<tempfile>

Note that the webserver needs a reload after every renew.

Mailboxes

The documentation about DicoAdo mailboxes is available here:

https://members.wikimedia.ch/wiki/Infrastructure/Emails#Mailboxes_in_dicoado.org

Maintained by: User:ValerioBoz-WMCH

Editable by: any WMCH member.

Visible to: any WMCH member.

Backups

IMPORTANT: At the moment WMCH have not automatic snapshots. They need to be requested by the volunteer project leader. WMCH has to verify budget accordingly. Contact User:Ilario for budget verification.

IMPORTANT: At the moment WMCH has not offsite backups. They need to be managed by the volunteer project leader. WMCH probably cannot provide this kind of service at the moment.

The volunteer project leader is supposed to be the only one to have off-site backups.

Wikimedia CH can be contacted to activate a single snapshot and eventually restore it (in case, budget to be verified).

Wikimedia CH can be contacted to evaluate automatic daily snapshots (in case, budget to be verified).

Visibility of this document

Team

Server info

DNS

VPN

SSH

Uploading files with Filezilla

Log files

Wiki (fr)

Homepage (static files)

Production vs Demo

Procedure to pull production into demo

Procedure to publish demo in production

Maintenance

MariaDB

Permissions hardening

Cron jobs

Systemd service

Skin Source Code

SSL Certificates

Mailboxes

Backups

See also