The 2020 survey collected feedback from Toolforge project members and Cloud VPS project administrators on how the services offered can be improved to help their development and maintenance needs.
The participants of the survey are the members of the Wikimedia Toolforge project and Cloud VPS project administrators. 1825 participants were contacted via email. The emails were harvested from the Cloud Services' LDAP system ("developer accounts") and the Wikitech database. The final list was created by finding the union of the members of the Toolforge project and all users with "administrator" rights on one or more Cloud VPS projects. Users who have opted-out of email contact by other users in their Wikitech preferences or the survey specific opt-out page were excluded. Specific opt-outs for the Annual Survey can be made by users at wikitech:Annual Toolforge Survey/Opt out.
137 email recipients participated in the survey which represents 7.5% of those who were contacted.
Web apps (tools that are accessed through a web browser)
77%
Bots (including pywikibot)
63%
APIs for use by other tools
32%
On-wiki gadgets
24%
Microsites (small static websites)
18%
Dashboards and other data visualizations
14%
Extensions
2%
Data analysis
1%
Other
2%
Which Wikimedia projects are your tools built to work with?[edit]
Answer
%
Wikipedia
64%
Wikidata
36%
Wikimedia Commons
29%
All Wikimedia projects
26%
Wikisource
9%
Wiktionary
7%
Wikinews
6%
Wikiquote
6%
Wikibooks
4%
Wikiversity
4%
Wikivoyage
3%
Phabricator
3%
MediaWiki
2%
Wikispecies
2%
Gerrit
1%
Logstash
1%
Meta
1%
Toolforge
1%
Wikitech
1%
Wikimedia Cloud Services is one of many providers of cloud computing resources. What made you decide to choose Wikimedia Cloud Services as opposed to other options?[edit]
Answer
%
Access to Wikimedia-specific resources; including wiki replicas
74%
Cost
60%
Ease of collaborating with other Wikimedians
54%
Philosophical or ideological reasons
47%
Ease of use
34%
Privacy and security considerations
32%
Wikimedia staff
2%
Help with maintenance
1%
Reliability
1%
What storage/caching services do you use when running your tools?[edit]
Answer
%
Mysql/Mariadb (Not Including Toolsdb Or The Wiki Replicas)
About how many years have you used Toolforge?[edit]
Answer
%
1
20%
2-3
23%
4-5
24%
6+
33%
Which programming languages do you use on Toolforge?[edit]
Answer
%
Python 3
67%
Php
44%
Nodejs
16%
Python 2
15%
Java
8%
Perl
6%
Bash
5%
Ruby
3%
Rust
3%
Javascript
3%
Other
8%
When you develop a tool, how much of your work developing code to run on Toolforge is done locally on your machine (as opposed to remotely on Toolforge)?[edit]
Answer
%
Almost all of the work
60%
More than half of the work
14%
About half of the work
7%
Less than half of the work
15%
I don’t know
4%
Which source control mechanism do you use to manage your tool's source code?[edit]
A series of qualitative questions were asked in order to gauge general satisfaction with Toolforge and Cloud VPS.
Services provided by Wikimedia Cloud Services, including Toolforge and Cloud VPS, have high uptime.[edit]
Answer
All
1 Year
2-3 Years
4-5 Years
6+ Years
0-1 Tools
2 Tool
3-6 Tools
7+ Tools
0-1 Maintain
2 Maintain
3-5 Maintain
6+ Maintain
0 Hours
1 Hour
2-3 Hours
4-8 Hours
10+ Hours
Toolforge
Cloud VPS
Toolforge+Cloud VPS
Tools
No tools
Agree/Strongly Agree
82%
100%
83%
65%
92%
83%
92%
78%
95%
83%
89%
82%
93%
64%
90%
96%
80%
83%
80%
82%
94%
86%
59%
Disagree/Strongly Disagree
5%
0%
13%
8%
3%
4%
4%
5%
5%
6%
7%
0%
7%
14%
3%
4%
0%
8%
7%
0%
3%
5%
6%
It is easy to have code run on Toolforge or Cloud VPS.[edit]
Answer
All
1 Year
2-3 Years
4-5 Years
6+ Years
0-1 Tools
2 Tool
3-6 Tools
7+ Tools
0-1 Maintain
2 Maintain
3-5 Maintain
6+ Maintain
0 Hours
1 Hour
2-3 Hours
4-8 Hours
10+ Hours
Toolforge
Cloud VPS
Toolforge+Cloud VPS
Tools
No tools
Agree/Strongly Agree
63%
76%
68%
50%
69%
52%
69%
65%
80%
49%
79%
71%
73%
47%
69%
77%
47%
75%
59%
55%
81%
66%
41%
Disagree/Strongly Disagree
10%
0%
20%
12%
8%
16%
12%
8%
0%
16%
7%
7%
0%
27%
5%
12%
0%
8%
9%
36%
0%
9%
18%
I feel I am supported by the Cloud Services team when I contact them via cloud@lists.wikimedia.org, the #wikimedia-cloud IRC channel, or Phabricator.[edit]
Answer
All
1 Year
2-3 Years
4-5 Years
6+ Years
0-1 Tools
2 Tool
3-6 Tools
7+ Tools
0-1 Maintain
2 Maintain
3-5 Maintain
6+ Maintain
0 Hours
1 Hour
2-3 Hours
4-8 Hours
10+ Hours
Toolforge
Cloud VPS
Toolforge+Cloud VPS
Tools
No tools
Agree/Strongly Agree
64%
67%
61%
64%
76%
30%
60%
72%
100%
39%
73%
77%
93%
54%
57%
80%
73%
58%
60%
64%
84%
65%
59%
Disagree/Strongly Disagree
2%
5%
4%
0%
0%
9%
0%
0%
0%
6%
0%
0%
0%
8%
0%
0%
7%
0%
3%
0%
0%
2%
0%
I receive useful information via cloud-announce / cloud mailing list.[edit]
Answer
All
1 Year
2-3 Years
4-5 Years
6+ Years
0-1 Tools
2 Tool
3-6 Tools
7+ Tools
0-1 Maintain
2 Maintain
3-5 Maintain
6+ Maintain
0 Hours
1 Hour
2-3 Hours
4-8 Hours
10+ Hours
Toolforge
Cloud VPS
Toolforge+Cloud VPS
Tools
No tools
Agree/Strongly Agree
53%
48%
57%
63%
57%
39%
68%
57%
72%
36%
81%
56%
79%
43%
47%
72%
67%
75%
49%
27%
77%
59%
18%
Disagree/Strongly Disagree
7%
10%
4%
8%
3%
13%
4%
3%
6%
11%
4%
4%
0%
14%
8%
4%
0%
0%
10%
0%
0%
6%
12%
Cloud Services documentation is easy-to-find.[edit]
Answer
All
1 Year
2-3 Years
4-5 Years
6+ Years
0-1 Tools
2 Tool
3-6 Tools
7+ Tools
0-1 Maintain
2 Maintain
3-5 Maintain
6+ Maintain
0 Hours
1 Hour
2-3 Hours
4-8 Hours
10+ Hours
Toolforge
Cloud VPS
Toolforge+Cloud VPS
Tools
No tools
Agree/Strongly Agree
59%
48%
64%
65%
67%
48%
65%
62%
80%
49%
71%
68%
73%
50%
69%
65%
53%
67%
58%
50%
66%
63%
29%
Disagree/Strongly Disagree
16%
5%
28%
15%
8%
41%
12%
5%
5%
28%
14%
0%
13%
31%
13%
8%
13%
17%
19%
20%
3%
15%
18%
Cloud Services documentation is comprehensive.[edit]
The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form.
If we could improve one thing in Toolforge in the next year, what should that be?[edit]
Responses to this question were varied, here is a summary of the comments clustered in different areas:
Documentation:
Improving documentation for beginners
Updating and removing outdated documentation
Better documentation for batch jobs / cron workflows
Support:
More responsive help desk
Help to beginners, people getting started
Centralized support site/forum
Workflow:
Deleting tools
Automated deployments
Development / Production environment parity for developing
Platform:
Replicas
Local access
Cross DB joins
Reliability
Stability (breaking changes)
Performance
Kubernetes
Usage for jobs is hard
Memory quotas
Tooling
NFS performance
Language / Version support
Node.js versions
Ruby on Rails support
Rust support
Docker support
Logging
Logs and emails from cron jobs
Monitoring
Health checks and alerts
Show general service availability
If we could improve one thing in Cloud VPS in the next year, what should that be?[edit]
Thank you for those who answered positive comments and praise, it is very much appreciated.
There are less responses in this question, so the summary will be less detailed.
Improve documentation
Better deployment workflows
Docker support
Managed Kubernetes clusters
Backups
Monitoring services
If you would like to share more about your experience finding, reading, or maintaining technical documentation on Wikitech, please do so here.[edit]
Hard to understand
Beginner documentation is very hard to understand
Needs more graphics
Cloud VPS docs are harder to understand than Toolforge's
Lacking
What should I use? What is supported?
Reference lists for features, instead of all tutorials
More documentation about effective workflows
Out of date
Structure
Self-study is very hard, complicated to navigate the docs (guided-study is not so bad)
i18n
If you have any other comments, please share them with us here.[edit]
There were many praise responses, thanks a lot, they are very much appreciated.
Besides those, there were some comments about language/framework support, using propietary software for this surveys instead of OSS, and a variety of comments about topics mentioned in the other freeform questions.
Is there anything you have tried but not been able to do in Toolforge?[edit]
The responses here were very varied, the ones mentioned several times were:
Improve language support and use newer versions of languages and applications
Memory intensive work on Toolforge. People end up moving to Cloud VPS for these
Remote access issues (SSH, SFTP, etc.)
Analysis work and very CPU/IO intensive programs
Problems with blocked IPs
The rest were single comments and in interest of anonymity won't be detailed here.
.svg)
.svg)
.svg)
.svg)
.svg)
