Jump to content

Research:Cloud Services Annual Survey/2022

From Meta, a Wikimedia project coordination wiki
Tracked in Phabricator:
Task T322500 open

The 2022 survey collected feedback from Toolforge project members and Cloud VPS project admins on how these services can be improved to meet their development and maintenance needs, as well as the user experience.



The participants of the survey are members of the Wikimedia Toolforge project and Cloud VPS project administrators. A total of 2553 contributors were reached via email for the survey, we selected those users who opted to receive surveys.

159 email recipients participated in the survey, which means we had a total response rate of 6.2%.



During the first stage of the process, we prepared a questionnaire consisting of 73 questions, covering topics such as basic demographic information, use of the Toolforge and Cloud VPS platforms, documentation, tool maintenance and overall feedback around our services. Most of the questions had a multiple-choice format, the other questions had a free-text response that aimed to give us insight about specific comments on the user's side.

For the implementation, this is the first time we are conducting this survey using LimeSurvey. This has been requested by members of the community for a while(we were previously using Google Forms). The survey was active from November 21st, 2022 to December 20th, 2022.

For the analysis, we retrieved the raw responses from Limesurvey and proceeded to process the data for each question through different cohorts. For the free-text responses, we analyzed each one of them and categorized them according to the most mentioned topics.

Questions and answers


Basic demographic data


Do you use Toolforge or Cloud VPS to run or develop tools?

Answer %
Yes 80.6
No 19.4

How many tools have you developed on Toolforge or Cloud VPS?

Answer %
0-1 33.67
2-4 37.76
5-7 11.22
8+ 17.35

How many tools do you actively maintain in Toolforge or Cloud VPS?

Answer %
0-1 42
2-4 36
5-7 12
8+ 9

How many hours per week do you spend developing or maintaining tools on Toolforge or Cloud VPS?

Hours %
0-1 45.7
2-4 32.1
5-7 7.4
8+ 14.8

Are you currently employed by the Wikimedia Foundation as an employee, contractor, vendor, or intern?

Answer %
Yes 22.9
No 77.1

Additional demographics


What kinds of tools do you work on?

Answers Number of people working on the tool
Web Apps 71
Bots 68
APIs use 31
Microsites 20
Dashboard 19
On-Wiki gadgets 18

Which Wikimedia projects are your tools built to work with?

Projects Count
Wikipedia 69
Wiki Commons 29
Wikidata 23
All Wiki Projects 22
Wikisource 6
Wiktionary 10
Wikiquote 5
Wikinews 5
Wikiversity 5
Wikibooks 4
Wikivoyage 6
Wikispecies 3
Others: Wikitech 3

What storage/caching services do you use when running your tools?

Answer Number
MySQL/MariaDB (Not Including Toolsdb Or The Wiki Replicas) 35
Redis 17
SQLite 15
Other 14
Memcached 2
PostgreSQL 4
None 1

Others included: No storage, Filesystem, JSON on NFS, ElasticSearch, On disk, Blazegraph.



Are you a member of the Toolforge project?

Answer %
Yes 73.3
No 26.7

How many years have you used Toolforge?

Answer %
0-1 18.29
2-4 28.05
5-7 23.17
8+ 30.49

Which programming languages do you use on Toolforge?

Programming language Answers %
Python 3 44.53
PHP 24.09
NodeJS 10.95
Other 6.57
Python 2 5.11
Java 2.92
Perl 2.92
Ruby 2.92
Mono/.NET/C# 0

Others: Rust(5), Bash(1), Dart(1), React/js(1), php8(1)

When you develop a tool, how much of your work developing code to run on Toolforge is done locally on your machine (as opposed to remotely on Toolforge)?

Answer %
Almost all of the work 55.81
More than half of the work 18.61
About half of the work 6.98
Less than half of the work 13.95
I don’t know 4.65

Which source control mechanism do you use to manage your tool's source code?

Answer %
Git 80.23
I Do Not Use Source Control 17.44
Mercurial 0
Subversion 0
Other 2

Others include Gerrit, Wikipedia pages

Cloud VPS


Are you an administrator of one or more Cloud VPS projects?

Answer %
Yes 35.34
No 64.66

How many years have you used Cloud VPS?

Answer %
0-1 18.42
2-4 28.95
5-7 23.68
8+ 28.95

What do you use Cloud VPS for?

Answer %
Hosting one or more tools or other public services such as a web app; bot; dashboard; API; etc. 40.48
Testing and experimenting with software 19.05
Running a backing service (database; cache; etc.) for a tool or other public service 16.67
Running one or more MediaWiki instances 20.24
Conducting data analysis or other large computational tasks 3.57

Do you rely on NFS for accessing the same files across different servers?

Answer %
I do not know 17.39
No 69.57
Yes 13.04



Responses on service quality and support. Compared with 2021, fewer respondents agreed that WMCS services provide high uptime. However, more respondents agreed that "it's easier to run code" on our platforms.

Responses indicating usage of all supports channels decreased compared to last year.

Services provided by Wikimedia Cloud Services, including Toolforge and Cloud VPS, have high uptime.

Excluding "neutral" responses


It is easy to have code run on Toolforge or Cloud VPS.


I feel that I am supported by the Cloud Services team when I contact them via cloud@lists.wikimedia.org, the #wikimedia-cloud IRC channel, or Phabricator.


I receive useful information via the “cloud-announce” and/or “cloud” mailing lists.


Wikimedia Cloud Services is one of many providers of cloud computing resources. What made you decide to choose Wikimedia Cloud Services as opposed to other options?

Reason Number of selections %
Cost 21.03
Ease of use 9.31
Access to Wikimedia-specific resources; including wiki replicas 5.17
Privacy and security considerations 23.1
Philosophical or ideological reasons 17.24
Ease of collaborating with other Wikimedians 18.62
Other 2.07

Community and Support


Do you subscribe to the Cloud-Announce mailing list?

Answer Number of times chosen%
No 40.63
Yes; but I usually don't read messages 13.54
Yes; and I usually read messages 45.83

Do you subscribe to the Cloud mailing list?

Answer Number of times chosen%
No 58.95
Yes; but I usually don't read messages 16.84
Yes; and I usually read messages 24.21

When seeking help from WMCS staff, which support channels do you use?

Answer Number of times chosen%
Phabricator 42.57
IRC 34.46
Mailing list/Other email 14.19
Others 8.11
Zulip 0.68

Others: Telegram(6), Wiki talk pages(3), Slack(2), Discord(1)

When discussing WMCS issues with other volunteers, what channels do you use?

Channel Number of times mentioned %
Phabricator 38.69
IRC 32.12
On-wiki talk pages 27.01
Mailing list/Other email 16.79
Discord 8.76
Other 7.3
Facebook 3.65
Slack 2.92

Other channels included Matrix and Telegram



Responses related to documentation.

Cloud Services documentation is easy to find.

In 2021, we asked "When I look for documentation on Wikitech about Toolforge or Cloud VPS I usually find what I was looking for."


Cloud Services documentation is up-to-date.

In 2021, we asked "The documentation I find on Wikitech about Toolforge or Cloud VPS usually answers my question."


How much do you agree or disagree: Cloud Services documentation is clear.

In 2021, we asked "The documentation I find on Wikitech about Toolforge or Cloud VPS is easy to read and understand."


Free-form responses


The survey included several free-form response sections. Survey participants were told that we would only share their responses in an anonymous form. In order to do this, we read each comment, analyzed them and created a summarized view of their insights. Responses regarding documentation can be found in the Documentation section of this report. In situations, where the comments are brief and concise, they are presented verbatim.

If we could improve one thing in Toolforge in the next year, what should that be?.

  • Support:
    • Make it easier for new users to know how to get access and find information. Documentation improvement (5 comments)
    • Need more tutorials (3 comments)
    • Documentation support in other languages(eg. German)
  • Platform:
    • Being able to bring your own Docker container to deploy to Toolforge Kubernetes (including perhaps a complete MediaWiki package?) would be a massive step up. Otherwise I am pretty satisfied.
    • Logging. Currently transitioning from SGE to k8s. It would be great to have logging sorted out (T301901, T302211) and the new retry policy ready (T315114) (2 comments).
    • Better monitoring and reporting for tools (3 comments).
    • Add k8s images for python 3.11 & 3.10 and pywikibot to toolforge-jobs.
    • PHP8+, readonly wiki replicas should be exposed
    • Add support for running .NET apps
    • Improve Kubernetes cron jobs stability
    • Buildpacks will make things easier. Custom image support (4 comments)
    • Improve Kubernetes support (2 comments)
  • Workflow:
    • Improved login. Better SSH experience (3 comments).
    • Better accounts management for multiple maintainers.

Is there anything you have tried but not been able to do in Toolforge?

  • Monitoring tool performance and uptime (3 comments)
  • GPU-intensive tasks (eg. machine learning, computer vision). (2 comments)
  • Running PHP8+
  • Running .NET tools
  • Porting a tool written in multiple languages from the Grid to Kubernetes

If we could improve one thing in Cloud VPS in the next year, what should that be?

  • Easier setup for making backups (3 comments)
  • Alerting and monitoring (2 comments)
  • More tutorials and examples. (2 comments)

If you would like to share more about your experience finding, reading, or maintaining technical documentation on Wikitech, please do so here.

  • Typically what I find is clear and readable, and it has been getting better.
  • Documentation is mostly okay and I remember it has not always been that way, so kudos to whoever helped making it what it is today. There might be some pages that could be merged into some others to avoid having the same topic discussed in several places, but even that, I'm not really sure. I think having a language/runtime-agnostic k8s image would be clearer for people doing Rust/C++/whatever that using the golang image for that (it works very well, but it's not obvious at first).
  • More examples around applying puppet classes to CloudVPS instances would be nice.
  • Existing documentation is good but more tutorials/templates for types of projects would be nice.
  • The documentation tens to lack adequate detail and require a lot of human interaction rather.

If you participate in existing WMCS-related communities outside of email, IRC, and phabricator, please tell us about them!

  • Telegram groups (3 comments).
  • Discord: (6 comments).
    • Polish community Discord server "Wikimedianie".
    • Discord server for specific communities.
  • On-wiki discussions (2 comments)
  • Twitter/Mastodon

If you have any other comments, please share them with us here.

We saw some users showing appreciation and others also gave various feedbacks and suggestions.

  • Conduct more technical training in the Asia region.
  • Thanks for using Limesurvey!

If you would like to share more about your experience as a tool maintainer, please do so here.

  • Kubernetes on Toolforge is a welcome addition (2 comments)
  • Sometimes, getting support from the WMF staff can be frustrating

Thank you


We appreciate and value your feedback since it makes it possible for us to improve our services and get to know more about your needs. Thank you for participating in our 2022 survey! See you in 2023 :)