Research:Cloud Services Annual Survey/2022
The 2022 survey collected feedback from Toolforge project members and Cloud VPS project admins on how these services can be improved to meet their development and maintenance needs, as well as the user experience.
Participants[edit]
The participants of the survey are members of the Wikimedia Toolforge project and Cloud VPS project administrators. A total of 2553 contributors were reached via email for the survey, we selected those users who opted to receive surveys.
159 email recipients participated in the survey, which means we had a total response rate of 6.2%.
Methodology[edit]
During the first stage of the process, we prepared a questionnaire consisting of 73 questions, covering topics such as basic demographic information, use of the Toolforge and Cloud VPS platforms, documentation, tool maintenance and overall feedback around our services. Most of the questions had a multiple-choice format, the other questions had a free-text response that aimed to give us insight about specific comments on the user's side.
For the implementation, this is the first time we are conducting this survey using LimeSurvey. This has been requested by members of the community for a while(we were previously using Google Forms). The survey was active from November 21st, 2022 to December 20th, 2022.
For the analysis, we retrieved the raw responses from Limesurvey and proceeded to process the data for each question through different cohorts. For the free-text responses, we analyzed each one of them and categorized them according to the most mentioned topics.
Questions and answers[edit]
Basic demographic data[edit]
Do you use Toolforge or Cloud VPS to run or develop tools?
| Answer | % |
|---|---|
| Yes | 80.6 |
| No | 19.4 |
How many tools have you developed on Toolforge or Cloud VPS?
| Answer | % |
|---|---|
| 0-1 | 33.67 |
| 2-4 | 37.76 |
| 5-7 | 11.22 |
| 8+ | 17.35 |
How many tools do you actively maintain in Toolforge or Cloud VPS?
| Answer | % |
|---|---|
| 0-1 | 42 |
| 2-4 | 36 |
| 5-7 | 12 |
| 8+ | 9 |
How many hours per week do you spend developing or maintaining tools on Toolforge or Cloud VPS?
| Hours | % |
|---|---|
| 0-1 | 45.7 |
| 2-4 | 32.1 |
| 5-7 | 7.4 |
| 8+ | 14.8 |
Are you currently employed by the Wikimedia Foundation as an employee, contractor, vendor, or intern?
| Answer | % |
|---|---|
| Yes | 22.9 |
| No | 77.1 |
Additional demographics[edit]
What kinds of tools do you work on?
| Answers | Number of people working on the tool |
|---|---|
| Web Apps | 71 |
| Bots | 68 |
| APIs use | 31 |
| Microsites | 20 |
| Dashboard | 19 |
| On-Wiki gadgets | 18 |
Which Wikimedia projects are your tools built to work with?
| Projects | Count |
|---|---|
| Wikipedia | 69 |
| Wiki Commons | 29 |
| Wikidata | 23 |
| All Wiki Projects | 22 |
| Wikisource | 6 |
| Wiktionary | 10 |
| Wikiquote | 5 |
| Wikinews | 5 |
| Wikiversity | 5 |
| Wikibooks | 4 |
| Wikivoyage | 6 |
| Wikispecies | 3 |
| Others: Wikitech | 3 |
What storage/caching services do you use when running your tools?
| Answer | Number |
|---|---|
| MySQL/MariaDB (Not Including Toolsdb Or The Wiki Replicas) | 35 |
| Redis | 17 |
| SQLite | 15 |
| Other | 14 |
| Memcached | 2 |
| PostgreSQL | 4 |
| None | 1 |
Others included: No storage, Filesystem, JSON on NFS, ElasticSearch, On disk, Blazegraph.
Toolforge[edit]
Are you a member of the Toolforge project?
| Answer | % |
|---|---|
| Yes | 73.3 |
| No | 26.7 |
How many years have you used Toolforge?
| Answer | % |
|---|---|
| 0-1 | 18.29 |
| 2-4 | 28.05 |
| 5-7 | 23.17 |
| 8+ | 30.49 |
Which programming languages do you use on Toolforge?
| Programming language | Answers % |
|---|---|
| Python 3 | 44.53 |
| PHP | 24.09 |
| NodeJS | 10.95 |
| Other | 6.57 |
| Python 2 | 5.11 |
| Java | 2.92 |
| Perl | 2.92 |
| Ruby | 2.92 |
| Mono/.NET/C# | 0 |
Others: Rust(5), Bash(1), Dart(1), React/js(1), php8(1)
When you develop a tool, how much of your work developing code to run on Toolforge is done locally on your machine (as opposed to remotely on Toolforge)?
| Answer | % |
|---|---|
| Almost all of the work | 55.81 |
| More than half of the work | 18.61 |
| About half of the work | 6.98 |
| Less than half of the work | 13.95 |
| I don’t know | 4.65 |
Which source control mechanism do you use to manage your tool's source code?
| Answer | % |
|---|---|
| Git | 80.23 |
| I Do Not Use Source Control | 17.44 |
| Mercurial | 0 |
| Subversion | 0 |
| Other | 2 |
Others include Gerrit, Wikipedia pages
Cloud VPS[edit]
Are you an administrator of one or more Cloud VPS projects?
| Answer | % |
|---|---|
| Yes | 35.34 |
| No | 64.66 |
How many years have you used Cloud VPS?
| Answer | % |
|---|---|
| 0-1 | 18.42 |
| 2-4 | 28.95 |
| 5-7 | 23.68 |
| 8+ | 28.95 |
What do you use Cloud VPS for?
| Answer | % |
|---|---|
| Hosting one or more tools or other public services such as a web app; bot; dashboard; API; etc. | 40.48 |
| Testing and experimenting with software | 19.05 |
| Running a backing service (database; cache; etc.) for a tool or other public service | 16.67 |
| Running one or more MediaWiki instances | 20.24 |
| Conducting data analysis or other large computational tasks | 3.57 |
Do you rely on NFS for accessing the same files across different servers?
| Answer | % |
|---|---|
| I do not know | 17.39 |
| No | 69.57 |
| Yes | 13.04 |
Services[edit]
Responses on service quality and support. Compared with 2021, fewer respondents agreed that WMCS services provide high uptime. However, more respondents agreed that "it's easier to run code" on our platforms.
Responses indicating usage of all supports channels decreased compared to last year.
Services provided by Wikimedia Cloud Services, including Toolforge and Cloud VPS, have high uptime.
Excluding "neutral" responses
It is easy to have code run on Toolforge or Cloud VPS.
I feel that I am supported by the Cloud Services team when I contact them via cloud@lists.wikimedia.org, the #wikimedia-cloud IRC channel, or Phabricator.
I receive useful information via the “cloud-announce” and/or “cloud” mailing lists.
Wikimedia Cloud Services is one of many providers of cloud computing resources. What made you decide to choose Wikimedia Cloud Services as opposed to other options?
| Reason | Number of selections % |
|---|---|
| Cost | 21.03 |
| Ease of use | 9.31 |
| Access to Wikimedia-specific resources; including wiki replicas | 5.17 |
| Privacy and security considerations | 23.1 |
| Philosophical or ideological reasons | 17.24 |
| Ease of collaborating with other Wikimedians | 18.62 |
| Other | 2.07 |
Community and Support[edit]
Do you subscribe to the Cloud-Announce mailing list?
| Answer | Number of times chosen% |
|---|---|
| No | 40.63 |
| Yes; but I usually don't read messages | 13.54 |
| Yes; and I usually read messages | 45.83 |
Do you subscribe to the Cloud mailing list?
| Answer | Number of times chosen% |
|---|---|
| No | 58.95 |
| Yes; but I usually don't read messages | 16.84 |
| Yes; and I usually read messages | 24.21 |
When seeking help from WMCS staff, which support channels do you use?
| Answer | Number of times chosen% |
|---|---|
| Phabricator | 42.57 |
| IRC | 34.46 |
| Mailing list/Other email | 14.19 |
| Others | 8.11 |
| Zulip | 0.68 |
Others: Telegram(6), Wiki talk pages(3), Slack(2), Discord(1)
When discussing WMCS issues with other volunteers, what channels do you use?
| Channel | Number of times mentioned % |
|---|---|
| Phabricator | 38.69 |
| IRC | 32.12 |
| On-wiki talk pages | 27.01 |
| Mailing list/Other email | 16.79 |
| Discord | 8.76 |
| Other | 7.3 |
| 3.65 | |
| Slack | 2.92 |
Other channels included Matrix and Telegram
Documentation[edit]
Responses related to documentation.
Cloud Services documentation is easy to find.
In 2021, we asked "When I look for documentation on Wikitech about Toolforge or Cloud VPS I usually find what I was looking for."
Cloud Services documentation is up-to-date.
In 2021, we asked "The documentation I find on Wikitech about Toolforge or Cloud VPS usually answers my question."
How much do you agree or disagree: Cloud Services documentation is clear.
In 2021, we asked "The documentation I find on Wikitech about Toolforge or Cloud VPS is easy to read and understand."
Free-form responses[edit]
The survey included several free-form response sections. Survey participants were told that we would only share their responses in an anonymous form. In order to do this, we read each comment, analyzed them and created a summarized view of their insights. Responses regarding documentation can be found in the Documentation section of this report. In situations, where the comments are brief and concise, they are presented verbatim.
If we could improve one thing in Toolforge in the next year, what should that be?.
- Support:
- Make it easier for new users to know how to get access and find information. Documentation improvement (5 comments)
- Need more tutorials (3 comments)
- Documentation support in other languages(eg. German)
- Platform:
- Being able to bring your own Docker container to deploy to Toolforge Kubernetes (including perhaps a complete MediaWiki package?) would be a massive step up. Otherwise I am pretty satisfied.
- Logging. Currently transitioning from SGE to k8s. It would be great to have logging sorted out (T301901, T302211) and the new retry policy ready (T315114) (2 comments).
- Better monitoring and reporting for tools (3 comments).
- Add k8s images for python 3.11 & 3.10 and pywikibot to toolforge-jobs.
- PHP8+, readonly wiki replicas should be exposed
- Add support for running .NET apps
- Improve Kubernetes cron jobs stability
- Buildpacks will make things easier. Custom image support (4 comments)
- Improve Kubernetes support (2 comments)
- Workflow:
- Improved login. Better SSH experience (3 comments).
- Better accounts management for multiple maintainers.
Is there anything you have tried but not been able to do in Toolforge?
- Monitoring tool performance and uptime (3 comments)
- GPU-intensive tasks (eg. machine learning, computer vision). (2 comments)
- Running PHP8+
- Running .NET tools
- Porting a tool written in multiple languages from the Grid to Kubernetes
If we could improve one thing in Cloud VPS in the next year, what should that be?
- Easier setup for making backups (3 comments)
- Alerting and monitoring (2 comments)
- More tutorials and examples. (2 comments)
If you would like to share more about your experience finding, reading, or maintaining technical documentation on Wikitech, please do so here.
- Typically what I find is clear and readable, and it has been getting better.
- Documentation is mostly okay and I remember it has not always been that way, so kudos to whoever helped making it what it is today. There might be some pages that could be merged into some others to avoid having the same topic discussed in several places, but even that, I'm not really sure. I think having a language/runtime-agnostic k8s image would be clearer for people doing Rust/C++/whatever that using the golang image for that (it works very well, but it's not obvious at first).
- More examples around applying puppet classes to CloudVPS instances would be nice.
- Existing documentation is good but more tutorials/templates for types of projects would be nice.
- The documentation tens to lack adequate detail and require a lot of human interaction rather.
If you participate in existing WMCS-related communities outside of email, IRC, and phabricator, please tell us about them!
- Telegram groups (3 comments).
- Discord: (6 comments).
- Polish community Discord server "Wikimedianie".
- Discord server for specific communities.
- On-wiki discussions (2 comments)
- Twitter/Mastodon
If you have any other comments, please share them with us here.
We saw some users showing appreciation and others also gave various feedbacks and suggestions.
- Conduct more technical training in the Asia region.
- Thanks for using Limesurvey!
If you would like to share more about your experience as a tool maintainer, please do so here.
- Kubernetes on Toolforge is a welcome addition (2 comments)
- Sometimes, getting support from the WMF staff can be frustrating
Thank you[edit]
We appreciate and value your feedback since it makes it possible for us to improve our services and get to know more about your needs. Thank you for participating in our 2022 survey! See you in 2023 :)