Research:Cloud Services Annual Survey/2019

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search
Tracked in Phabricator:
task T238181

The 2019 survey collected feedback from Toolforge project members and Cloud VPS project administrators on how the services offered can be improved to help their development and maintenance needs.

Participants[edit]

The participants of the survey are the members of the Wikimedia Toolforge project and Cloud VPS project administrators. 1857 participants were contacted via email. The emails were harvested from the Cloud Services' LDAP system ("developer accounts") and the Wikitech database. The final list was created by finding the union of the members of the Toolforge project and all users with "administrator" rights on one or more Cloud VPS projects. Users who have opted-out of email contact by other users in their Wikitech preferences or the survey specific opt-out page were excluded. Specific opt-outs for the Annual Survey can be made by users at wikitech:Annual Toolforge Survey/Opt out.

108 email recipients participated in the survey which represents 6% of those who were contacted.

Dates[edit]

The 2019 survey was active between 2019-11-25 and 2019-12-13.

Questions and answers[edit]

Basic demographics[edit]

About how many years have you used Toolforge or Cloud VPS?[edit]

2019-cloud-survey-01-cohort-age.svg
Cohort Percentage
1 Year 20.7%
2-3 Years 24.1%
4-5 Years 27.6%
6+ Years 27.6%

How many tools have you developed on Toolforge or Cloud VPS?[edit]

2019-cloud-survey-02-cohort-develop.svg
Cohort Percentage
0-1 Tools 25.3%
2 Tools 27.6%
3-6 Tools 25.3%
7+ Tools 21.8%

How many tools do you actively maintain in Toolforge or Cloud VPS?[edit]

2019-cloud-survey-03-cohort-maintain.svg
Cohort Percentage
0-1 Maintain 31.4%
2 Maintain 29.1%
3-5 Maintain 23.3%
6+ Maintain 16.3%

About how many hours per week do you spend developing or maintaining tools on Toolforge or Cloud VPS?[edit]

2019-cloud-survey-04-cohort-hours.svg
Cohort Percentage
0 Hours 22.1%
1 Hour 21.8%
2-3 Hours 12.6%
4-8 Hours 19.5%
10+ Hours 22.1%

Which Cloud Services platforms do you use?[edit]

2019-cloud-survey-05-cohort-platform.svg

Derived from answers to the questions "Are you a member of the Toolforge project?" and "Are you an administrator of one of more Cloud VPS projects?"

Cohort Percentage
Toolforge 61.1%
Cloud VPS 5.6%
Both 24.1%

Do you use Toolforge or Cloud VPS to run or develop tools?[edit]

2019-cloud-survey-06-cohort-tools.svg
Cohort Percentage
No 13.9%
Yes 86.1%

Additional demographics[edit]

What kinds of tools do you work on?[edit]

2019 Cloud Services Survey - What kinds of tools do you work on?.svg
Tool kind Percentage
Bots 75.0%
Webapps 72.8%
Apis 30.4%
Gadgets 25.0%
Dataviz 22.8%
Microsites 17.4%
Abuse Ai 1.1%
Ai Systems Deployment 1.1%
Analytics 1.1%
Irc Bot 1.1%
Reports 1.1%
Wikidata Import Scripts 1.1%

Which Wikimedia projects are your tools built to work with?[edit]

2019 Cloud Services Survey - Which Wikimedia projects are your tools built to work with ?.svg
Project Percentage
Wikipedia 76.1%
Wikidata 35.9%
Wikimedia Commons 33.7%
All Wikimedia projects 26.1%
Wikisource 9.8%
Wiktionary 8.7%
Wikibooks 7.6%
Wikiquote 6.5%
Wikinews 5.4%
Wikivoyage 5.4%
Wikiversity 4.4%
MediaWiki 2.2%
Wikispecies 2.2%
IRC 1.1%
Wikitech 1.1%

What storage/caching services do you use when running your tools?[edit]

2019 Cloud Services Survey - What storage caching services do you use when running your tools.svg
Yes/No Percentage
MySQL/MariaDB 73.8%
Redis 26.2%
SQLite 16.4%
MongoDB 4.9%
PostgreSQL 3.3%
Static files 3.3%
ToolsDB 3.3%
Custom 1.6%
ElasticSearch 1.6%
Filesystem 1.6%
Memcached 1.6%
Wiki Replicas 1.6%

Are you a member of the Toolforge project?[edit]

2019 Cloud Services Survey - Are you a member of the Toolforge project.svg
Yes/No Percentage
Yes 85.2%
No 14.8%

Which programming languages do you use on Toolforge?[edit]

2019 Cloud Services Survey - Which programming languages do you use on Toolforge?.svg
Project Percentage
Python 3 68.1%
PHP 37.4%
Python 2 26.4%
NodeJS 15.4%
Other 13.2%
Perl 11.0%
Javascript 5.5%
Java 4.4%
Mono/.NET/C# 4.4%

When you develop a tool, how much of your work developing code to run on Toolforge is done locally on your machine (as opposed to remotely on Toolforge)?[edit]

2019 Cloud Services Survey - What percentage of your work is done locally.svg
Answer Percentage
Almost all of the work 54.4%
Less than half of the work 16.3%
More than half of the work 14.1%
About half of the work 8.7%
I don't know 6.5%

Which source control mechanism do you use to manage your tool's source code?[edit]

2019 Cloud Services Survey - Which source control mechanism do you use to manage your tool's source code.svg
Answer Percentage
Git 84.8%
I do not use source control 15.2%

If we could improve one thing in Toolforge in the next year, what should that be?[edit]

This a curated summary of the different topics mentioned in the comments. Further commentary in the last section about Freeform comments.

2019 Cloud Services Survey - If we could improve one thing in Toolforge in the next year, what should that be.svg
Topic Mentions in answers
Documentation 13
Support 10
Languages and libraries 8
Performance 8
Storage 7
Deployment 4
Network 4
Docker 3
Kubernetes 3
Logs 3
Monitoring 3
Getting started 2
Local development with Toolforge resources / data 2
Security (subdomains) 2
SSH 2
Other 11

Are you an administrator of one of more Cloud VPS projects?[edit]

2019 Cloud Services Survey - Are you an administrator of one of more Cloud VPS projects.svg
Answer Percentage
Yes 29.6%
No 70.4%

About how many years have you used Cloud VPS?[edit]

2019 Cloud Services Survey - How many years have you used Cloud VPS.svg
Cohort Percentage
1 Year 7.7%
2-3 Years 38.5%
4-5 Years 26.9%
6+ Years 26.9%

What do you use Cloud VPS for?[edit]

2019 Cloud Services Survey - What do you use Cloud VPS for.svg
Answer Percentage
Hosting one or more tools or other public services such as a web app; bot; dashboard; API; etc. 75.0%
Testing and experimenting with software 50.0%
Running one or more MediaWiki instances 37.5%
Running a backing service (database; cache; etc.) for a tool or other public service 34.4%
Conducting data analysis or other large computational tasks 15.6%

Do you rely on NFS for accessing the same files across different servers?[edit]

2019 Cloud Services Survey - Do you rely on NFS for accessing the same files across different servers.svg
Answer Percentage
No 61.3%
Yes 38.7%

If we could improve one thing in Cloud VPS in the next year, what should that be?[edit]

This a curated summary of the different topics mentioned in the comments. Further commentary in the last section about Freeform comments.

2019 Cloud Services Survey - If we could improve one thing in Cloud VPS in the next year, what should that be.svg
Topic Mentions in answers
Configuration Management 4
Storage 3
Maintenance 2
Quotas 2
Deployment 1
Documentation 1
Monitoring 1
Network 1
Other 2

Wikimedia Cloud Services is one of many providers of cloud computing resources. What made you decide to choose Wikimedia Cloud Services as opposed to other options?[edit]

2019 Cloud Services Survey - What made you decide to choose Wikimedia Cloud Services.svg
Answer Percentage
Access to Wikimedia-specific resources 81.2%
Cost 60.4%
Ease of collaborating with other Wikimedians 56.4%
Ease of use 36.6%
Philosophical or ideological reasons 32.7%
Privacy and security considerations 24.8%

Are you currently employed by the Wikimedia Foundation as an employee, contractor, vendor, or intern?[edit]

2019 Cloud Services Survey - Are you currently employed by the Wikimedia Foundation as an employee, contractor, vendor, or intern.svg
Answer Percentage
No 86.8%
Yes 13.2%

Qualitative questions[edit]

A series of qualitative questions were asked in order to gauge general satisfaction with Toolforge and Cloud VPS.

When I look for documentation on Wikitech about Toolforge or Cloud VPS I usually find what I was looking for.[edit]

2019-cloud-survey-11-documentation-disc.svg

The documentation I find on Wikitech about Toolforge or Cloud VPS usually answers my question.[edit]

2019-cloud-survey-12-documentation-comp.svg

The documentation I find on Wikitech about Toolforge or Cloud VPS is easy to read and understand.[edit]

2019-cloud-survey-13-documentation-clear.svg

If you would like to share more about your experience finding, reading, or maintaining technical documentation on Wikitech, please do so here.[edit]

This a curated summary of the different topics mentioned in the comments. Further commentary in the last section about Freeform comments.

2019 Cloud Services Survey - If you would like to share more about your experience finding, reading, or maintaining technical documentation on Wikitech, please do so here.svg
Topic Mentioned in comments
Out of date 6
Getting started 3
Hard to find 3
Praise 3
Lacking (DB Schemas) 2
Scattered 2
Support 2
Unclear organization 2

Services provided by Wikimedia Cloud Services, including Toolforge and Cloud VPS, have high uptime[edit]

2019-cloud-survey-07-high-reliability.svg

It is easy to have code run on Toolforge or Cloud VPS[edit]

2019-cloud-survey-08-easy-to-write-and-run.svg

I feel that I am supported by the Cloud Services team when I contact them via cloud@lists.wikimedia.org, the #wikimedia-cloud IRC channel, or Phabricator.[edit]

2019-cloud-survey-09-support.svg

I receive useful information via the “cloud-announce” and/or “cloud” mailing lists.[edit]

2019-cloud-survey-10-information.svg

Free form responses[edit]

The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form.

If we could improve one thing in Toolforge in the next year, what should that be?[edit]

Responses fell mainly under 5 main categories:

Documentation (14 comments)[edit]

The most mentioned topic was having more and better docs for getting started and introductory materials. Closely following and related was having more documentation about how to use a bigger variety of languages and libraries. And finally, some comments about having more video material (tutorials, talks and presentations).

Support & Community (12 comments)[edit]

These comments mentioned more support for beginners via documentation and tutorials, having places for discussion, resolving problems and knowledge sharing, better direct support from the team.

Platform (40 comments)[edit]

Within this theme, there is a wide variety in the comments.

Top mentions are better support for more languages, software and libraries, both having the latest versions and the ability to use other languages and libraries (and having more varied documentation with this regard). And also multiple mentions of improving performance of SSH sessions, file system and WikiReplicas.

Related but separately, there were multiple comments mentioning improving storage, both capacity and performance, of various DBs and the file systems (NFS).

There were also some comments related to workflows, that will be summarized in the next section, related to platform wanting to have other deployment mechanisms related to Docker images and Kubernetes.

Other comments not included in previous mentions talked about improving security, particularly around sharing subdomain.

The rest of the comments cannot be anonymously summarized because of their small numbers.

Workflow (13 comments)[edit]

In the workflow category, the most mentioned topic was deployment, having easier and more flexible options for deployment (git or docker based for example).

Following is observability, people mention both logs and monitoring, easier access and setup of logs and error reporting, and ways to monitor their projects.

Lastly it was mentioned wanting to have a way to develop software locally having access to Toolforge's data.

If we could improve one thing in Cloud VPS in the next year, what should that be?[edit]

We have less responses here so the summary will be a bit broad.

There is one clear ask mentioned in many comments, on the improving the configuration management side (better puppet support, more flexibility, more options).

Another topic mentioned multiple times as well is improving storage (faster, more space, easier maintenance when full).

The rest of the responses fell on a variety of topics: easier maintenance, quota related, deployment, documentation, monitoring, network, etc.

If you would like to share more about your experience finding, reading, or maintaining technical documentation on Wikitech, please do so here.[edit]

The main complaint was about documentation being out of date, followed by being hard to find, and related, having a hard time getting started.

There were also comments praising the quality of the documentation (even if old and hard to find), which are greatly appreciated. Thank you.

Other topics mentioned including having better documentation for the DB schemas, the docs being too scattered or having unclear organization.

If you have any other comments, please share them with us here.[edit]

Here there was a bunch of praise for the improvements and the services provided, which is greatly appreciated. We want to continue improving and doing a better job where we can. Thank you!

There were also some comments related to confusion with some of the questions, and some other comments that can't be shared anonymized.


Thanks to everyone that took the time and participated in the 2019 survey. See you soon in the 2020 one!