Research:Cloud Services Annual Survey/2020

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search
Tracked in Phabricator:
task T270122

The 2020 survey collected feedback from Toolforge project members and Cloud VPS project administrators on how the services offered can be improved to help their development and maintenance needs.

Participants[edit]

The participants of the survey are the members of the Wikimedia Toolforge project and Cloud VPS project administrators. 1825 participants were contacted via email. The emails were harvested from the Cloud Services' LDAP system ("developer accounts") and the Wikitech database. The final list was created by finding the union of the members of the Toolforge project and all users with "administrator" rights on one or more Cloud VPS projects. Users who have opted-out of email contact by other users in their Wikitech preferences or the survey specific opt-out page were excluded. Specific opt-outs for the Annual Survey can be made by users at wikitech:Annual Toolforge Survey/Opt out.

137 email recipients participated in the survey which represents 7.5% of those who were contacted.

Dates[edit]

The 2020 survey was active between 2021-01-14 and 2021-02-05.

Questions and answers[edit]

Basic demographics[edit]

Do you use Toolforge or Cloud VPS to run or develop tools?[edit]

2020 Cloud Survey - Do you use Toolforge or Cloud VPS to run or develop tools.svg
Answer %
No 13%
Yes 87%

How many tools have you developed on Toolforge or Cloud VPS?[edit]

2020 Cloud Survey - How many tools have you developed on Toolforge or Cloud VPS.svg
Answer %
0-1 25%
2 23%
3-6 34%
7+ 17%

How many tools do you actively maintain in Toolforge or Cloud VPS?[edit]

2020 Cloud Survey - How many tools do you actively maintain in Toolforge or Cloud VPS.svg
Answer %
0-1 37%
2 25%
3-5 25%
6+ 13%

About how many hours per week do you spend developing or maintaining tools on Toolforge or Cloud VPS?[edit]

2020 Cloud Survey - About how many hours per week do you spend developing or maintaining tools on Toolforge or Cloud VPS.svg
Answer %
0 17%
1 36%
2-3 24%
4-8 13%
9+ 11%

Are you currently employed by the Wikimedia Foundation as an employee, contractor, vendor, or intern?[edit]

2020 Cloud Survey - Are you currently employed by the Wikimedia Foundation as an employee, contractor, vendor, or intern.svg
Answer %
No 84%
Yes 16%

Additional demographics[edit]

What kinds of tools do you work on?[edit]

2020 Cloud Survey - What kinds of tools do you work on.svg
Answer %
Web apps (tools that are accessed through a web browser) 77%
Bots (including pywikibot) 63%
APIs for use by other tools 32%
On-wiki gadgets 24%
Microsites (small static websites) 18%
Dashboards and other data visualizations 14%
Extensions 2%
Data analysis 1%
Other 2%

Which Wikimedia projects are your tools built to work with?[edit]

2020 Cloud Survey - Which Wikimedia projects are your tools built to work with.svg
Answer %
Wikipedia 64%
Wikidata 36%
Wikimedia Commons 29%
All Wikimedia projects 26%
Wikisource 9%
Wiktionary 7%
Wikinews 6%
Wikiquote 6%
Wikibooks 4%
Wikiversity 4%
Wikivoyage 3%
Phabricator 3%
MediaWiki 2%
Wikispecies 2%
Gerrit 1%
Logstash 1%
Meta 1%
Toolforge 1%
Wikitech 1%


Wikimedia Cloud Services is one of many providers of cloud computing resources. What made you decide to choose Wikimedia Cloud Services as opposed to other options?[edit]

2020 Cloud Survey - What made you decide to choose Wikimedia Cloud Services as opposed to other options.svg
Answer %
Access to Wikimedia-specific resources; including wiki replicas 74%
Cost 60%
Ease of collaborating with other Wikimedians 54%
Philosophical or ideological reasons 47%
Ease of use 34%
Privacy and security considerations 32%
Wikimedia staff 2%
Help with maintenance 1%
Reliability 1%


What storage/caching services do you use when running your tools?[edit]

2020 Cloud Survey - What storage caching services do you use when running your tools.svg
Answer %
Mysql/Mariadb (Not Including Toolsdb Or The Wiki Replicas) 63%
Redis 24%
Sqlite 18%
Filesystem 9%
Mongodb 6%
None 5%
Memcached 4%
Postgresql 2%
Toolsdb 2%
Elasticsearch 1%
Kafka 1%
Wiki Pages 1%

Toolforge[edit]

Are you a member of the Toolforge project?[edit]

2020 Cloud Survey - Are you a member of the Toolforge project.svg
Answer %
No 14%
Yes 86%

About how many years have you used Toolforge?[edit]

2020 Cloud Survey - About how many years have you used Toolforge.svg
Answer %
1 20%
2-3 23%
4-5 24%
6+ 33%


Which programming languages do you use on Toolforge?[edit]

2020 Cloud Survey - Which programming languages do you use on Toolforge.svg
Answer %
Python 3 67%
Php 44%
Nodejs 16%
Python 2 15%
Java 8%
Perl 6%
Bash 5%
Ruby 3%
Rust 3%
Javascript 3%
Other 8%

When you develop a tool, how much of your work developing code to run on Toolforge is done locally on your machine (as opposed to remotely on Toolforge)?[edit]

2020 Cloud Survey - When you develop a tool, how much of your work developing code to run on Toolforge is done locally on your machine (as opposed to remotely on Toolforge).svg
Answer %
Almost all of the work 60%
More than half of the work 14%
About half of the work 7%
Less than half of the work 15%
I don’t know 4%

Which source control mechanism do you use to manage your tool's source code?[edit]

2020 Cloud Survey - Which source control mechanism do you use to manage your tool's source code.svg
Answer %
Git 86%
I Do Not Use Source Control 13%
Mercurial 1%

Cloud VPS[edit]

Are you an administrator of one of more Cloud VPS projects?[edit]

2020 Cloud Survey - Are you an administrator of one of more Cloud VPS projects.svg
Answer %
No 68%
Yes 32%

About how many years have you used Cloud VPS?[edit]

2020 Cloud Survey - About how many years have you used Cloud VPS.svg
Answer %
1 22%
2-3 24%
4-5 29%
6+ 24%

What do you use Cloud VPS for?[edit]

2020 Cloud Survey - What do you use Cloud VPS for.svg
Answer %
Hosting one or more tools or other public services such as a web app; bot; dashboard; API; etc. 72%
Testing and experimenting with software 44%
Running a backing service (database; cache; etc.) for a tool or other public service 37%
Running one or more MediaWiki instances 35%
Conducting data analysis or other large computational tasks 9%

Do you rely on NFS for accessing the same files across different servers?[edit]

2020 Cloud Survey - Do you rely on NFS for accessing the same files across different servers.svg
Answer %
I do not know 14%
No 63%
Yes 23%

Community, support & communication[edit]

Do you subscribe to the Cloud-Announce mailing list?[edit]

2020 Cloud Survey - Do you subscribe to the Cloud-Announce mailing list (https lists.wikimedia.org mailman listinfo cloud-announce).svg
Answer %
No 39%
Yes; but I usually don't read messages 16%
Yes; and I usually read messages 45%

Do you subscribe to the Cloud mailing list?[edit]

2020 Cloud Survey - Do you subscribe to the Cloud mailing list (https lists.wikimedia.org mailman listinfo cloud).svg
Answer %
No 57%
Yes; but I usually don't read messages 13%
Yes; and I usually read messages 30%

When seeking help from WMCS staff, which support channels do you use?[edit]

2020 Cloud Survey - When seeking help from WMCS staff, which support channels do you use (Check all that apply).svg
Answer %
Phabricator 73%
IRC 66%
Mailing list/Other email 30%
Telegram 2%
Other 3%

When discussing WMCS issues with other volunteers, what channels do you use?[edit]

2020 Cloud Survey - When discussing WMCS issues with other volunteers, what channels do you use (Check all that apply).svg
Answer %
IRC 63%
Phabricator 59%
On-wiki talk pages 39%
Mailing list/Other email 34%
Facebook 7%
Telegram 7%
Discord 6%
Slack 6%
Matrix 2%
Real life meet-ups 2%
Zulip 2%
Other 3%

If you access or have accessed documentation outside of the Wikitech site, please tell us about it![edit]

Not a lot of people answered this question, so percentages are avoided here.

  • StackOverflow
  • In code repository
  • Debian wiki
  • Digital Ocean's tutorials
  • Kubernetes website
  • Meta wiki
  • Sourceforge

If you participate in existing WMCS-related communities outside of email, IRC, and phabricator, please tell us about them![edit]

Not a lot of people answered this question, so percentages are avoided here.

  • Local user groups and events
  • Telegram
  • Telegram #wmhack
  • Twitter
  • Wikimedia Community Discord

Qualitative questions[edit]

A series of qualitative questions were asked in order to gauge general satisfaction with Toolforge and Cloud VPS.

Services provided by Wikimedia Cloud Services, including Toolforge and Cloud VPS, have high uptime.[edit]

2020 Cloud Survey - Services provided by Wikimedia Cloud Services, including Toolforge and Cloud VPS, have high uptime.svg
Answer All 1 Year 2-3 Years 4-5 Years 6+ Years 0-1 Tools 2 Tool 3-6 Tools 7+ Tools 0-1 Maintain 2 Maintain 3-5 Maintain 6+ Maintain 0 Hours 1 Hour 2-3 Hours 4-8 Hours 10+ Hours Toolforge Cloud VPS Toolforge+Cloud VPS Tools No tools
Agree/Strongly Agree 82% 100% 83% 65% 92% 83% 92% 78% 95% 83% 89% 82% 93% 64% 90% 96% 80% 83% 80% 82% 94% 86% 59%
Disagree/Strongly Disagree 5% 0% 13% 8% 3% 4% 4% 5% 5% 6% 7% 0% 7% 14% 3% 4% 0% 8% 7% 0% 3% 5% 6%

It is easy to have code run on Toolforge or Cloud VPS.[edit]

2020 Cloud Survey - It is easy to have code run on Toolforge or Cloud VPS.svg
Answer All 1 Year 2-3 Years 4-5 Years 6+ Years 0-1 Tools 2 Tool 3-6 Tools 7+ Tools 0-1 Maintain 2 Maintain 3-5 Maintain 6+ Maintain 0 Hours 1 Hour 2-3 Hours 4-8 Hours 10+ Hours Toolforge Cloud VPS Toolforge+Cloud VPS Tools No tools
Agree/Strongly Agree 63% 76% 68% 50% 69% 52% 69% 65% 80% 49% 79% 71% 73% 47% 69% 77% 47% 75% 59% 55% 81% 66% 41%
Disagree/Strongly Disagree 10% 0% 20% 12% 8% 16% 12% 8% 0% 16% 7% 7% 0% 27% 5% 12% 0% 8% 9% 36% 0% 9% 18%

I feel I am supported by the Cloud Services team when I contact them via cloud@lists.wikimedia.org, the #wikimedia-cloud IRC channel, or Phabricator.[edit]

2020 Cloud Survey - I feel I am supported by the Cloud Services team when I contact them via cloud@lists.wikimedia.org, the -wikimedia-cloud IRC channel, or Phabricator.svg
Answer All 1 Year 2-3 Years 4-5 Years 6+ Years 0-1 Tools 2 Tool 3-6 Tools 7+ Tools 0-1 Maintain 2 Maintain 3-5 Maintain 6+ Maintain 0 Hours 1 Hour 2-3 Hours 4-8 Hours 10+ Hours Toolforge Cloud VPS Toolforge+Cloud VPS Tools No tools
Agree/Strongly Agree 64% 67% 61% 64% 76% 30% 60% 72% 100% 39% 73% 77% 93% 54% 57% 80% 73% 58% 60% 64% 84% 65% 59%
Disagree/Strongly Disagree 2% 5% 4% 0% 0% 9% 0% 0% 0% 6% 0% 0% 0% 8% 0% 0% 7% 0% 3% 0% 0% 2% 0%

I receive useful information via cloud-announce / cloud mailing list.[edit]

2020 Cloud Survey - I receive useful information via cloud-announce cloud mailing list.svg
Answer All 1 Year 2-3 Years 4-5 Years 6+ Years 0-1 Tools 2 Tool 3-6 Tools 7+ Tools 0-1 Maintain 2 Maintain 3-5 Maintain 6+ Maintain 0 Hours 1 Hour 2-3 Hours 4-8 Hours 10+ Hours Toolforge Cloud VPS Toolforge+Cloud VPS Tools No tools
Agree/Strongly Agree 53% 48% 57% 63% 57% 39% 68% 57% 72% 36% 81% 56% 79% 43% 47% 72% 67% 75% 49% 27% 77% 59% 18%
Disagree/Strongly Disagree 7% 10% 4% 8% 3% 13% 4% 3% 6% 11% 4% 4% 0% 14% 8% 4% 0% 0% 10% 0% 0% 6% 12%

Cloud Services documentation is easy-to-find.[edit]

2020 Cloud Survey - Cloud Services documentation is easy-to-find.svg
Answer All 1 Year 2-3 Years 4-5 Years 6+ Years 0-1 Tools 2 Tool 3-6 Tools 7+ Tools 0-1 Maintain 2 Maintain 3-5 Maintain 6+ Maintain 0 Hours 1 Hour 2-3 Hours 4-8 Hours 10+ Hours Toolforge Cloud VPS Toolforge+Cloud VPS Tools No tools
Agree/Strongly Agree 59% 48% 64% 65% 67% 48% 65% 62% 80% 49% 71% 68% 73% 50% 69% 65% 53% 67% 58% 50% 66% 63% 29%
Disagree/Strongly Disagree 16% 5% 28% 15% 8% 41% 12% 5% 5% 28% 14% 0% 13% 31% 13% 8% 13% 17% 19% 20% 3% 15% 18%

Cloud Services documentation is comprehensive.[edit]

2020 Cloud Survey - Cloud Services documentation is comprehensive.svg
Answer All 1 Year 2-3 Years 4-5 Years 6+ Years 0-1 Tools 2 Tool 3-6 Tools 7+ Tools 0-1 Maintain 2 Maintain 3-5 Maintain 6+ Maintain 0 Hours 1 Hour 2-3 Hours 4-8 Hours 10+ Hours Toolforge Cloud VPS Toolforge+Cloud VPS Tools No tools
Agree/Strongly Agree 58% 48% 56% 62% 66% 44% 60% 62% 80% 44% 67% 71% 73% 50% 66% 58% 47% 83% 55% 70% 63% 61% 38%
Disagree/Strongly Disagree 13% 10% 16% 15% 9% 26% 4% 14% 0% 21% 7% 7% 7% 25% 8% 12% 7% 8% 14% 10% 9% 12% 19%

Cloud Services documentation is clear.[edit]

2020 Cloud Survey - Cloud Services documentation is clear.svg
Answer All 1 Year 2-3 Years 4-5 Years 6+ Years 0-1 Tools 2 Tool 3-6 Tools 7+ Tools 0-1 Maintain 2 Maintain 3-5 Maintain 6+ Maintain 0 Hours 1 Hour 2-3 Hours 4-8 Hours 10+ Hours Toolforge Cloud VPS Toolforge+Cloud VPS Tools No tools
Agree/Strongly Agree 48% 38% 56% 40% 61% 23% 62% 54% 70% 28% 59% 68% 67% 44% 47% 69% 40% 58% 47% 50% 56% 52% 25%
Disagree/Strongly Disagree 17% 24% 12% 20% 11% 31% 12% 14% 5% 23% 11% 11% 13% 25% 13% 15% 13% 8% 20% 0% 13% 15% 31%

Free form responses[edit]

The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form.

If we could improve one thing in Toolforge in the next year, what should that be?[edit]

Responses to this question were varied, here is a summary of the comments clustered in different areas:

Documentation:

  • Improving documentation for beginners
  • Updating and removing outdated documentation
  • Better documentation for batch jobs / cron workflows

Support:

  • More responsive help desk
  • Help to beginners, people getting started
  • Centralized support site/forum

Workflow:

  • Deleting tools
  • Automated deployments
  • Development / Production environment parity for developing

Platform:

  • Replicas
    • Local access
    • Cross DB joins
    • Reliability
    • Stability (breaking changes)
    • Performance
  • Kubernetes
    • Usage for jobs is hard
    • Memory quotas
    • Tooling
  • NFS performance
  • Language / Version support
    • Node.js versions
    • Ruby on Rails support
    • Rust support
  • Docker support
  • Logging
    • Logs and emails from cron jobs
  • Monitoring
    • Health checks and alerts
    • Show general service availability

If we could improve one thing in Cloud VPS in the next year, what should that be?[edit]

Thank you for those who answered positive comments and praise, it is very much appreciated.

There are less responses in this question, so the summary will be less detailed.

  • Improve documentation
  • Better deployment workflows
  • Docker support
  • Managed Kubernetes clusters
  • Backups
  • Monitoring services

If you would like to share more about your experience finding, reading, or maintaining technical documentation on Wikitech, please do so here.[edit]

  • Hard to understand
    • Beginner documentation is very hard to understand
    • Needs more graphics
    • Cloud VPS docs are harder to understand than Toolforge's
  • Lacking
    • What should I use? What is supported?
    • Reference lists for features, instead of all tutorials
    • More documentation about effective workflows
  • Out of date
  • Structure
    • Self-study is very hard, complicated to navigate the docs (guided-study is not so bad)
  • i18n

If you have any other comments, please share them with us here.[edit]

There were many praise responses, thanks a lot, they are very much appreciated.

Besides those, there were some comments about language/framework support, using propietary software for this surveys instead of OSS, and a variety of comments about topics mentioned in the other freeform questions.


Is there anything you have tried but not been able to do in Toolforge?[edit]

The responses here were very varied, the ones mentioned several times were:

  • Improve language support and use newer versions of languages and applications
  • Memory intensive work on Toolforge. People end up moving to Cloud VPS for these
  • Remote access issues (SSH, SFTP, etc.)
  • Analysis work and very CPU/IO intensive programs
  • Problems with blocked IPs

The rest were single comments and in interest of anonymity won't be detailed here.


Thank you[edit]

Thanks to everyone that took the time and participated in the 2020 survey. Your input is extremely valuable and very much appreciated.

See you soon in the 2021 survey!