Research:Cloud Services Annual Survey/2021
The 2021 survey collected feedback from Toolforge project members and Cloud VPS project admins on how these services can be improved to meet their development and maintenance needs, as well as the user experience.
Participants[edit]
The participants of the survey are members of the Wikimedia Toolforge project and Cloud VPS project administrators. A total of 2388 contributors were reached via email for the survey, we selected those users who opted to receive surveys.
118 email recipients participated in the survey, this means we had a total response rate of 4.9%.
Methodology[edit]
During the first stage of the process, we prepared a questionnaire consisting of 36 questions, covering topics such as basic demographic information, use of the Toolforge and Cloud VPS platforms, and overall feedback around our services. Most of the questions had a multiple choice format, the other questions had a free-text response that aimed to give us insight about specific comments on the user’s side.
As for the implementation stage, the survey was active from December 21st, 2021 to February 5th, 2022.
For the analysis, we retrieved the responses from the forms and proceeded to process the data for each question through different cohorts. As for the free-text responses, we analyzed each one of them and categorized them according to the most mentioned topics.
Questions and answers[edit]
Basic demographic data[edit]
Do you use Toolforge or Cloud VPS to run or develop tools? (118 responses)
| Answer | % |
|---|---|
| Yes | 88.14 |
| No | 11.86 |
How many tools have you developed on Toolforge or Cloud VPS? (101 responses)
| Answer | % |
|---|---|
| 0-1 | 32.67 |
| 2-4 | 37.62 |
| 5-7 | 9.90 |
| 8+ | 19.80 |
How many tools do you actively maintain in Toolforge or Cloud VPS? (118 responses)
| Answer | % |
|---|---|
| 0-1 | 50 |
| 2-4 | 33.05 |
| 5-7 | 8.47 |
| 8+ | 8.47 |
How many hours per week do you spend developing or maintaining tools on Toolforge or Cloud VPS? (99 responses)
| Hours | % |
|---|---|
| 0-1 | 58.59 |
| 2-4 | 27.27 |
| 5-7 | 10.10 |
| 8+ | 4.04 |
Are you currently employed by the Wikimedia Foundation as an employee, contractor, vendor, or intern? (118 responses)
| Answer | % |
|---|---|
| Yes | 20 |
| No | 80 |
Additional demographics[edit]
What kinds of tools do you work on? (103 responses)
| Answers | Number of people
working on the tool |
|---|---|
| Bots | 69 |
| Web Apps | 74 |
| Microsites | 18 |
| Dashboard | 25 |
| APIs use | 33 |
| On-Wiki gadgets | 28 |
Which Wikimedia projects are your tools built to work with? (102 responses)
| Projects | Answer |
|---|---|
| Wikipedia | 74 |
| Wiki Commons | 34 |
| Wikidata | 32 |
| All Wiki Projects | 23 |
| Wikisource | 11 |
| Wiktionary | 9 |
| Wikiquote | 6 |
| Wikinews | 4 |
| Wikiversity | 4 |
| Wikibooks | 3 |
| Wikivoyage | 2 |
| Wikispecies | 2 |
| Others: Wikitech | 2 |
What storage/caching services do you use when running your tools? (84 responses)
| Answer | Number |
| MySQL/MariaDB (Not Including Toolsdb Or The Wiki Replicas) | 37 |
| Redis | 19 |
| SQLite | 11 |
| Other | 11 |
| Memcached | 4 |
| PostgreSQL | 1 |
| None | 1 |
Others included: hard drive, JSON files on NFS, ElasticSearch, OpenSearch, CSV.
Toolforge[edit]
Are you a member of the Toolforge project? (118 responses)
| Answer | % |
|---|---|
| Yes | 81 |
| No | 19 |
How many years have you used Toolforge? (118 responses)
| Answer | % |
|---|---|
| 0-1 | 37.3 |
| 2-4 | 23.7 |
| 5-7 | 15.3 |
| 8+ | 23.7 |
Which programming languages do you use on Toolforge? (96 responses)
| Programming language | Answers |
| Python 3 | 67 |
| PHP | 38 |
| NodeJS | 18 |
| Python 2 | 13 |
| Perl | 6 |
| Java | 5 |
| Other - Rust | 5 |
| Ruby | 4 |
| Other - Golang | 2 |
| Mono/.NET/C# | 1 |
| Other - Linux Shell | 1 |
| Other - Lua | 1 |
| Other - React | 1 |
When you develop a tool, how much of your work developing code to run on Toolforge is done locally on your machine (as opposed to remotely on Toolforge)? (95 responses)
| Answer | % |
| Almost all of the work | 51 |
| More than half of the work | 19 |
| About half of the work | 8 |
| Less than half of the work | 16 |
| I don’t know | 6 |
Which source control mechanism do you use to manage your tool's source code? (96 responses)
| Answer | % |
| Git | 89 |
| I Do Not Use Source Control | 10 |
| Mercurial | 1 |
Cloud VPS[edit]
Are you an administrator of one or more Cloud VPS projects? (118 responses)
| Answer | % |
|---|---|
| Yes | 32 |
| No | 68 |
How many years have you used Cloud VPS? (38 responses)
| Answer | % |
|---|---|
| 0-1 | 13 |
| 2-4 | 34 |
| 5-7 | 32 |
| 8+ | 21 |
What do you use Cloud VPS for? (19 responses)
| Answer | % |
| Hosting one or more tools or other public services such as a web app; bot; dashboard; API; etc. | 31 |
| Testing and experimenting with software | 30 |
| Running a backing service (database; cache; etc.) for a tool or other public service | 18 |
| Running one or more MediaWiki instances | 13 |
| Conducting data analysis or other large computational tasks | 8 |
Do you rely on NFS for accessing the same files across different servers? (43 responses)
| Answer | % |
| I do not know | 12 |
| No | 81 |
| Yes | 7 |
Services[edit]
For these qualitative sections, we looked back to the 2020 survey and found that the satisfaction levels are similar. However, it is important to look at the fact that people with less years, less tools developed, and less tools maintained, are the ones that present a lowest satisfaction level. This can be linked to the discomfort they presented in some of the freeform questions regarding the experience for new users. Users with more time using our Cloud Services are more familiar with the platforms and present higher levels of satisfaction.
However, when asked if it was easy to have code run on Toolforge or Cloud VPS and if they received useful information via the "cloud-announce" and/or "cloud" mailing lists, the satisfaction levels were lower this year than the ones in 2020.
Services provided by Wikimedia Cloud Services, including Toolforge and Cloud VPS, have high uptime. (115 responses)
| Answer | All | 0-1 Years | 2-4 Years | 5-7 Years | 8+ Years | 0-1 Tools | 2-4 Tools | 5-7 Tools | 8+ Tools | 0-1 Maintain | 2-4 Maintain | 5-7 Maintain | 8+ Maintain | 0-1 Hour | 2-4 Hours | 5-7 Hours | 8+ Hours | Toolforge | Cloud VPS | Toolforge+Cloud VPS | Tools | No tools |
| Agree/Strongly Agree % | 83 | 77 | 86 | 82 | 92 | 78 | 83 | 80 | 100 | 76 | 89 | 90 | 100 | 82 | 89 | 70 | 100 | 88 | 82 | 89 | 84 | 71 |
| Disagree/Strongly Disagree % | 6 | 7 | 11 | 0 | 4 | 6 | 11 | 0 | 0 | 9 | 5 | 0 | 0 | 7 | 4 | 10 | 0 | 3 | 10 | 4 | 6 | 7 |
It is easy to have code run on Toolforge or Cloud VPS. (115 responses)
| Answer | All | 0-1 Years | 2-4 Years | 5-7 Years | 8+ Years | 0-1 Tools | 2-4 Tools | 5-7 Tools | 8+ Tools | 0-1 Maintain | 2-4 Maintain | 5-7 Maintain | 8+ Maintain | 0-1 Hour | 2-4 Hours | 5-7 Hours | 8+ Hours | Toolforge | Cloud VPS | Toolforge+Cloud VPS | Tools | No tools |
| Agree/Strongly Agree % | 60 | 55 | 57 | 65 | 69 | 55 | 61 | 50 | 75 | 52 | 65 | 70 | 80 | 53 | 67 | 90 | 75 | 66 | 60 | 75 | 59 | 64 |
| Disagree/Strongly Disagree % | 13 | 14 | 18 | 6 | 12 | 18 | 11 | 10 | 5 | 17 | 8 | 10 | 10 | 16 | 11 | 0 | 0 | 5 | 24 | 4 | 11 | 29 |
I feel that I am supported by the Cloud Services team when I contact them via cloud@lists.wikimedia.org, the #wikimedia-cloud IRC channel, or Phabricator. (104 responses)
| Answer | All | 0-1 Years | 2-4 Years | 5-7 Years | 8+ Years | 0-1 Tools | 2-4 Tools | 5-7 Tools | 8+ Tools | 0-1 Maintain | 2-4 Maintain | 5-7 Maintain | 8+ Maintain | 0-1 Hour | 2-4 Hours | 5-7 Hours | 8+ Hours | Toolforge | Cloud VPS | Toolforge+Cloud VPS | Tools | No tools |
| Agree/Strongly Agree % | 66 | 54 | 68 | 75 | 77 | 52 | 68 | 63 | 95 | 57 | 71 | 78 | 90 | 65 | 62 | 78 | 100 | 70 | 79 | 89 | 66 | 71 |
| Disagree/Strongly Disagree % | 4 | 5 | 4 | 0 | 4 | 10 | 0 | 0 | 0 | 8 | 0 | 0 | 0 | 5 | 4 | 0 | 0 | 1 | 8 | 4 | 4 | 0 |
I receive useful information via the “cloud-announce” and/or “cloud” mailing lists. (106 responses)
| Answer | All | 0-1 Years | 2-4 Years | 5-7 Years | 8+ Years | 0-1 Tools | 2-4 Tools | 5-7 Tools | 8+ Tools | 0-1 Maintain | 2-4 Maintain | 5-7 Maintain | 8+ Maintain | 0-1 Hour | 2-4 Hours | 5-7 Hours | 8+ Hours | Toolforge | Cloud VPS | Toolforge+Cloud VPS | Tools | No tools |
| Agree/Strongly Agree % | 50 | 42 | 42 | 44 | 73 | 34 | 53 | 63 | 75 | 36 | 62 | 56 | 80 | 48 | 46 | 56 | 100 | 56 | 67 | 81 | 52 | 36 |
| Disagree/Strongly Disagree % | 8 | 5 | 12 | 13 | 4 | 9 | 9 | 0 | 5 | 8 | 9 | 11 | 0 | 7 | 4 | 22 | 0 | 2 | 13 | 7 | 8 | 7 |
Wikimedia Cloud Services is one of many providers of cloud computing resources. What made you decide to choose Wikimedia Cloud Services as opposed to other options? (114 responses)
| Reason | Number of selections |
| Access to Wikimedia-specific resources; including wiki replicas | 80 |
| Cost | 67 |
| Ease of collaborating with other Wikimedians | 58 |
| Philosophical or ideological reasons | 52 |
| Privacy and security considerations | 39 |
| Ease of use | 31 |
| Other | 1 |
For this specific question, we thought it would be good exercise to make an analysis between the years some of the users have used our services and the reasons why they choose our Cloud Services. In the graph below you can see some interesting facts. For example, it was surprising to find out that users that have been here for under 4 years picked "Philosophical or ideological reasons" more than those users that have been around for over 5 years.
Community and Support[edit]
Do you subscribe to the Cloud-Announce mailing list? (117 responses)
| Answer | Number of times chosen |
| No | 49 |
| Yes; but I usually don't read messages | 19 |
| Yes; and I usually read messages | 52 |
Do you subscribe to the Cloud mailing list? (117 responses)
| Answer | Number of times chosen |
| No | 61 |
| Yes; but I usually don't read messages | 20 |
| Yes; and I usually read messages | 36 |
When seeking help from WMCS staff, which support channels do you use? (105 responses)
| Answer | Number of times chosen |
| Phabricator | 75 |
| IRC | 68 |
| Mailing list/Other email | 27 |
| Telegram | 8 |
| Slack | 2 |
| Zulip | 0 |
When discussing WMCS issues with other volunteers, what channels do you use? (104 responses)
| Channel | Number of times mentioned |
| Phabricator | 60 |
| IRC | 59 |
| On-wiki talk pages | 45 |
| Mailing list/Other email | 34 |
| Discord | 18 |
| Other | 17 |
| Slack | 4 |
| 1 | |
| Zulip | 0 |
Other channels included Telegram, Element, GitHub, and Matrix.
Documentation[edit]
Documentation was the most mentioned topic on the freeform questions, so we thought it was a good idea to create one section and view all the results regarding this topic. The information from the freeform questions was summarized, there were some single comments that were not included because of anonymity and privacy.
Compared to the Documentation qualitative questions made in the 2020 survey, the satisfaction levels were similar. However, it's important to review documentation since the levels of satisfaction have a low percentage compared to other parts of our services.
When I look for documentation on Wikitech about Toolforge or Cloud VPS I usually find what I was looking for. (113 responses)
| Answer | All | 0-1 Years | 2-4 Years | 5-7 Years | 8+ Years | 0-1 Tools | 2-4 Tools | 5-7 Tools | 8+ Tools | 0-1 Maintain | 2-4 Maintain | 5-7 Maintain | 8+ Maintain | 0-1 Hour | 2-4 Hours | 5-7 Hours | 8+ Hours | Toolforge | Cloud VPS | Toolforge+Cloud VPS | Tools | No tools |
| Agree/Strongly Agree % | 59 | 52 | 57 | 56 | 74 | 48 | 68 | 56 | 74 | 54 | 61 | 63 | 80 | 58 | 65 | 50 | 75 | 68 | 50 | 67 | 61 | 50 |
| Disagree/Strongly Disagree % | 19 | 26 | 18 | 6 | 15 | 29 | 13 | 0 | 10 | 27 | 11 | 50 | 10 | 20 | 20 | 10 | 0 | 6 | 33 | 15 | 19 | 14 |
The documentation I find on Wikitech about Toolforge or Cloud VPS usually answers my question. (113 responses)
| Answer | All | 0-1 Years | 2-4 Years | 5-7 Years | 8+ Years | 0-1 Tools | 2-4 Tools | 5-7 Tools | 8+ Tools | 0-1 Maintain | 2-4 Maintain | 5-7 Maintain | 8+ Maintain | 0-1 Hour | 2-4 Hours | 5-7 Hours | 8+ Hours | Toolforge | Cloud VPS | Toolforge+Cloud VPS | Tools | No tools |
| Agree/Strongly Agree % | 57 | 52 | 50 | 69 | 63 | 48 | 57 | 67 | 74 | 53 | 58 | 50 | 80 | 53 | 65 | 40 | 100 | 63 | 53 | 67 | 57 | 57 |
| Disagree/Strongly Disagree % | 12 | 17 | 11 | 6 | 11 | 23 | 5 | 0 | 5 | 19 | 5 | 0 | 10 | 12 | 20 | 0 | 0 | 4 | 24 | 11 | 13 | 7 |
The documentation I find on Wikitech about Toolforge or Cloud VPS is easy to read and understand. (113 responses)
| Answer | All | 0-1 Years | 2-4 Years | 5-7 Years | 8+ Years | 0-1 Tools | 2-4 Tools | 5-7 Tools | 8+ Tools | 0-1 Maintain | 2-4 Maintain | 5-7 Maintain | 8+ Maintain | 0-1 Hour | 2-4 Hours | 5-7 Hours | 8+ Hours | Toolforge | Cloud VPS | Toolforge+Cloud VPS | Tools | No tools |
| Agree/Strongly Agree % | 59 | 48 | 64 | 69 | 67 | 42 | 76 | 44 | 79 | 53 | 63 | 50 | 90 | 56 | 58 | 70 | 100 | 66 | 49 | 59 | 61 | 50 |
| Disagree/Strongly Disagree % | 12 | 19 | 11 | 6 | 7 | 18 | 5 | 11 | 10 | 15 | 8 | 12 | 10 | 11 | 19 | 10 | 0 | 5 | 24 | 7 | 13 | 7 |
Freeform documentation questions[edit]
If you would like to share more about your experience finding, reading, or maintaining technical documentation on Wikitech, please do so here. (24 comments).
- Improve and update current documentation, and remove documentation that it’s not needed anymore. (5 comments)
- Documentation is not new user-friendly, it needs to have more guidelines or tutorials, and make the onboarding process easier for newcomers. (3 comments)
- Documentation is written for technical contributors rather than full-time developers. (2 comments)
- Most information it’s hard to find, it feels fragmentary. (6 comments)
- The layout isn’t intuitive and can be hard to navigate, an easy-search function is needed. (3 comments)
If you access or have accessed documentation outside of the Wikitech site, please tell us about it! (6 comments).
- Stack Overflow. (2 comments)
- Blogs or GoogleDocs. (2 comments)
- Phabricator and templates created by teams on Github. (2 comments)
Comments regarding documentation in other freeform questions[edit]
- Video Tutorials, guidelines and updated documentation for newcomers.
- Improve, update and remove documentation when needed.
- Clearer documentation on web service management.
- Documentation is perceived as being written for technical contributors rather than full time developers.
Other free form responses[edit]
The survey included several free form response sections. Survey participants were told that we would only share their responses in an anonymous form. In order to do this, we read each comment, analyzed them and created a summarized view of their insights. Responses regarding documentation can be found in the Documentation section of this report.
Among all of the responses that we received, there were single comments and, in interest of this anonymity, will not be detailed here.
If we could improve one thing in Toolforge in the next year, what should that be? (52 comments).
- Support:
- Improve help to beginners and have a friendlier onboarding process (2 comments).
- Platform:
- Update and support programming languages and its different versions(Python 3, node.js,). (6 comments).
- The login process can be considered slow and complicated (5 comments).
- Make it easier to use, the service is somewhat slow and not very intuitive (4 comments).
- More job management and monitoring functions, as well as job status updates. (4 comments).
- Improve storage capacity and overall IO performance (3 comments).
- Make cross database queries available (3 comments).
- Workflow:
- Easier migration of grid to k8s for scripts and bots (mostly CRONS) (4 comments).
- Improve automated deployments (3 comments).
- Deleting tools should be easier (2 comments).
- Grid Engine. (2 comments).
Is there anything you have tried but not been able to do in Toolforge? (22 comments)
- Continuous integration and deployment. (3 comments)
- Improve language support. (2 comments)
- Memory issues. (2 comments)
- Difficulties while navigating k8s. (2 comments)
- Celery worker queue. (2 comments)
If we could improve one thing in Cloud VPS in the next year, what should that be? (19 comments)
- There’s a need for an efficient backup service. (4 comments)
- Support for automating frameworks. (2 comments)
- Create tutorials and templates. (2 comments)
- API access to currently-Horizon-only VPS features and for provisioning, maintaining and destroying VMs. (2 comments)
If you participate in existing WMCS-related communities outside of email, IRC, and phabricator, please tell us about them! (7 comments).
- Telegram groups (3 comments).
- Discord: (3 comments).
- Discord channel on plwiki server.
- Discord server for specific communities.
If you have any other comments, please share them with us here. (21 comments)
A lot of users said thank you and gave a lot of praise to the Cloud Services, we appreciate your words and the time you put into answering our questions. Your thoughts and feedback are incredibly valuable!
Besides these comments, we had comments about language support, documentation, and some recommendations for the next survey, the most mentioned recommendations are listed below:
- Questionnaire improvement. (3 comments)
- Alternative platforms for the survey. (2 comments)
Thank you[edit]
We appreciate and value your feedback since it makes it possible for us to improve our services and get to know more about your needs. Thank you for participating in our 2021 survey! See you in 2022 :)