Research:Cloud Services Annual Survey/2016
The 2016 Tool Labs survey collects feedback from Tool Labs' developers and maintainers on how the services offered can be improved to help their development and maintenance needs.
Participants[edit]
The participants of the survey are the members of the Wikimedia Tool Labs project. 984 participants were contacted via email. The emails were harvested from Labs' LDAP system and the Wikitech database. The final list was created by finding the intersection of the members of the Tool Labs project gathered from LDAP and the Wikitech users with confirmed email addresses who had not opted-out of email contact by other users in their Wikitech preferences.
175 email recipients participated in the survey which represents 18% of those who were contacted.
Dates[edit]
The 2016 survey was active between 2016-10-20 and 2016-11-04.
Questions and answers[edit]
Basic demographics[edit]
How many years have you used Tool Labs?[edit]
Help: If you were a Toolserver user, please include the total number of years that you have used Toolserver and Tool Labs.
| Cohort | Percentage |
| 1 Year | 28.74% |
| 2-3 Years | 33.33% |
| 4+ Years | 37.93% |
How many tools have you developed in Tool Labs?[edit]
Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you have developed.
| Cohort | Percentage |
| 0 Tools | 12.64% |
| 1 Tool | 32.18% |
| 2-3 Tools | 27.01% |
| 4+ Tools | 28.16% |
How many tools do you actively maintain in Tool Labs?[edit]
Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you maintain.
| Cohort | Percentage |
| 0 Maintain | 21.97% |
| 1 Maintain | 34.10% |
| 2 Maintain | 18.50% |
| 3+ Maintain | 25.43% |
How many hours per week do you spend developing or maintaining tools on Tool Labs?[edit]
| Cohort | Percentage |
| 0 Hours | 28.83% |
| 1 Hour | 28.22% |
| 2-8 Hours | 33.74% |
| 9+ Hours | 9.20% |
Were you a developer and/or maintainer in the Toolserver as well?[edit]
| Cohort | Percentage |
| No Toolserver | 62.07% |
| Toolserver | 37.93% |
Additional demographics[edit]
What programming languages do you prefer to program in?[edit]
For this question, the participants had the following choices: Perl, PHP, Python 2, Python 3, Ruby, or they could enter the language of their choice. Note that participants could report more than one language.
| Language | % |
| Python3 | 48.00% |
| PHP | 46.86% |
| Python2 | 34.86% |
| JavaScript | 9.71% |
| Perl | 9.14% |
| Java | 8.57% |
| Other | 8.57% |
| Ruby | 7.43% |
| C# | 1.71% |
When you develop a tool, what percentage of your work is done locally (as opposed to remotely on Tool Labs)?[edit]
Help: For example, enter 30 if 30% of your development work is done locally.
Answers were rounded up to the nearest 5% for analysis.
| Answer | Percentage reported |
| 0 | 9.09% |
| 5 | 1.82% |
| 10 | 5.45% |
| 15 | 1.21% |
| 20 | 3.64% |
| 25 | 0.61% |
| 30 | 3.64% |
| 40 | 2.42% |
| 50 | 7.27% |
| 60 | 3.03% |
| 65 | 1.21% |
| 70 | 7.88% |
| 75 | 1.82% |
| 80 | 15.15% |
| 85 | 0.61% |
| 90 | 17.58% |
| 95 | 6.67% |
| 100 | 10.91% |
Do you prefer using source control to manage your tool's source code?[edit]
| Yes | No |
| 137 | 31 |
Qualitative questions[edit]
A series of qualitative questions were asked in order to gauge general satisfaction with Tool Labs.
Services have high reliability (up time)[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 87.57% | 91.67% | 90.91% | 81.82% | 95.65% | 86.79% | 88.89% | 83.33% | 89.47% | 85.96% | 86.67% | 88.64% | 83.64% | 91.11% | 88.89% | 86.67% | 89.42% | 84.62% |
| Disagree/Strongly Disagree | 10.06% | 0.00% | 9.09% | 18.18% | 0.00% | 7.55% | 11.11% | 16.67% | 5.26% | 10.53% | 13.33% | 11.36% | 9.09% | 8.89% | 11.11% | 13.33% | 7.69% | 13.85% |
It is easy to write code and have it running on Tool Labs[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 71.76% | 73.47% | 65.45% | 75.76% | 36.36% | 80.00% | 73.91% | 76.60% | 43.24% | 77.97% | 80.65% | 81.40% | 51.79% | 80.43% | 83.02% | 80.00% | 71.15% | 72.73% |
| Disagree/Strongly Disagree | 22.35% | 18.37% | 27.27% | 21.21% | 31.82% | 16.36% | 23.91% | 23.40% | 35.14% | 20.34% | 16.13% | 18.60% | 32.14% | 17.39% | 16.98% | 20.00% | 20.19% | 25.76% |
I feel I am supported by the Tool Labs team when I contact them via labs-l, wikimedia-labs IRC channel, or phabricator.[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 71.35% | 66.00% | 70.91% | 75.76% | 50.00% | 60.00% | 80.43% | 85.42% | 56.76% | 66.10% | 80.65% | 84.09% | 52.63% | 82.22% | 75.93% | 93.33% | 66.67% | 78.79% |
| Disagree/Strongly Disagree | 9.36% | 14.00% | 9.09% | 6.06% | 18.18% | 9.09% | 8.70% | 6.25% | 13.51% | 8.47% | 9.68% | 6.82% | 8.77% | 6.67% | 14.81% | 0.00% | 12.38% | 4.55% |
I receive useful information via labs-announce / labs-l mailing list.[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 46.71% | 43.75% | 38.89% | 55.38% | 34.78% | 41.51% | 48.84% | 56.25% | 27.78% | 42.11% | 46.67% | 68.18% | 29.63% | 60.00% | 50.00% | 57.14% | 40.38% | 57.14% |
| Disagree/Strongly Disagree | 17.37% | 14.58% | 18.52% | 18.46% | 8.70% | 9.43% | 25.58% | 22.92% | 13.89% | 15.79% | 26.67% | 15.91% | 16.67% | 11.11% | 22.22% | 21.43% | 15.38% | 20.63% |
Documentation is easy-to-find[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 46.43% | 48.94% | 45.45% | 45.45% | 45.00% | 48.15% | 50.00% | 41.67% | 38.89% | 47.37% | 54.84% | 45.45% | 50.00% | 51.11% | 42.59% | 33.33% | 43.14% | 51.52% |
| Disagree/Strongly Disagree | 49.40% | 42.55% | 50.91% | 53.03% | 35.00% | 46.30% | 50.00% | 58.33% | 47.22% | 49.12% | 45.16% | 54.55% | 37.04% | 48.89% | 57.41% | 66.67% | 50.98% | 46.97% |
How do you compare the support you received when using Toolserver with the support you receive when using Tool Labs?[edit]
Only the participants who had indicated previous Toolserver use were shown this question. Therefore, the responses are based on this subset of participants.
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | |
| Better | 47.69% | 50.00% | 40.00% | 48.98% | 100.00% | 27.27% | 52.63% | 48.48% | 62.50% | 28.57% | 64.71% | 42.31% | 36.84% | 60.00% | 42.11% | 57.14% |
| As good | 41.54% | 50.00% | 40.00% | 40.82% | 0.00% | 63.64% | 36.84% | 39.39% | 25.00% | 64.29% | 29.41% | 42.31% | 47.37% | 35.00% | 47.37% | 28.57% |
| Worse | 10.77% | 0.00% | 20.00% | 10.20% | 0.00% | 9.09% | 10.53% | 12.12% | 12.50% | 7.14% | 5.88% | 15.38% | 15.79% | 5.00% | 10.53% | 14.29% |
Tool Labs documentation is comprehensive.[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 50.29% | 55.10% | 50.00% | 46.97% | 45.45% | 54.55% | 52.17% | 45.83% | 48.65% | 52.54% | 51.61% | 47.73% | 48.21% | 50.00% | 50.00% | 60.00% | 50.48% | 50.00% |
| Disagree/Strongly Disagree | 44.44% | 36.73% | 42.86% | 51.52% | 27.27% | 40.00% | 47.83% | 54.17% | 35.14% | 42.37% | 48.39% | 52.27% | 37.50% | 47.83% | 50.00% | 40.00% | 41.90% | 48.48% |
Tool Labs documentation is clear.[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Agree/Strongly Agree | 47.95% | 53.06% | 42.86% | 48.48% | 40.91% | 49.09% | 56.52% | 41.67% | 37.84% | 47.46% | 61.29% | 47.73% | 39.29% | 56.52% | 50.00% | 46.67% | 43.81% | 54.55% |
| Disagree/Strongly Disagree | 46.20% | 36.73% | 50.00% | 50.00% | 31.82% | 43.64% | 43.48% | 58.33% | 45.95% | 45.76% | 38.71% | 52.27% | 46.43% | 39.13% | 50.00% | 53.33% | 47.62% | 43.94% |
Service usage[edit]
A section of questions on the frequency of use of various services offered by Tool Labs was included in the survey. The responses to these questions may be used to prioritize support and/or new service development by the Tool Labs administrators. These questions will not necessarily appear in future surveys.
How often have you used LabsDB access?[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Often | 38.89% | 20.45% | 28.85% | 59.09% | 4.55% | 18.00% | 52.38% | 64.58% | 18.92% | 18.87% | 62.07% | 65.12% | 13.21% | 55.81% | 47.06% | 53.33% | 28.28% | 55.56% |
| Seldom/Sometimes | 25.93% | 27.27% | 28.85% | 22.73% | 22.73% | 32.00% | 21.43% | 25.00% | 24.32% | 30.19% | 20.69% | 25.58% | 28.30% | 18.60% | 27.45% | 33.33% | 27.27% | 23.81% |
| Never | 27.78% | 40.91% | 36.54% | 12.12% | 45.45% | 46.00% | 16.67% | 10.42% | 40.54% | 45.28% | 10.34% | 6.98% | 43.40% | 20.93% | 21.57% | 13.33% | 33.33% | 19.05% |
How often have you used Webservices?[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Often | 49.08% | 30.43% | 55.77% | 56.92% | 9.09% | 44.23% | 54.76% | 68.09% | 16.22% | 48.15% | 51.72% | 76.74% | 21.82% | 64.29% | 58.82% | 73.33% | 42.57% | 59.68% |
| Seldom/Sometimes | 26.99% | 30.43% | 23.08% | 27.69% | 27.27% | 28.85% | 23.81% | 27.66% | 32.43% | 25.93% | 34.48% | 18.60% | 30.91% | 26.19% | 23.53% | 26.67% | 26.73% | 27.42% |
| Never | 17.18% | 30.43% | 15.38% | 9.23% | 40.91% | 23.08% | 14.29% | 2.13% | 37.84% | 18.52% | 10.34% | 2.33% | 34.55% | 4.76% | 13.73% | 0.00% | 21.78% | 9.68% |
How often have you used Cronjobs?[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Often | 42.07% | 20.00% | 39.62% | 59.09% | 0.00% | 35.29% | 51.16% | 60.42% | 16.67% | 35.71% | 68.97% | 53.49% | 30.91% | 48.84% | 50.00% | 35.71% | 32.35% | 58.06% |
| Seldom/Sometimes | 30.49% | 33.33% | 30.19% | 28.79% | 31.82% | 25.49% | 32.56% | 33.33% | 25.00% | 30.36% | 24.14% | 39.53% | 21.82% | 34.88% | 32.69% | 42.86% | 30.39% | 30.65% |
| Never | 22.56% | 40.00% | 26.42% | 7.58% | 45.45% | 37.25% | 11.63% | 6.25% | 44.44% | 30.36% | 6.90% | 4.65% | 36.36% | 13.95% | 15.38% | 21.43% | 29.41% | 11.29% |
How often have you used Redis?[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Often | 6.21% | 2.27% | 11.54% | 4.62% | 0.00% | 3.92% | 12.20% | 6.38% | 2.70% | 3.77% | 6.90% | 11.90% | 5.56% | 4.76% | 5.88% | 14.29% | 5.05% | 8.06% |
| Seldom/Sometimes | 18.01% | 9.09% | 19.23% | 23.08% | 4.55% | 13.73% | 17.07% | 29.79% | 2.70% | 15.09% | 24.14% | 30.95% | 3.70% | 23.81% | 21.57% | 42.86% | 16.16% | 20.97% |
| Never | 66.46% | 75.00% | 63.46% | 63.08% | 68.18% | 78.43% | 60.98% | 57.45% | 75.68% | 75.47% | 55.17% | 54.76% | 74.07% | 64.29% | 66.67% | 42.86% | 66.67% | 66.13% |
How often have you used GridEngine Continuous jobs?[edit]
| All | 1 Year | 2-3 Years | 4+ Years | 0 Tools | 1 Tool | 2-3 Tools | 4+ Tools | 0 Maintain | 1 Maintain | 2 Maintain | 3+ Maintain | 0 Hours | 1 Hour | 2-8 Hours | 9+ Hours | No toolserver | Toolserver | |
| Often | 24.39% | 20.00% | 20.75% | 30.30% | 0.00% | 19.23% | 28.57% | 37.50% | 8.11% | 25.45% | 31.03% | 32.56% | 12.73% | 20.93% | 36.54% | 35.71% | 20.79% | 30.16% |
| Seldom/Sometimes | 20.73% | 8.89% | 22.64% | 27.27% | 9.09% | 13.46% | 16.67% | 37.50% | 16.22% | 10.91% | 20.69% | 37.21% | 20.00% | 25.58% | 17.31% | 21.43% | 13.86% | 31.75% |
| Never | 48.17% | 64.44% | 49.06% | 36.36% | 63.64% | 63.46% | 47.62% | 25.00% | 59.46% | 56.36% | 48.28% | 27.91% | 54.55% | 46.51% | 44.23% | 42.86% | 54.46% | 38.10% |
Free form responses[edit]
The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form. The free form responses include comments broadly falling into these categories:
- Documentation (54 comments)
- Workflow (22 comments)
- Stability & performance (11 comments)
- Software versions (8 comments)
- Monitoring (8 comments)
- Kubernetes / PaaS (6 comments)
- Storage (4 comments)
- Support (4 comments)
- Discovery (3 comments)
- Other (15 comments)
Documentation[edit]
Over a third of the comments concerned the state of Tool Labs documentation. Given the survey scores on documentation related questions this is not surprising. Many suggestions for improvement focus on step by step guides, task based tutorials, and new user onboarding guides. Better coverage of database related issues also received multiple mentions.
Workflow[edit]
Workflow related comments included several calls for easier methods of uploading and downloading files in shared tool accounts. Others hoped for some form of automated deployment related to GitHub repositories and web-based editing capabilities. Several people commented on difficulties learning and using Linux commands with a Windows background.
Stability & performance[edit]
Three comments asked for overall stability to be a focus for the coming year's work. Increased performance of Tool Labs provided services such as database servers and the hosted tools themselves also received several mentions.
Software versions[edit]
Java 8, PHP 7, and general software and library upgrades were requested.
Monitoring[edit]
Services for monitoring and alerting on webservice uptime, grid jobs, and log files were included in the comments.
Kubernetes / PaaS[edit]
Comments asked for expanded Kubernetes services, Docker support, and Platform as a Service rollout.
Storage[edit]
Storage comments mentioned expanded capacity for database and large file storage. NFS speed improvements were also mentioned.
Support[edit]
Comments on the topic of support called for expanded staffing by the WMF and additional coverage in the #wikimedia-labs irc channel.
Discovery[edit]
Better systems to track and search existing tools are desired.