Research:Cloud Services Annual Survey/2015

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search

The 2015 Tool Labs survey collected feedback from Tool Labs' developers and maintainers on how the services offered can be improved to help their development and maintenance needs.

Participants[edit]

The participants of the survey were the members of the Wikimedia Tool Labs. 962 participants were contacted via email. The emails were harvested from Labs' LDAP system, which is available to everyone with a labs account. The list initially involved 977 participants. All those who had opted out of contacts via Wikitech were eventually excluded.

106 email recipients participated in the survey which represents 11% of those who are contacted.

Dates[edit]

The 2015 survey was active between 2015-09-25 and 2015-10-08.

Questions and answers[edit]

Basic demographics[edit]

Responses to the basic demographic questions were divided into roughly equal sized groups to provide "cohorts" of similar Tool Labs users. These groups were then plotted against each other in the qualitative responses to determine if there were significant differences in the opinions of the sub-groups.

How many years have you used Tool Labs?[edit]

Help: If you were a Toolserver user, please include the total number of years that you have used Toolserver and Tool Labs.

2015-tool-labs-survey-01-cohort-tool-labs-age.svg
Cohort Percentage
1 Year 29.81%
2-3 Years 36.54%
4+ Years 33.65%

Many of the respondents have been using Tool Labs for 2 or fewer years. In fact, nearly 30% of the respondents are using Tool Labs only for a year. This can be the sign of many young users or that young users are the ones that are more willing to participate in surveys. The distribution has a long tail indicating there are quite a few users who have been around for a long time.

How many tools have you developed in Tool Labs?[edit]

Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you have developed.

2015-tool-labs-survey-02-cohort-tool-developed.svg
Cohort Percentage
0 Tools 8.57%
1 Tool 31.43%
2-3 Tools 28.57%
4+ Tools 31.43%

40% of respondents have developed 0-1 tools. This may correlate with the amount of time that they have been active on Tool Labs. The distribution of the 4+ Tools cohort again has a long tail indicating that some tool developers are highly prolific.

How many tools do you actively maintain in Tool Labs?[edit]

Help: We do not have a canonical definition of "tools". We are trying to estimate how much burden is on you, so please use your best judgement in assessing how many tools you maintain.

2015-tool-labs-survey-03-cohort-tools-maintained.svg
Cohort Percentage
0 Maintain 16.19%
1 Maintain 27.62%
2 Maintain 26.67%
3+ Maintain 29.52%

The maintenance numbers roughly correlate with the number of Tools developed although the raw numbers are skewed downward slightly. This seems to suggest that very few respondents have become maintainers of tools that they didn't build themselves and also possibly indicates that some tools have been abandoned or replaced over time.

How many hours per week do you spend developing or maintaining tools on Tool Labs?[edit]

2015-tool-labs-survey-04-cohort-hours.svg
Cohort Percentage
0 Hours 24.51%
1 Hour 34.31%
2-8 Hours 29.41%
9+ Hours 11.76%

Over 50% of respondents spend an hour or less per week on average working on their tools.

Were you a developer and/or maintainer in the Toolserver as well?[edit]

2015-tool-labs-survey-05-cohort-toolserver.svg
Cohort Percentage
No Toolserver 59.43%
Toolserver 40.57%

The majority of survey responses were received from users with no history using Toolserver. It is currently unknown how this distribution correlates with the full number of active Tool Labs users.

Additional demographics[edit]

What programming languages do you prefer to program in?[edit]

For this question, the participants had the following choices: Perl, PHP, Python 2, Python 3, Ruby, or they could enter the language of their choice. The following table shows the ranked list of languages in terms of the percentage of time they were chosen as the language of choice. Note that participants could report more than one language.

Language %
PHP 58.49%
Python3 37.74%
Python2 36.79%
JavaScript 9.43%
Perl 8.49%
Ruby 7.55%
Other 7.55%
Java 6.60%
C# 4.72%

High usage of PHP and Python were not unexpected results. MediaWiki itself is written in PHP and Python is both widely used for data analysis tasks and well supported by the Pywikibot community. Freeform responses were normalized to make this analysis easier. The JavaScript numbers include responses of "JavaScript", "node", and "nodejs". The "Other" grouping includes several common scripting tools and a few very esoteric languages.

When you develop a tool, what percentage of your work is done locally (as opposed to remotely on Tool Labs)?[edit]

Help: For example, enter 30 if 30% of your development work is done locally.

2015-tool-labs-survey-06-local-dev-pcnt.svg

Answers were rounded up to the nearest 5% for analysis.

Answer Percentage reported
0 8.65%
5 3.85%
10 9.62%
20 2.88%
25 0.96%
30 3.85%
50 9.62%
60 3.85%
70 10.58%
75 2.88%
80 16.35%
85 1.92%
90 9.62%
95 5.77%
100 9.62%

Do you prefer using source control to manage your tool's source code?[edit]

2015-tool-labs-survey-07-source-cntrl.svg
Yes No
73% 26%

Qualitative questions[edit]

A series of qualitative questions were asked in order to gauge general satisfaction with Tool Labs. Since this was the first survey of this type given to the Tool Labs community these results will form a historical baseline to compare the results future surveys against.

Services have high reliability (up time)[edit]

2015-tool-labs-survey-09-high-reliability.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 63.81% 67.74% 54.05% 70.27% 70.00% 65.63% 76.67% 48.48% 66.67% 65.52% 70.37% 54.84% 68.97% 57.14% 63.33% 72.73% 68.25% 57.14%
Disagree/Strongly Disagree 34.29% 29.03% 43.24% 29.73% 20.00% 31.25% 23.33% 51.52% 22.22% 34.48% 29.63% 45.16% 24.14% 42.86% 36.67% 27.27% 28.57% 42.86%

It is easy to write code and have it running on Tool Labs[edit]

2015-tool-labs-survey-10-easy-to-write-and-run.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 68.57% 61.29% 67.57% 75.68% 50.00% 53.13% 73.33% 84.85% 55.56% 58.62% 74.07% 80.65% 51.72% 77.14% 66.67% 90.91% 66.67% 71.43%
Disagree/Strongly Disagree 24.76% 25.81% 27.03% 21.62% 0.00% 40.63% 26.67% 15.15% 16.67% 34.48% 25.93% 19.35% 31.03% 22.86% 30.00% 0.00% 23.81% 26.19%

I feel I am supported by the Tool Labs team when I contact them via labs-l, wikimedia-labs IRC channel, or phabricator.[edit]

2015-tool-labs-survey-11-support.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 66.67% 54.84% 72.97% 70.27% 60.00% 53.13% 73.33% 75.76% 50.00% 55.17% 81.48% 74.19% 55.17% 68.57% 70.00% 81.82% 65.08% 69.05%
Disagree/Strongly Disagree 10.48% 16.13% 5.41% 10.81% 0.00% 18.75% 13.33% 3.03% 11.11% 20.69% 3.70% 6.45% 17.24% 2.86% 16.67% 0.00% 7.94% 14.29%

I receive useful information via labs-announce / labs-l mailing list.[edit]

2015-tool-labs-survey-12-information.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 52.88% 40.00% 54.05% 62.16% 20.00% 45.16% 56.67% 66.67% 17.65% 44.83% 74.07% 61.29% 32.14% 48.57% 63.33% 90.91% 49.21% 58.54%
Disagree/Strongly Disagree 16.35% 6.67% 18.92% 21.62% 10.00% 9.68% 13.33% 27.27% 11.76% 13.79% 11.11% 25.81% 10.71% 17.14% 26.67% 0.00% 11.11% 24.39%

Documentation is easy-to-find[edit]

2015-tool-labs-survey-13-documenation-discovery.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 45.71% 58.06% 43.24% 37.84% 50.00% 43.75% 43.33% 48.48% 44.44% 51.72% 40.74% 45.16% 55.17% 45.71% 26.67% 72.73% 50.79% 38.10%
Disagree/Strongly Disagree 51.43% 35.48% 56.76% 59.46% 40.00% 53.13% 56.67% 48.48% 50.00% 44.83% 55.56% 54.84% 41.38% 54.29% 66.67% 27.27% 44.44% 61.90%

How do you compare the support you received when using Toolserver with the support you receive when using Tool Labs?[edit]

Only the participants who had indicated previous Toolserver use were shown this question. Therefore, the responses in the following table are based on this subset of participants. This was an optional question, and 4% of the participants chose not to respond.

2015-tool-labs-survey-14-toolserver-support.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Better 34.15% 33.33% 0.00% 41.94% 0.00% 12.50% 50.00% 31.58% 33.33% 28.57% 46.15% 27.78% 50.00% 31.25% 35.71% 0.00% 0.00% 34.15%
As good 36.59% 0.00% 42.86% 38.71% 0.00% 25.00% 35.71% 42.11% 33.33% 14.29% 23.08% 55.56% 12.50% 37.50% 50.00% 33.33% 0.00% 36.59%
Worse 29.27% 66.67% 57.14% 19.35% 0.00% 62.50% 14.29% 26.32% 33.33% 57.14% 30.77% 16.67% 37.50% 31.25% 14.29% 66.67% 0.00% 29.27%

Tool Labs documentation is comprehensive.[edit]

2015-tool-labs-survey-15-documentation-comprehensive.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 46.67% 54.84% 43.24% 43.24% 60.00% 40.63% 46.67% 48.48% 50.00% 44.83% 40.74% 51.61% 51.72% 51.43% 33.33% 54.55% 49.21% 42.86%
Disagree/Strongly Disagree 49.52% 41.94% 54.05% 51.35% 20.00% 56.25% 53.33% 48.48% 38.89% 51.72% 55.56% 48.39% 37.93% 48.57% 63.33% 45.45% 46.03% 54.76%

Tool Labs documentation is clear.[edit]

2015-tool-labs-survey-16-documentation-clear.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Agree/Strongly Agree 49.52% 58.06% 37.84% 54.05% 50.00% 46.88% 43.33% 57.58% 38.89% 48.28% 51.85% 54.84% 58.62% 45.71% 43.33% 54.55% 49.21% 50.00%
Disagree/Strongly Disagree 47.62% 38.71% 59.46% 43.24% 30.00% 53.13% 56.67% 39.39% 50.00% 51.72% 48.15% 41.94% 34.48% 51.43% 56.67% 45.45% 47.62% 47.62%

Service usage[edit]

A section of questions on the frequency of use of various services offered by Tool Labs was included in the survey. The responses to these questions may be used to prioritize support and/or new service development by the Tool Labs administrators. These questions will not necessarily appear in future surveys.

How often have you used LabsDB access?[edit]

2015-tool-labs-survey-17-usage-labsdb.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Often 44.90% 29.63% 47.06% 54.05% 11.11% 44.44% 43.33% 56.25% 12.50% 44.00% 51.85% 56.67% 28.00% 38.24% 67.86% 45.45% 39.66% 52.50%
Seldom/Sometimes 32.65% 25.93% 35.29% 35.14% 44.44% 18.52% 33.33% 40.63% 43.75% 32.00% 29.63% 30.00% 40.00% 35.29% 25.00% 27.27% 32.76% 32.50%
Never 18.37% 37.04% 11.76% 10.81% 22.22% 33.33% 20.00% 3.13% 25.00% 24.00% 18.52% 10.00% 20.00% 23.53% 7.14% 27.27% 24.14% 10.00%

How often have you used Webservices?[edit]

2015-tool-labs-survey-18-usage-webservices.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Often 59.60% 46.43% 67.65% 62.16% 11.11% 59.26% 53.33% 78.79% 35.29% 64.00% 57.69% 70.97% 52.00% 64.71% 62.07% 54.55% 55.17% 65.85%
Seldom/Sometimes 27.27% 32.14% 20.59% 29.73% 55.56% 18.52% 33.33% 21.21% 35.29% 12.00% 38.46% 25.81% 28.00% 26.47% 31.03% 18.18% 31.03% 21.95%
Never 11.11% 14.29% 11.76% 8.11% 11.11% 22.22% 13.33% 0.00% 17.65% 24.00% 3.85% 3.23% 12.00% 8.82% 6.90% 27.27% 10.34% 12.20%

How often have you used Cronjobs?[edit]

2015-tool-labs-survey-19-usage-cronjobs.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Often 53.61% 53.57% 54.55% 52.78% 11.11% 40.74% 73.33% 58.06% 11.76% 52.00% 73.08% 62.07% 20.83% 54.55% 72.41% 72.73% 56.14% 50.00%
Seldom/Sometimes 25.77% 25.00% 21.21% 30.56% 22.22% 25.93% 20.00% 32.26% 29.41% 20.00% 23.08% 31.03% 29.17% 30.30% 24.14% 9.09% 22.81% 30.00%
Never 18.56% 14.29% 24.24% 16.67% 44.44% 33.33% 6.67% 9.68% 47.06% 28.00% 3.85% 6.90% 41.67% 15.15% 3.45% 18.18% 17.54% 20.00%

How often have you used Redis?[edit]

2015-tool-labs-survey-20-usage-redis.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Often 8.42% 8.00% 12.12% 5.41% 0.00% 4.00% 10.00% 12.90% 0.00% 8.33% 7.69% 13.79% 0.00% 5.88% 14.29% 18.18% 10.91% 5.00%
Seldom/Sometimes 12.63% 8.00% 12.12% 16.22% 11.11% 4.00% 10.00% 22.58% 6.25% 4.17% 15.38% 20.69% 4.55% 17.65% 14.29% 9.09% 10.91% 15.00%
Never 74.74% 76.00% 75.76% 72.97% 66.67% 92.00% 80.00% 58.06% 75.00% 87.50% 76.92% 62.07% 81.82% 76.47% 67.86% 72.73% 74.55% 75.00%

How often have you used GridEngine Continuous jobs?[edit]

2015-tool-labs-survey-21-usage-gridengine.svg
All 1 Year 2-3 Years 4+ Years 0 Tools 1 Tool 2-3 Tools 4+ Tools 0 Maintain 1 Maintain 2 Maintain 3+ Maintain 0 Hours 1 Hour 2-8 Hours 9+ Hours No toolserver Toolserver
Often 35.05% 51.85% 27.27% 29.73% 11.11% 30.77% 43.33% 37.50% 12.50% 32.00% 50.00% 36.67% 16.67% 29.41% 46.43% 63.64% 36.84% 32.50%
Seldom/Sometimes 21.65% 18.52% 24.24% 21.62% 22.22% 11.54% 20.00% 31.25% 18.75% 4.00% 26.92% 33.33% 16.67% 23.53% 25.00% 18.18% 21.05% 22.50%
Never 41.24% 22.22% 48.48% 48.65% 44.44% 57.69% 36.67% 31.25% 56.25% 64.00% 23.08% 30.00% 58.33% 47.06% 28.57% 18.18% 38.60% 45.00%

Free form responses[edit]

The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form. The free form responses include comments broadly falling into these categories:

  • Documentation (33 comments)
  • Stability and performance (18 comments)
  • Version control and Deployment (14 comments)
  • Logs, Metrics, and Monitoring (12 comments)
  • Package management (10 comments)
  • SGE (8 comments)
  • Database (7 comments)
  • Account/Tool creation (5 comments)
  • SSH (5 comments)
  • Other (11 comments)

Documentation[edit]

In the responses mentioning documentation either directly or indirectly, generally the respondents were looking for more detailed how-to, getting started, and tutorial content. The need for more documentation for database contents was mentioned several times. Multiple respondents felt they did not know enough or have enough free time to contribute to the on-wiki documentation themselves.

Stability and performance[edit]

Stability and performance concerns often mentioned NFS. Some respondents are hoping for more proactive notification of planned and emergency downtime.

Version control and Deployment[edit]

Deployment comments mentioned the lack of a clear best practices and easy workflows for getting tools and their dependencies deployed to the servers. The version control comments were very similar in nature.

Logs, Metrics, and Monitoring[edit]

Monitoring for job failures and errors in user developed applications topped the comments related to logs, metrics and monitoring. Some facility for automated health checks and email alerts was mentioned several times.

Package Management[edit]

Package management comments either asked for specific system wide packages to be installed or lamented the difficulty in managing libraries needed for particular tools.

SGE[edit]

Sun Grid Engine comments generally mentioned the unexpected need to restart jobs and difficulty understanding how to pick the correct options for running jobs.

Database[edit]

Database comments ranged from complaints about using MySQL Workbench, to the naming of replica.my.cnf, to lack of EXPLAIN command support, and less open data than Toolserver provided.

Account/Tool creation[edit]

Account creation comments centered on the difficulty of registering for a Tool Labs account, largely mirroring the documentation concerns.

SSH[edit]

Comments on ssh mentioned the complexity of using ssh in general and frustration with bastions and tunneling.