Community Insights/Community Insights 2020 Report/Collaboration, Diversity & Inclusion (2020)/Methodology

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search
2020 Collaboration, Diversity & Inclusion Supplement

Appendix: Methodology[edit]

To do develop measures to assess Collaborative Engagement, beginning in 2016 we consulted some of the engagement constructs assessed in our own Foundation staff assessment as well as WestEd’s California Schools (CalSCHLS) surveys School Climate and Gender & Sex-Based Harassment supplements and the Gay, Lesbian & Straight Education Network (GLSEN) US Schools Climate Survey.[1] [2] Items were developed after consultation of a few existing measurement frameworks and their validated questions available  for adapting to our ecosystem:

  • Radical Collaboration’s TLT Collaborative Skills Climate Survey framework specifically aiming to develop questions relevant to their concepts of Collaborative Intention, Awareness of Self and Others, and Problem Solving & Negotiation factors.[3]
  • Culture Amp’s engagement model specifically aiming to develop questions relevant to their concepts of Engagement, Belonging, Enablement, and Leadership factors.[4]
  • GLSEN’s School Climate Survey specifically aiming to develop questions relevant to the Wikimedia ecosystem develop questions relevant to their concepts of Harassment and Bullying, Safety, and Reporting Harassment and Assault.[3]

As with the collaborative engagement constructs we worked to develop for the first insights survey, we sought to enhance our assessment by doing similar for the assessment of key risk and protective factors for diversity and inclusion within the Wikimedia ecosystem. In 2017 we revisited our consultation of existing measures to include our own Foundation staff assessment’s Diversity & Inclusion Survey supplement, as well as the Society for Human Resource Management’s (SHRM) Diversity Survey and the Corporate Leadership Council’s Diversity Survey.[4][5][6] Items were developed after consultation with these existing frameworks to adapt similar constructs to our ecosystem from:

  • Culture Amp’s Diversity & Inclusion Survey framework drilling into diversity and inclusion aspects of Belonging.[2][4]
  • Corporate Leadership Council’s Diversity Survey specifically aiming to develop questions relevant to their concepts of Individual Commitment to Diversity.[5]
  • SHRM’s diversity model specifically aiming to develop questions relevant to their concepts of Attitudes Toward Diversity, Discrimination, Culture of Inclusion, Inclusive Interactions.[6]

After developing contextually relevant items to our movement ecosystem, we consulted internally to synthesize key concepts from across the consulted frameworks into proposed question sets to capture similar social climate constructs about Wikimedia environments.  Following response collection items were explored to confirm the reliability of the scale items as a construct and confirm the factor loading of our developed question sets.

These social climate constructs are intended to help gauge and track progress in reaching these shared goals. For this reason, it is not only important to understand the audience differences in the measured experiences, but also to understand the attitude target of survey participant’s ratings. For the most part, all audiences were considering online projects, primarily Wikipedia (64-68%), other Wikimedia projects (34-45%), or, for developers and affiliates, their own developer (23%) or affiliate group (35%) respectively. This was especially the case for questions focused on , Strategy Leadership, Engagement, Collaboration, Awareness of Self and Others, and Problem-Solving and Negotiation where 39-41% of developers and affiliates would report reflecting on those community spaces compared the normal 10% and 20% distribution for the affiliate, and developer community spaces, respectively.

Overall, Editors were significantly less likely to reflect on non-Wikipedia projects than either Developers or Organizers and developers were three times more likely than editors to reflect on an Affiliate group.

Figure. Wikipedia was the most cited community space for reflection when answering the questions about community climates. The second most reflected upon space for these constructs is other online project spaces for all audiences of editors.


Wikipedia was the most cited community space for reflection when answering the questions about community climates. The second most reflected upon space for these constructs is other online project spaces for all audiences of editors.  Wikipedia especially dominates thinking about Feelings of Belonging (75-78%), Community Attitude Toward Diversity (65-72%), & Non-Discrimination (63-70%) for all groups. For editors and movement organizers.  The questions about  of the movement, Engagement, and Recognition as well as for Collaboration, Awareness of Self & Others, and Problem-Solving and Negotiation led to similar levels of reflection about Wikipedia spaces for editors and movement organizers (i.e., 70-72% and 67-80%, respectively), but less so for Developers who cited Wikipedia significantly less in those areas in comparison (i.e., 8-10% and 13-27% less often, respectively). Developers and editors were more likely than movement organizers to reflect on Wikipedia when asked about   overall, Strategy Leadership, Engagement, and Individual Commitment to Diversity (68-73% compared to 26% for organizers)

Other Wikimedia online projects were second most noted and most often cited as the point of reflection when it came to questions about  overall, Strategy Leadership, Engagement, and Individual Commitment to Diversity, In fact, for movement organizers, in these reflection areas, other online Wikimedia projects replaced Wikipedia as most noted, with participants reporting reflection on those space more than twice as often as Wikipedia.

For the construct data as a whole, scales had good reliability for the most part, however, skewness and kurtosis were significantly non-normal with a regular tendency of moderate to strong negative skew and occasional strong positive kurtosis for certain constructs demonstrating that the distribution tails extend further than normal in comparison to the distribution mean.

Table. Normality checks[edit]

Kolmogorov-Smirnova Statistic df Sig. Skewness Kurtosis
Awareness of Others 0.142 1053 0.000 -0.442 0.028
Awareness of Self 0.153 1053 0.000 0.206 -0.068
Collaborative Intention 0.127 1239 0.000 -0.655 0.632
Engagement (Key to Inclusion) 0.142 2552 0.000 -1.149 2.035
Fairness 0.254 2305 0.000 -0.668 -0.403
Feelings of Belonging 0.123 2397 0.000 -0.669 0.562
Problem Solving & Negotiating 0.206 1224 0.000 -0.583 0.008
Movement Leadership 0.112 2401 0.000 -0.504 0.264
Movement Strategy 0.152 1230 0.000 0.750 -1.311
Non-Discrimination 0.237 2492 0.000 -0.457 -1.505
Inclusive Interactions 0.094 1178 0.000 -0.448 0.220
Inclusive Culture 0.164 2146 0.000 -0.463 0.396
Individual Commitment to Diversity 0.152 1230 0.000 -1.311 3.752
Wikimedia Leadership Commitment to Diversity 0.292 949 0.000 0.254 -1.535

Although results showed the data as non-normal and in violation of the assumptions of homogeneity of variance, the suggested statistical transformations (i.e., reflect and square root OR reflect and log transform), while successful at reducing the skewness for the most part and kurtosis to a great extent, none were successful at helping to normalize the distribution to any significant effect so the original data were maintained for analysis and nonparametric tests have been used in place of more powerful tests which rely on meeting the statistical assumptions of normal distributions. For this reason, both means and medians are reported in the footers of data tables as well as in more detail in the appendix (See Appendix: Methodological and statistical endnotes). When tested individually, not all groups demonstrated the same level of skew, sometimes individual group response distributions were more skewed for certain comparison groups and those differences are noted in endnote notations within the supplements exploring demographic differences.

Limitations and next steps[edit]

  • There were inconsistent meta-data for grouping and weighting for consistency in year over year comparisons as well as changes to the items and response scales which have made many comparisons unreliable this year. It is important to continue to stabilize the tool to bolster our power for detecting change.
  • There was also lack of sufficient statistical power inhibiting our ability to enter all predictive factors into a single model. It is important to continue to balance randomization of presented scales within the survey as well as continued efforts to increase the response rates as well to ensure adequate data capture for more elegant statistical modeling.
  • The Awareness of Self and Others construct needed to be split to separate out a factor of Awareness of Self separate from a factor of Awareness of Others. These items continue to have low reliability according to their Cronbach’s alpha and continued work to refine the indicators is recommended.
  • Similarly, the Inclusive Interactions construct failed to hang together well with a Cronbach’s alpha which is insufficient. It will be important to work to improve the proxy indicators within this construct as well.

Following careful assessment of the Wikimedia spaces which were being reflected on, the overwhelming tendency to reflect on Wikipedia firstly and other Wikimedia online projects secondarily, leads to the recommendation to reduce the frequency of checks on what community reflection space was considered.

Appendix. Methodological and statistical endnotes[edit]

All those who completed at least 50 percent of the Community Insights survey were weighted based on their monthly edit count used for sampling and compared across groups to understand differences. As many as 1517 and as few as 441 individuals responded to the various question sets to capture our social climate factors. Participant counts for the different question sets vary as some were randomized to shorten survey length while others were presented to every survey participant (See Appendix: 2019 Descriptive statistics).   All responses were collected on a 5-point likert scale of agreement with an option to respond “unsure.” All responses within each question set were scored such that a higher score is positive, some items were reverse scored where appropriate (as noted) and averaged to produce each factor score presented (See also Appendix: Methodological and Statistical Endnotes).

To examine the potential effects of the 50 percent or more completion requirement for inclusion in the  final analysis, due to the non-normal distribution of the data which could not be corrected via statistical transformation, a Mann-Whitney U-test was used to determine if there were differences in factor scores between those who did and did not complete at least half the survey.[7][8] Importantly, in two cases there was a detectable difference in factor scores between those who did and did not complete at least 50% of the survey. Those differences were explored to understand the nature of the differences although ultimately, those who did not complete at least half and thus, had completed no demographic information, were excluded from the final analysis. Factors which the analyzed sample reported significantly higher scores included: Engagement (n = 2572; mean rank of 1734.70 compared to 1494.12 for those who did not make it to the half-way point, n = 784; U = 1152778.0, p = .000), Non-Discrimination (n = 2468; mean rank of 1399.81 compared to 1312.15 for those who did not make it to the half-way point, n = 311; U = 407985.5, p = .048).

Along with the following statistical tests, means and medians are also reported in the data tables for clarity even though assumptions of normality and outliers are violated.[9] Sometimes t-tests data are also reported alongside nonparametric test results to ease interpretation.

  1. Due to the non-normal distribution of the data which could not be corrected via statistical transformation, a Kruskal-Wallis[10] test was conducted to determine if there were differences in Collaborative Engagement factor scores between contributor groups were different based on mean ranks[11]: Editors,  On-wiki Admins, Developers, Movement Organizers), and Movement Organizing Admin. (Note: n-value  varies by item see details in Appendix: 2019 Descriptive statistics for n-values, means, and medians)
    1. Distributions of Engagement scores were not similar for all groups, as assessed by visual inspection of a boxplot. Engagement scores were statistically significantly different between the different contributor types, χ2(4) = 79.153, p = .000, N = 1673. Subsequently, pairwise comparisons were performed using Dunn's (1964) procedure with a Bonferroni correction for multiple comparisons.[12] This post hoc analysis revealed statistically significantly lower Engagement scores among Developers (mean rank = 632.08) compared to Movement Organizers (mean rank = 974.04) (p = .000), and Movement organizing Admins (mean rank = 1125.66) (p = .000). Editors  (mean rank = 805.51) also tended to score higher than Developers  (p = .003) and lower than Movement organizing admins (p = .000). Among non-organizing on-wiki Admins (mean rank = 775.20). Engagement was also lower compared to Movement Organizing Admins  (p = .000), but not between Movement Organizers and Movement organizing Admins, Developers and non-organizing Admins, or Admins and Editors.
    2. Distributions of Feelings of Belonging scores were not similar for all groups, as assessed by visual inspection of a boxplot. Feelings of Belonging scores were statistically significantly different between the different contributor groups, χ2(4) = 25.823, p = .000, N = 1600. Subsequently, pairwise comparisons were performed using Dunn' procedure with a Bonferroni correction for multiple comparisons[12]. This post hoc analysis revealed statistically significant differences in Feelings of Belonging scores between Developers (mean rank = 714.34) and Movement Organizers (mean rank = 891.52)(p = .010), and Movement organizing Admins (mean rank = 943.66)(p = .005), as well as all between movement organizers and Editors (mean rank = 775.33) (p = .003), and Editors and Movement organizing Admins (p = .008), but not between the Developers and Editors, or non-organizing Admins (mean rank = 822.08) or any other group combination.
    3. Distributions of Fairness scores were not similar for all groups, as assessed by visual inspection of a boxplot. Fairness scores were statistically significantly different between the different contributor groups, χ2(4) =30.581, p = .000, N = 1531. Subsequently, pairwise comparisons were performed using Dunn's procedure with a Bonferroni correction for multiple comparisons.[12] This post hoc analysis revealed statistically significant differences in Fairness scores between Movement Organizers (mean rank = 695.65) and movement organizing Admins (mean rank = 971.04) (p = .000), as well as movement organizing Admins and  Editors (mean rank =767.42) (p = .000), and movement organizing Admins and Developers (mean rank = 703.73) (p = .000), but not between non-organizing Admins (mean rank = 825.27) and any other group combination.
    4. Distributions of Movement Leadership scores were not similar for all groups, as assessed by visual inspection of a boxplot. Movement Leadership scores were statistically significantly different between the different contributor groups, χ2(4) = 10.352, p = .035, N = 1630. Subsequently, pairwise comparisons were performed using Dunn's procedure with a Bonferroni correction for multiple comparisons.[12] This post hoc analysis revealed statistically significant differences in Movement Leadership scores between Developers (mean rank = 693.38) and Movement Organizers (mean rank = 864.95) (p = .013), and to a lesser extent, Developers and Editors (mean rank = 816.72) (p = .086), but not between the any other group combination.
    5. Distributions of Movement Strategy scores were not similar for all groups, as assessed by visual inspection of a boxplot. Movement Strategy scores were statistically significantly different between the different contributor groups, χ2(4) = 114.43, p = .000, N = 1337. Subsequently, pairwise comparisons were performed using Dunn's procedure with a Bonferroni correction for multiple comparisons.[12] This post hoc analysis revealed statistically significantly lower scores among Developers (mean rank = 632.31) compared to Movement Organizers (mean rank = 873.33) (p = .000), and Movement organizing Admins (mean rank = 843.28) (p = .000). Editors  (mean rank = 601.24) also tended to score lower than Movement Organizers (p = .000), and Movement organizing Admins (p = .002). Movement Strategy was also lower among non-organizing on-wiki Admins (mean rank = 650.49) compared to Movement Organizing Admins  (p = .011), but not between any other group pairs.
  2. Due to the non-normal distribution of the data which could not be corrected via statistical transformation, a Mann-Whitney U-test was used to determine if there were differences in factors scores between 2018  and 2019 data.[7] (Note: n-values vary by indicator and year and are specified in the parenthetical notes). Year over year analysis included only the editors group for which we are able to apply partial propensity score matching to weight for better representation based on the higher tendency for our more active editors to both start and complete the survey. When compared to 2018, an independent samples U-test found contributors less likely to report experiencing high levels of Fairness (mean = 3.60, N = 1672, t =-2.124, p = 0.034; U = 773692.5, p = .001), Feelings of Belonging (mean = 3.43, N =1735, t =-2.241, p = 0.025; U = 912252, p = .000), and Movement Leadership (mean = 3.34, N = 1766, t =-3.798, p = 0.000; U=632879, p = .000). While these may indicate a true difference, we recommend caution as we also have undergone changes to the sample metadata and, while we have worked to ensure alignment as best as possible, the threshold for reaching 50% or more completion in order to be included in the analysis was much more reliable with the 2019 data. Due to this difference, we may have been more, or less, conservative with retention for analysis than intended.  It is unknown what effect this may also have had on the comparison. (See also Appendix: Changes from 2018 to 2019)
  3. Due to the non-normal distribution of the data which could not be corrected via statistical transformation, again a Kruskal-Wallis test was conducted to determine if there were differences in Collaborative Engagement factor scores between contributor groups as follows:
    1. Distributions of Inclusive Culture scores were not similar for all groups, as assessed by visual inspection of a boxplot. Inclusive Culture scores were statistically significantly different between the different contributor groups, χ2(4) = 29.929, p = .000, N = 1439. Subsequently, pairwise comparisons were performed using Dunn's procedure with a Bonferroni correction for multiple comparisons.[12] This post hoc analysis revealed statistically significant differences in Inclusive Culture scores between Movement organizing Admins (mean rank = 902.12)  (p = .000) and Movement Organizers  (p = .014) compared to Editors (mean rank = 689.38) as well as between Movement Organizing Admins and non-organizing Admins (mean rank = 660.72) (p = .017), and between Movement organizing Admins and Developers (mean rank = 720.65) (p = .000), but not between any other group combination.
    2. Distributions of Non-Discrimination scores were not similar for all groups, as assessed by visual inspection of a boxplot. Non-Discrimination scores were statistically significantly different between the different contributor groups, χ2(4) = 45.532, p = .000, N = 1600. Subsequently, pairwise comparisons were performed using Dunn's procedure with a Bonferroni correction for multiple comparisons.[12] This post hoc analysis revealed statistically significant differences in Non-Discrimination scores between Editors (mean rank = 843.88) and Developers (mean rank = 690.40) (p = .003), as well as between Editors and Movement Organizers (mean rank = 671.88) (p = .000), but not between any other group combination.
    3. Distributions of Individual Commitment to Diversity scores were not similar for all groups, as assessed by visual inspection of a boxplot. Individual Commitment to Diversity scores were statistically significantly different between the different contributor groups, χ2(4) = 13.351, p = .01, N = 524.  Subsequently, pairwise comparisons were performed using Dunn's procedure with a Bonferroni correction for multiple comparisons.[12] This post hoc analysis revealed statistically significant differences in Individual Commitment to Diversity between Movement organizing Admins and Editors (mean rank = 248.30) (p = .036), but not between any other group combination.
  4. Due to the non-normal distribution of the data which could not be corrected via statistical transformation, a Mann-Whitney U-test was used to determine if there were differences in factors scores between 2018  and 2019 data.[7] (Note: n-values vary by indicator and year and are specified in the parenthetical notes). Once again, year over year analysis included only the editors group for which we are able to apply partial propensity score matching to weight for better representation based on the higher tendency for our more active editors to both start and complete the survey. When compared to 2018, an independent samples U-test found overall that contributors reported lower levels of Non-Discrimination  (mean = 4.27, n = 1732, t =-8.923, p = 0.000; mean rank of 1052.08 compared to 1161.62 in 2018,, U = 727937.5, p = .000) and Inclusive Culture  (mean = 3.43, n = 1544, t =-5.686, p = 0.419; mean rank of 869.61 compared to 1006.53 in 2018,, U = 727937.5, p = .000), and higher levels of Inclusive Interactions (mean = 3.67, n = 549, t =-0.809, p = 0.416;  mean rank of 736.48 compared to 517.31 in 2018, U = 154913.5, p = .000), compared to 2018. While this may indicate a true difference, we recommend caution as we also have undergone changes to the sample metadata and, while we have worked to ensure alignment as best as possible, the threshold for reaching 50% or more completion in order to be included in the analysis was much more reliable with the 2019 data. Due to this difference, we may have been more, or less, conservative with retention for analysis than intended.  It is unknown what effect this may also have had on the comparison. (See also Appendix: Changes from 2018 to 2019.

Appendix: Changes from 2018 to 2019[edit]

Collaboration & Engagment Factors
Construct Mann-Whitney U Change Notes
Mean 2019 Mean Rank 2018 Mean Rank 2019 Change

2018-2019

Sig.
Awareness of Others 3.29 570 610 0.048 All Items altered from "Most people" to "People"
Awareness of Self 2.70 569 500 0.000 All Items altered from "Most people" to "People"
Collaborative Intention 3.43 644 590 0.010 All Items altered from "Most people" to "People"
Engagement (Key to Inclusion) 4.04 1147 1095 0.061 Some items altered from "Wikimedia" to "Wikimedia projects"
Fairness 3.60 n/a n/a 🔻 0.000 No items were altered
Feelings of Belonging 3.43 1104 1003 🔻 0.005 No items were altered
Movement Leadership 3.34 1101 995 🔻 0.000 No items were altered
Movement Strategy 2.83 n/a n/a -- All new items
Problem Solving & Negotiating 3.25 n/a n/a 0.000 All Items altered from "Most people" to "People"
Diversity & Inclusion Factors
Construct Mann-Whitney U Change Notes
Mean 2019 Mean Rank 2018 Mean Rank 2019 Change

2018-2019

Sig. Mean 2018
Non-Discrimination 4.27 1162 1052 🔻 0.000 No items were altered 4.58
Inclusive Interactions 3.67 517 736 0.000 No items were altered 3.20
Inclusive Culture 3.43 1007 870 🔻 0.000 No items were altered 3.68
Individual Commitment to Diversity 4.01 637 577 0.005 Single item altered from "to take appropriate action to respond to incidents of harassment" to "to respond appropriately to incidents of harassment" 4.10
Wikimedia community attitudes toward value of diversity 3.50 269 712 0.000 Response scale changed from binary to likert to align with other indicators 1.00

Appendix: 2019 Descriptive statistics[edit]

Editors On-wiki Admins Developers Movement Organizers Movement Organizer Admins N Overall Mean Cronbach's Alpha
n Mean SE n Mean SE n Mean SE n Mean SE n Mean SE
Awareness of Others 470 3.3 0.04 45 3.2 0.11 27 3.1 0.18 106 3.4 0.09 37 3.4 0.14 685 3.3 0.67
Awareness of Self ' 441 2.7 0.04 43 2.9 0.11 24 2.4 0.19 97 2.6 0.09 36 2.6 0.11 641 3.45 0.60
Collaborative Intention 480 3.4 0.04 46 3.5 0.10 28 3.4 0.18 107 3.6 0.08 37 3.7 0.11 698 2.67 0.83
Problem Solving & Negotiating 457 3.2 0.04 46 3.3 0.12 28 3.2 0.20 101 3.4 0.10 37 3.4 0.14 669 3.26 0.80
Engagement *** 1517 4.0 0.02 131 4.1 0.06 108 3.8 0.08 294 4.2 0.04 121 4.4 0.06 2171 4.06 0.80
Fairness*** 1383 3.6 0.03 126 3.9 0.10 90 3.4 0.13 277 3.4 0.07 120 4.0 0.10 1996 3.59 --
Feelings of Belonging*** 1442 3.4 0.02 129 3.5 0.07 102 3.3 0.09 286 3.6 0.05 120 3.6 0.07 2079 3.45 0.73
Movement Leadership* 1468 3.3 0.02 130 3.2 0.08 109 3.1 0.10 293 3.4 0.05 121 3.3 0.08 2121 3.32 0.82
Movement Strategy*** 1152 2.8 0.03 103 2.7 0.10 90 2.8 0.12 266 3.5 0.06 116 3.4 0.08 1727 2.91 0.85
Leadership Commitment to Diversity*** 445 3.5 0.04 40 3.6 0.11 42 3.7 0.14 75 3.7 0.10 44 3.6 0.14 646 3.52 0.96
Non-Discrimination*** 1444 4.4 0.03 126 4.2 0.09 105 4.0 0.13 286 4.0 0.07 115 4.0 0.12 2076 4.27 0.89
Inclusive interactions* 463 3.7 0.04 41 3.7 0.11 44 3.6 0.13 77 3.8 0.09 45 3.8 0.14 670 4.03 0.48
Inclusive Culture*** 1260 3.4 0.02 122 3.3 0.08 97 3.5 0.10 284 3.6 0.05 120 3.8 0.08 1883 3.67 0.79
Individual Commitment to Diversity** 468 4.0 0.03 43 4.0 0.10 34 4.0 0.14 104 4.2 0.06 38 4.3 0.08 687 3.45 0.70
* Across all audiences and continents Engagement and Non-Discrimination were each negatively skewed to a significant degree and needed have a reflect and squareroot and reflect and logarithmic transformation, respectively, applied to convert the data distribution to meet assumptions of normality.

* Inclusive Culture was somewhat negatively skewed in all continents but especially so in Africa and South America.

* Leadership Commitment to Diversity was especially negatively skewed in Africa with positive kurtosis, meaning that those scores tended to group around a low median.

Demographics of the analyzed sample set[edit]

Frequency Proportion
Language Fluency Oth Lang Native 1419 82.1%
English Fluency 1294 74.8%
No English Fluency 435 25.2%
English Native 310 17.9%
Male - Non-Male Non-Male 308 17.9%
Non-Binary or Other Gender 120 7.0%
Male 1408 82.1%
Online Wikimedia Space Wikipedia 1224 66.5%
Commons 245 13.3%
Wikidata 59 3.2%
Other Wikimedia 312 17.0%
Continent coded Africa 60 1.7%
Asia & Pacific 321 9.0%
Europe 2523 70.4%
North America 416 11.6%
South America 263 7.3%

Appendix: Metrics to movement goals mapping[edit]

#1 Promote sustainability and resilience

Support dynamic, volunteer base; staff; and local groups in new acquiring funds and resources.

#6 Foster and develop distributed leadership

Train, support and retain socially- and technically-skilled individuals from different backgrounds that reflect the diversity of the global communities.

#2 Create cultural change for inclusive communities

Foster an inclusive, welcoming, safe, and collaborative environment for future growth.

#7 Invest in skills development

Foster technical and people-centered skills - e.g. communication, conflict resolution, intercultural dialogue.

#3 Improve user experience

Research, development and testing new features to enable contributors from all backgrounds to enjoy a fluid, effective, and positive experience.

#9 Coordinate across stakeholders

Cooperate and collaborate with different stakeholders to advance towards more equitable decision-making, with relevant training and enabling systems.

#4 Provide for safety and security

Ensure contributors have the proper conditions and resources enabling them to work without compromising their security.

#13 Plan infrastructure scalability

Create a fluid infrastructure to serve our needs as we grow; establish protocols, communication, and roles and responsibilities to invest sufficient resources for scalability

#5 Ensure equity in decision-making

Ensure a diversity and richness of perspectives, by focusing on the knowledge and communities that have been left out due to power and privilege.

Factor/Metric Reporting Favorable Conditions YoY Trend Strategy Recommendation
#1 #2 #3 #4 #5 #6 #7 #9 #13
Individual Commitment to Diversity 85% O O
Engagement 83% O O O O O O O O
Non-Discrimination*** 82% 🔻 O O O O
Fairness** 62% 🔻 O O O
Leadership Commitment to Diversity 61% O O O O O O O
Inclusive Interactions*** 59% O O O O O O O
Collaborative Intention 59% O O O O O
Feelings of Belonging*** 58% 🔻 O O O O O O
Awareness of Others 57% ^ O O O
Inclusive Culture*** 55% 🔻 O O O O O O O
Problem Solving & Negotiating 54% O O O O O O
Movement Leadership*** 51% 🔻 O O O O O O
Movement Strategy 35% O O O O O O
Self-Awareness 19% O O O

Appendix: 2019 Collaborative Engagement item averages by audience[edit]

Collaboration & Engagement[edit]

Scale/Item Editors Developers Admins Movement Organizers Movement Organizer Admins
Collaborative Intention Mean 3.4 3.4 3.5 3.6 3.7
Median 3.5 3.5 3.6 3.5 3.7
People are sincerely interested in supporting each other. Mean 3.4 3.4 3.5 3.6 3.8
Median 4.0 4.0 3.6 4.0 4.0
People are committed to building long-term cooperative relationships. Mean 3.3 3.3 3.2 3.6 3.5
Median 3.0 4.0 3.6 4.0 3.8
People with collaborative projects work together well. Mean 3.5 3.6 3.6 3.6 3.8
Median 4.0 4.0 3.6 4.0 4.0
People are good at reaching out when they need to consult with others. Mean 3.4 3.2 3.5 3.5 3.8
Median 4.0 4.0 3.5 4.0 4.0
Awareness of Others Mean 3.3 3.1 3.2 3.4 3.4
Median 3.5 3.5 3.4 3.5 3.5
People seek out and value feedback. Mean 3.5 3.5 3.5 3.6 3.7
Median 4.0 4.0 3.6 4.0 4.0
People know how to motivate others. Mean 3.1 2.7 2.9 3.2 3.1
Median 3.0 3.0 3.2 3.0 3.0
Awareness of Self Mean 2.7 2.4 2.9 2.6 2.6
Median 2.5 2.3 2.6 3.0 2.5
People lack self-awareness (reversed) Mean 2.7 2.4 3.1 2.7 2.8
Median 3.0 2.0 2.7 3.0 3.0
People are aware of their biases and patterns of behavior. Mean 2.9 2.7 2.9 3.0 2.7
Median 3.0 3.0 3.0 3.0 3.0
People are not aware of their own defensiveness to the situation at hand (reversed) Mean 2.6 2.4 2.7 2.6 2.5
Median 3.0 2.0 2.6 2.0 3.0
Problem Solving & Negotiating Mean 3.2 3.2 3.3 3.4 3.4
Median 3.5 3.3 3.4 3.5 3.5
People are committed to seeking fair solutions that are responsive to the interests of all parties. Mean 3.3 3.2 3.5 3.5 3.4
Median 3.0 3.0 3.5 4.0 3.0
People are sincerely willing to talk through competing personal interests to achieve fair solutions. Mean 3.2 3.2 3.1 3.4 3.4
Median 3.0 4.0 3.4 3.0 4.0

Collaboration & Engagement cont.[edit]

Scale/Item Editors Developers Admins Movement Organizers Movement Organizer Admins
Engagement (Key to Inclusion) Mean 4.0 3.8 4.1 4.2 4.4
Median 4.0 3.8 4.2 4.4 4.6
I would recommend Wikimedia projects as a great place to contribute. Mean 4.0 3.8 4.1 4.3 4.4
Median 4.0 4.0 4.3 5.0 5.0
The Wikimedia vision inspires me to contribute more to Wikimedia projects than I would somewhere else. Mean 4.0 3.7 4.1 4.3 4.3
Median 4.0 4.0 4.3 4.0 5.0
I take pride in contributing to the Wikimedia projects. Mean 4.1 3.8 4.3 4.4 4.6
Median 4.0 4.0 4.4 5.0 5.0
I often think about quitting as a contributor to Wikimedia projects (reversed) Mean 3.8 3.3 3.7 3.7 4.0
Median 4.0 4.0 3.7 4.0 4.0
I expect to be contributing to Wikimedia two years from now. Mean 4.2 4.2 4.4 4.5 4.6
Median 4.0 4.0 4.5 5.0 5.0
Fairness Mean 3.6 3.4 3.9 3.4 4.0
Median 4.0 3.0 3.4 4.0 4.0
My contributions to Wikimedia projects are often challenged or removed unfairly or inappropriately (reversed) Mean 3.6 3.4 3.9 3.4 4.0
Median 4.0 3.0 3.4 4.0 4.0
Feelings of Belonging Mean 3.4 3.3 3.5 3.6 3.6
Median 3.5 3.3 3.5 3.8 3.8
I feel like I belong in the Wikimedia movement. Mean 3.6 3.6 3.8 4.1 4.3
Median 4.0 4.0 4.0 4.0 4.0
I feel respected in the Wikimedia movement. Mean 3.5 3.5 3.7 3.8 4.0
Median 4.0 4.0 4.0 4.0 4.0

Collaborative Engagement (Movement)[edit]

Scale/Item Editors Developers Admins Movement Organizers Movement Organizer Admins
Movement Leadership Mean 3.3 3.1 3.2 3.4 3.2
Median 3.5 3.0 3.3 3.5 3.3
The Wikimedia Foundation communicates well about their projects and initiatives. Mean 3.1 2.9 3.0 3.3 3.2
Median 3.0 3.0 3.0 3.0 3.0
I am confident in the Wikimedia Foundation's ability to support the global movement. Mean 3.7 3.3 3.5 3.7 3.5
Median 4.0 4.0 4.0 4.0 4.0
There is open and honest communication between different organizations and groups in the Wikimedia movement. Mean 3.3 3.0 3.1 3.4 3.2
Median 3.0 3.0 3.0 4.0 3.5
I am satisfied with the Wikimedia movement's decision-making processes. Mean 3.2 3.0 3.1 3.2 3.0
Median 3.0 3.0 3.0 3.0 3.0
Movement Strategy Mean 2.8 2.8 2.7 3.5 3.4
Median 3.0 3.0 2.8 3.8 3.5
The 2030 movement strategy process is moving in the right direction. Mean 3.3 3.0 3.1 3.5 3.2
Median 3.0 3.0 3.0 4.0 3.0
I have faith in the leadership of the 2030 movement strategy process. Mean 3.3 2.9 3.2 3.4 3.2
Median 3.0 3.0 3.0 4.0 3.0
I understand the goals of the 2030 movement strategy process. Mean 2.6 2.7 2.8 3.5 3.6
Median 3.0 3.0 3.0 4.0 4.0
I know what the 2030 strategic direction is for. Mean 2.4 2.7 2.6 3.5 3.6
Median 2.0 3.0 2.0 4.0 4.0

Diversity & Inclusion[edit]

Scale/Item Editors Developers Admins Movement Organizers Movement Organizer Admins
Non-Discrimination Mean 4.4 4.0 4.2 4.0 4.0
Median 5.0 4.5 4.0 4.5 4.5
In the last 12 months, how often did you personally witness someone be treated unfairly because they are part of a specific cultural or social group (reversed) Mean 4.5 4.0 4.2 4.0 4.0
Median 5.0 5.0 4.0 5.0 5.0
In the last 12 months, how often have you felt unsafe or uncomfortable contributing to Wikimedia projects online? (reversed) Mean 4.3 4.0 4.2 4.0 4.0
Median 5.0 5.0 4.0 5.0 5.0
Inclusive interactions Mean 3.7 3.6 3.7 3.7 3.8
Median 3.8 3.8 3.7 3.8 4.0
Contributors of different backgrounds interact well in my Wikimedia project or community. Mean 3.5 3.5 3.6 3.7 3.7
Median 4.0 4.0 3.7 4.0 4.0
Contributors of different backgrounds are valued equally in my Wikimedia project or community. Mean 3.6 3.6 3.6 3.7 3.7
Median 4.0 4.0 3.7 4.0 4.0
Racial, ethnic, and gender-based jokes are tolerated in my Wikimedia project or community. (reversed) Mean 3.9 4.0 3.7 3.8 4.0
Median 4.0 4.0 4.0 4.0 4.0
My Wikimedia project or community provides an environment for the free and open expression of ideas. Mean 3.7 3.5 3.7 3.9 4.0
Median 4.0 4.0 3.9 4.0 4.0
Inclusive Culture Mean 3.4 3.5 3.3 3.6 3.8
Median 3.5 3.7 3.6 3.7 4.0
The Wikimedia movement is making progress with diversity initiatives (e.g., Wiki Women, LGBT, Gender Gap, indigenous languages, etc.). Mean 3.5 3.7 3.4 3.9 4.0
Median 4.0 4.0 3.9 4.0 4.0
Different project and contributor groups are treated fairly in the Wikimedia movement. Mean 3.3 3.2 3.2 3.3 3.4
Median 3.0 4.0 3.3 3.0 4.0
In the Wikimedia movement, people appreciate others whose context and cultural background is different from their own. Mean 3.4 3.5 3.3 3.6 3.8
Median 4.0 4.0 3.6 4.0 4.0

Diversity & Inclusion cont.[edit]

Scale/Item Editors Developers Admins Movement Organizers Movement Organizer Admins
Wikimedia Leadership Commitment to Diversity Mean 3.5 3.7 3.6 3.7 3.6
Median 3.5 4.0 3.7 3.8 4.0
In my Wikimedia project or community... - ...leadership encourages various cultural perspectives and contributors. Mean 3.4 3.6 3.6 3.7 3.6
Median 3.0 4.0 3.7 4.0 4.0
In my Wikimedia project or community... - ...there are policies and administrative actions that value content and contributors from different backgrounds. Mean 3.4 3.8 3.5 3.6 3.6
Median 4.0 4.0 3.6 4.0 4.0
In my Wikimedia project or community... - ...contributors are committed to including content from different cultures and perspectives. Mean 3.5 3.8 3.7 3.7 3.7
Median 4.0 4.0 3.7 4.0 4.0
In my Wikimedia project or community... - ...contributors respect individuals and value their differences. Mean 3.4 3.7 3.7 3.6 3.6
Median 4.0 4.0 3.6 4.0 4.0
Individual Commitment to Diversity Mean 4.0 4.0 4.0 4.2 4.3
Median 4.0 4.0 4.2 4.2 4.4
I am comfortable working with different kinds of people (e.g., ages, genders, ethnicities, religions, etc.) Mean 4.2 4.2 4.4 4.4 4.7
Median 4.0 4.0 4.4 5.0 5.0
I am confident when voicing my opinion, even when it’s different from those around me. Mean 3.9 3.8 3.9 4.1 4.0
Median 4.0 4.0 4.1 4.0 4.0
I believe that a diverse community positively impacts the quality of Wikimedia content. Mean 4.2 4.0 4.2 4.4 4.6
Median 4.0 4.0 4.4 5.0 5.0
I believe there are community policies and systems in place to respond appropriately to incidents of harassment. Mean 3.5 3.8 3.2 3.4 3.3
Median 4.0 4.0 3.4 4.0 4.0
I believe that multicultural collaboration produces positive outcomes. Mean 4.1 4.2 4.3 4.5 4.7
Median 4.0 4.0 4.5 5.0 5.0

References[edit]

  1. Kosciw, J. G.; Greytak, E. A.; Zongrone, A. D.,; Clark, C. M.; & Truong, N. L. (2018). "The 2017 National School Climate Survey". GLSEN. Retrieved May 2020. 
  2. a b WestEd (2016). "California School Modules". WestEd for the California Department of Education. Retrieved August 2016. 
  3. a b Tamm, J.; Luyet, & Thompson, R. & D. (2006). "Radical Collaboration: TLT Collaborative Skills Climate Survey™, Radical Collaboration. Retrieved online Aug 2016.". TLT Collaborative Skills Climate Survey™. Radical Collaboration. Retrieved Aug 2016. 
  4. a b c No author credited (2017). "6 Ways to Foster Belonging in the Workplace Taking Diversity & Inclusion to the Next Level" (PDF). Culture Amp. Retrieved October 2017. 
  5. a b Corporate Leadership Council (2017). "Diversity Survey" (PDF). www.diversityresources.stlrbc.org. Corporate Executive Board. Retrieved October 2017. 
  6. a b Society for Human Resource Management (2017). "Diversity Survey". www.shrm.org/. SHRM.org, California, USA. Retrieved Oct 2017. 
  7. a b c Mann, H.B.; Whitney, D. R. (1947). "On a test of whether one of two-random variables is stochastically larger than the other.". The Annals of Mathematical Statistics, 18(1), 50-60. 
  8. Sheskin, D. J. (2011). Handbook of parametric and nonparametric statistical procedures (5th ed.). Boca Raton, FL, USA.: Chapman & Hall/CRC Press. 
  9. Laerd Statistics (2015). "Statistical tutorials and software guides". Laerd Statistics. 
  10. Kruskal, W. H.; Wallis, W. A. (1964). "Use of ranks in one-criterion variance analysis". Journal of the American Statistical Association, 47(260), 583-621. 
  11. Lehmann, E. L. (2006). Nonparametrics: Statistical methods based on ranks. New York: Springer. 
  12. a b c d e f g h Dunn, O. J. (1964). "Multiple comparisons using rank sums". Technometrics, 6, 241-252.