Grants talk:IEG/The Missing Masses: Investigating the Absence of Women and Non-Cis People among Wikipedia Editors

April 12 Proposal Deadline: Reminder to change status to 'proposed'

@Chinmayisk and Lindsay Oliver: The deadline for Individual Engagement Grant (IEG) submissions this round is April 12th, 2016. To submit your proposal, you must (1) complete the proposal entirely, filling in all empty fields, and (2) change the status from "draft" to "proposed." As soon as you’re ready, you should begin to invite any communities affected by your project to provide feedback on your proposal talkpage. If you have any questions about finishing up or would like to brainstorm with us about your proposal, we're hosting an IEG proposal help session tomorrow on April 12th from 16:00 - 17:00 UTC on Google Hangouts.

I'm also happy to set up an individual session tomorrow if needed.

With thanks, I JethroBT (WMF) (talk) 22:38, 11 April 2016 (UTC)

Inactive editors

The proposers might want to include the rationale behind including "inactive editors", specifically those with less than five edits per month. -- Rohini (talk) 13:17, 12 April 2016 (UTC)

@Rohini: Thank you for your feedback have included that in our description --Chinmayisk (talk) 03:27, 14 April 2016 (UTC)

Suggestions and concerns

• The proposal needs to clarify the difference in methodology and outcomes compared to past studies of this nature.
• Interviewing 1000 (!) people is quite ambitious. The proposal can consider scaling down the scope, sample size or doing the project in phases. The budget shall also be reduced accordingly.
• Not sure if people who edit less than 5+ times a month (or people who haven't started writing) can give inputs related to the study. This set of contributors is also the hardest to reach out and get any survey done.
• If you see the Wikimedia Harassment Survey 2015, not many people are comfortable identifying their gender if they are non-cis. So, how do we reach 1000 people (with a good % of them under this demographic)? If you are going to talk to such people outside Wiki ecosystem, then their inputs may not reflect the real issues users who are inside the system face.
• Please take the help of Wikimedia Survey team who are experienced conducting research like this. --Ravi (talk) 11:45, 18 April 2016 (UTC)

@Ravidreams: Thanks for all the feedback - I've addressed several points below:

• Reaching and Interviewing People in the Targeted Demographics: We will be working with local organizations in the US and South/SouthEast Asia (including India, the Philippines, Indonesia, Pakistan, etc) to complete the survey, and are working with an expert in research, surveys, instrument construction, and data analysis to ensure we reach our goals. In addition, the 1000 person goal is representative not only of editors in the US, but as a sum of the demographics in each of the target geographic areas, which increases the pool we can draw from for potential survey takers.
• We are targeting non-editor survey takers specifically so we can compare their inputs to users inside the Wiki ecosystem and identify what factors are preventing women and non-cis editors from joining and contributing.
• We have reached out to the Wikimedia Survey team and are actively dialoguing with them through this process.

Thank you! --Womanontheinternet (talk) 11:19, 20 April 2016 (UTC)

@Ravidreams: Some additional comments:

• Regarding the interviews, we don't intend to do in-person interviews with ~1,000 people. That number is the target for the questionnaire instrument. For in-person interviews, the number is an order of magnitude lower, at around a 100 people. That is a much more achievable size with a small team of interviewers, and also a far easier number of people to reach. It will of course be necessary to reach out to a larger group of people in order to reach the desired response rate, but for the in-person interviews it's possible to use institutional connections and snowball sampling methods to find additional interviewees. I hope that helps! -- (MS Patterson) Archimedes52 (talk) 00:40, 21 April 2016 (UTC)

Concerns re: scope and interviews

First, I love this project!

I too am a bit concerned about the scope. Interviewing 150 people, especially if you plan to transcribe all of the interviews and then do iterative, qualitative doing, is a HUGE task. Generally researchers interview until they've reached saturation (i.e., they are no longer hearing about new phenomena and/or themes). I'd be happy to talk more about this if it helps. I'd highly recommend revisiting your scope so that you're less frustration and more opportunities to succeed!

--Mssemantics (talk) 18:01, 23 April 2016 (UTC)

@Mssemantics:

Thanks for the feedback! The actual structure and length of the interviews remains to be determined; they will likely be at least semi-structured, and of a manageable length. Transcription and analysis at that scale DO of course still present real challenges. We'll be investigating the possibilities of transcription software, and qualitative analytical suites like ATLAS.ti to make the process more efficient and less painful.

Re: the saturation approach, I'm not sure we'd be entirely comfortable with that approach, given the relatively unexplored nature of what we are proposing to investigate, the differing cultural contexts we'd be working in, and the diversity of the groups in question. When looked at that way, the 100-150 subjects potentially disaggregates into fairly small clusters of ~10-20 individuals of a particular "type". It's also likely that we will need to rely on snowball methods to find our interview subjects, and using saturation as a cut-off might introduce problems as it may take several steps of snowballing to reach subjects with novel data. None of this is to say that you're incorrect, simply that the finer details of the methods remain in flux. If you've any additional thoughts on the matter in light of my response, please do share them!

--MS Patterson Archimedes52 (talk) 00:45, 24 April 2016 (UTC)

• Thanks Archimedes52. Why not narrow your focus? See comments below to this point too. :) --Mssemantics (talk) 19:38, 25 April 2016 (UTC)

Proposal scope inquiry

What is the population that you intend to sample for this proposal? It isn't clear to me, and needs to be defined more clearly. It is stated as "women and non-cis people". I understand who women are. I don't know what is meant by non-cis people. I checked the Wikipedia English article for Cis, which leads me to a disambiguation page and from there to Cisgender. That page defines transgender as the complement of cisgender, i.e. the entire population is the sum of transgender plus cisgender people. Why not say transgender for the survey then?

Do you want to include women too, or ONLY include non-cisgender women? The latter may be your intent, which is consistent with how you named your study, for someone who chooses to parse the language carefully and with additional context, i.e. those who are women AND transgender, ONLY. If that is the case, then I don't know why you use the term, "the missing masses". The absence of cisgender women (I assume that includes lesbians who are biological women) is accurately described as a "mass" of the south/south-east Asian population, being slightly more than 50% of the total. However, those who identify as women but are not chromosomally XX, i.e. non-cisgender, are certainly not masses of the population.

Perhaps I have misunderstood. Please clarify by answering the following:

1. Are you excluding heterosexual XX females and lesbian XX females from your study?
2. Are you excluding heterosexual and homosexual XY males from your study?
3. Are you excluding non-cisgender males from your study?
4. Are you including non-cisgender females in your study?

Regardless of my confusion about the intended sample, I would recommend that you rename the study to "The Absence of Women and Non-Cisgender People among Wikipedia Editors" rather than non-cis.

I am a statistician by trade, and have done survey design and analysis for my work (although my work was for childhood non-infectious disease epidemiology, not gender-related) so I know how important the criteria for population definition is, in gathering a sample. Thank you.--FeralOink (talk) 18:57, 23 April 2016 (UTC)

@FeralOink: It's a delight to hear from a working statistician and researcher!

The question of sample population is a good one. When it comes to using "non-cis" instead of simply "trans" we do so because of the more inclusive framing. Not all non-cis people identify as trans; some may be genderqueer, or gender-fluid/flexible, or identify as ungendered, etc. To state that we were seeking out trans individuals would mean we were only sampling a particular slice of the non-cis population, when we in fact want to reach as many parts of that population as possible.

When it comes to women, yes, we are including all women, as they are drastically under-represented. Women + non-cis people adds up to a large chunk of the population (if not an outright majority) in most places.

As for your questions, it's a little bit difficult to respond in that framework, I'm afraid. A "heterosexual XY male" might still be genderqueer, for instance (I know several), and would thus be included, while one who did not identify as such would not. Perhaps my explanation of what we mean by non-cis helped sufficiently to make clear our who is and is not in our population? If not, please let us know. --MS Patterson Archimedes52 (talk) 00:29, 26 April 2016 (UTC)

Eligibility confirmed

This Individual Engagement Grant proposal is under review!

We've confirmed your proposal is eligible for review and scoring. Please feel free to ask questions and make changes to this proposal as discussions continue during this community comments period (through 2 May 2016).

The committee's formal review begins on 3 May 2016, and grants will be announced 17 June 2016. See the round 1 2016 schedule for more details.

Questions? Contact us at iegrantswikimedia · org .

--Marti (WMF) (talk) 04:26, 28 April 2016 (UTC)

@Mjohnson (WMF): Thank you and the team at IEG for this notification we look forward to your review and comments. Please feel free to reach out to us if you need any additonal information to make this possible. Offcourse we will continue to interact with the community about this.

Aggregated feedback from the committee for The Missing Masses: Investigating the Absence of Women and Non-Cis People among Wikipedia Editors

Scoring rubric	Score
(A) Impact potential Does it have the potential to increase gender diversity in Wikimedia projects, either in terms of content, contributors, or both? Does it have the potential for online impact? Can it be sustained, scaled, or adapted elsewhere after the grant ends?	7.6
(B) Community engagement Does it have a specific target community and plan to engage it often? Does it have community support?	6.0
(C) Ability to execute Can the scope be accomplished in the proposed timeframe? Is the budget realistic/efficient ? Do the participants have the necessary skills/experience?	5.0
(D) Measures of success Are there both quantitative and qualitative measures of success? Are they realistic? Can they be measured?	6.4
Additional comments from the Committee: All gender gap work is welcome. I can see a gender equity barrier among many online communities. More engagement is important for the Wikimedia movement. This project could be incredibly impactful. More quantitative and qualitative analysis of the gender gap is necessary. However, I am concerned about the scope, as this project is VERY ambitious for a 6 month project. This is an innovative proposal as it also takes into account non-binary genders, with a focus on Southeast Asia, which shows a higher percentage of female biographies in general than in Western languages. So knowing more may help us close the gap in Western Europe and in the US. I think this project definitely takes an innovative approach, in that this kind of large-scale research on the gender gap hasn't really been done since 2011. It's highly necessary. Including non-cis community members is a step in the right direction. Looks doable, and has support. There are very significant expenses for the project. I think this is a really important project. But, like many of the commenters have noted on the talk page, I'm concerned about the scope. I do think this NEEDS to be a large-scale project but whether or not that is implementable in 6 months is debatable. The participants have the necessary skills to implement and work with other professionals to implement the project. As I mentioned, my main concern is timing. This proposal does not have a target Wikipedia community. How would it ensure the diversity among the survey? It isn't very clear how community engagement would actually happen. For a proposal that would heavily involve community engagement, this is a critical issue that needs to be clear and well planned. I think this project supports diversity and will have a lot of community support. I have some uncertainties with the Gender gaps studies, because they give us new metrics or targets to be used in all projects, but I have not seen gender gap-related projects using these studies. I will stay neutral. I don't have an opinion about this project. It's good but I have concerns about the high cost. I am planning a similar project for my linguistic area, but it has no projected expenses. I do not understand why the project requires $30,000 when there could be other cost-effective methods. I would support it with a different budget. I will be honest, I was not sure what "white cis men" was supposed to refer to. The proposal should identify such abbreviations/terms before widespread usage in rest of the proposal. I would be more than willing to support this study if it were structured more like the proposal "Beyond the Gender Gap: Understanding Women's Participation in Wikipedia," which similarly focuses on non-cis users as well as other gender and sexual identities with a gender gap. This proposal is also too ambitious at this stage with a budget of $30,000. It would be best if we had multiple smaller scale academic studies better identifying problems before we increase the scale to this magnitude. The proposed budget should also have technical tool development as one of its outcomes.

-- MJue (WMF) (talk) 00:58, 3 June 2016 (UTC) on behalf of the IEG Committee

Major Proposal Revisions

After reviewing and thinking about the various comments and feedback we have received (for which we are very grateful) we have conducted considerable revision to the proposal, summarized in brief below.

• The language describing purpose of the grant has been clarified.
• We have provided an explanation of why we are interested in studying the people and locations we are proposing to study.
• We have added more detailed explanations of our methods and outputs.
• We have emphasized the process of building the support structures needed to do the work.
• Begun working on a subpage detailing the relationship of this proposal to previous work.
• Added information about where we would like this work to lead.
• Added a list of supporting organizations, associated volunteers, and possible sources of external funding we intend to pursue.

@Rohini, Ravidreams, Mssemantics, and FeralOink: We would love to hear from those who have commented previously, and if our revisions have addressed or clarified any concerns you may have had or might still have about the work.

@Mjohnson (WMF) and MJue (WMF): Some additional material about community engagement/outreach may be added later today, in response to the committee comments that have recently been posted. --Archimedes52 (talk) 17:53, 3 June 2016 (UTC)

Apologies for being late in my response, but you're funded, so: Yay! And I think you've done a great job of narrowing the scope. If you'd like to chat about the project more, let me know. --Mssemantics (talk) 20:26, 21 June 2016 (UTC)

Non-cis people?

Is there any evidence that these people are not participating on the Wikimedia sites? 0.3% of Americans identify as trans-gendered, and this number gets even smaller when you leave the developed world. I think people might be confused here; I see no evidence that trans people contribute less, but there are just much fewer of them to begin with. Obviously this doesn't apply to women, but that's a separate issue. Ajraddatz (talk) 18:05, 3 June 2016 (UTC)

@Ajraddatz: Thanks for the question!

First, a clarification: We are not attempting to study only trans people, but all people who identify as non-cisgender, which is a much more inclusive umbrella, and includes non-binary and genderqueer folks that identify a lot of different ways.

Re: the size of this population (that it is very small); yes, it is small, in one sense! In another, it is quite large: using the 0.3% number, we may have something like 700,000 transpeople in the United States. However, estimates of the number of non-cisgender people in the general population are quite variable, and studies estimating their numbers are often suffer from problems of definition, etc. If we look at various numbers [1] we see estimates ranging as high as 2% for people who feel transgender. I believe that the study which produced the 0.3% number you cite counts only those who have transitioned; but not all people who are trans undergo physical or social transition, many do not. And of course, since we are seeking to sample from all non-cisgender people, the portion of the population expands.

To answer your question directly: no, to our knowledge there is little to no hard evidence that they are not participating. There is also little to no evidence that they are participating. Because very little work has been done on the matter. However, if we consider how under-represented many communities are within Wikipedia, we find it reasonable to suspect that non-cisgender people are likely participating at a rate well below that of their population make-up, and the value they could bring to the community is being lost. But the only way to know for sure is to go look for them, and, if they are indeed absent, find ways to best invite their participation. --Archimedes52 (talk) 03:32, 4 June 2016 (UTC)

Round 1 2016 decision

This project has not been selected for an Individual Engagement Grant at this time.

We love that you took the chance to creatively improve the Wikimedia movement. The committee has reviewed this proposal and not recommended it for funding, but we hope you'll continue to engage in the program. Please drop by the IdeaLab to share and refine future ideas!

Comments regarding this decision:
The committee readily saw the merit in increasing understanding of the lack of gender diversity on Wikipedia. They also recognized the value of the skills this team of applicants brings together, and appreciated your efforts to revise your proposal in response to the first round of community and committee feedback. Nevertheless, in spite of the revisions, the committee continued to express concerns that the scope of this proposal is not yet appropriately scaled to be likely to succeed within the constraints of the IEG program. Though they recommended against funding this proposal in its current state, they agreed that they would welcome a further revised version in a future round.

Next steps:

1. Review the feedback provided on your proposal and to ask for any clarifications you need using this talk page.
2. Visit the IdeaLab to continue developing this idea and share any new ideas you may have.
3. To reapply with this project in the future, please make updates based on the feedback provided in this round before resubmitting it for review in a new round.
4. Check the schedule for the next open call to submit proposals - we look forward to helping you apply for a grant in a future round.

Questions? Contact us.