Research talk:Women and Wikipedia

Question set[edit]

I have a few questions that I'll elaborate on below. When considering these questions, I also recommend looking over the checklist I'm working from.

Sample size[edit]

How many completed surveys will you need? --EpochFail (talk) 14:57, 10 December 2013 (UTC)[reply]

The surveys will be used only for screening purposes. If snowball sampling yields 20+ participants, surveys will not be used at all. Mssemantics (talk) 19:47, 10 December 2013 (UTC)[reply]

Recruitment message[edit]

Can you post the message you intend to use to recruit participants? Also, it says that you'll be posting messages to "userpages", but I strongly suspect you meant "user talk pages". --EpochFail (talk) 14:57, 10 December 2013 (UTC)[reply]

Yes, I meant "user talk pages." Thanks for the catch; I'll fix that!

The message will read along the lines of

Hi, I'm currently a PhD student at the University of Washington's Information school, and I'm working on a research project about women and Wikipedia.

I've put together a screening survey for female editors of Wikipedia to see if you might be willing to participate in an interview. The purpose of the interview (and the larger study) is to explore women's motivations and experiences within the Wikimedia community. It'd be wonderful if you could participate.

Just click this link to participate in the screening survey.

If you have any questions or concerns, feel free to email me or stop by my user talk page. Also, feel free to share this any other female Wikipedians you may know. I am primarily interested in female Wikipedians who contribute to the English langaguage site, but I'd be happy to have other participants as well.

Thanks! Mssemantics (talk) 19:58, 10 December 2013 (UTC)[reply]

Looks good. Copied to Research:Women_and_Wikipedia/Recruitment_message and linked from project page. --EpochFail (talk) 23:48, 12 December 2013 (UTC)[reply]

Survey/interview questions[edit]

Usually we request a copy of the survey or interview questions before completing the review. Would you be able to provide such a copy? --EpochFail (talk) 14:57, 10 December 2013 (UTC)[reply]

I would prefer not to provide a copy of the interview questions as I don't want to prime and/or bias any potential participants. If you'd like a high-level summary of the interview protocol, I'd be happy to provide that. The screening survey hasn't been drafted, but it will consist of basic questions re: self-identification of gender, willingness and availability to participate in an interview, and preferred method[s] of contact. Mssemantics (talk) 20:03, 10 December 2013 (UTC)[reply]

Note that, for anyone following this conversation, I am working with Mssemantics over email to determine an appropriate method for sharing survey and interview questions. --EpochFail (talk) 20:25, 12 December 2013 (UTC)[reply]

In RCom, we walk a thin line between community advocacy and research support. On one hand, we'd like to help Wikipedians protect themselves from survey fatigue and unethical practices. On the other, we want to help you perform important research. We usually aim for transparency to do the former and having a process at all seems to satisfy the latter.

It's in this context that I'd like you to interpret my challenge to your withholding study details because they *might* bias participants. When I ask if sharing your questions with us presents theoretical concern vs. a serious one, I'm wondering if you are optimizing for your own prerogative (results that minimize all theoretical confounds) or if you were seriously concerned that details about your survey/interview would invalidate your study. If it was the former, I'm obligated to pressure you into being more transparent for the benefit of the Wikipedia community.

All I really care to know is if you are going to ask for and particularly sensitive information that Wikipedians will want to know about in advance and if your questions are just a re-hash of the editor motivations surveys we see submitted to RCom every couple of months.

I recommend that you guys find ways to maximize transparency that won't raise serious methodological concerns. One way you can do this is by posting an overview of the topics that your interview questions will address (as you suggested). Also, I don't see a good reason that you can't post a copy of your screening survey when that is ready. I'd be mostly satisfied with these compromises. --EpochFail (talk) 21:15, 12 December 2013 (UTC)[reply]

I'm more than happy to post both a high-level summary of the interview questions today and an exact copy of the screening survey once it is drafted. Mssemantics (talk) 21:34, 12 December 2013 (UTC)[reply]

Looks solid and as expected. We can close this section once youve got the survey. --EpochFail (talk) 23:37, 12 December 2013 (UTC)[reply]

I forgot to mention that Mssemantics sent me the protocol via email and I can confirm that the sample questions represent the protocol well. --EpochFail (talk) 23:38, 12 December 2013 (UTC)[reply]

Consent form[edit]

Please provide a copy. --EpochFail (talk) 14:57, 10 December 2013 (UTC)[reply]

Here's a copy of the verbal consent. I can also provide a copy of the written consent if you'd like. It's similar.

Hello, my name is Amanda, and I’m a PhD student at the University of Washington's Information School. Today, I’d like to ask you some questions about Wikipedia. I'm interested in what motivates women to participate in Wikipedia. I will be asking you about your personal experiences editing Wikipedia and interacting with the community.

What you share with me today is confidential, only to be used between me and my fellow researcher for the purposes of learning more about women and Wikipedia. The information we collect from you may be shared in academic articles, conferences, or similar venues. However, this information will be reported in the aggregate and no information that could identify you, personally, will be shared. I will be audio recording this session, and once I transcribe the session, I will immediately destroy the audio file. Is that okay with you?

Do you have any questions for me? Mssemantics (talk) 20:26, 10 December 2013 (UTC)[reply]

Copied to sup-page Research:Women and Wikipedia/Verbal consent --EpochFail (talk) 23:40, 12 December 2013 (UTC)[reply]

Related work[edit]

There's been a few papers exploring gender/Women in Wikipedia as well as the motivations of Wikipedia editors. Could you give us a brief explanation of what your proposed study will show that previous studies have not explored? --EpochFail (talk) 14:57, 10 December 2013 (UTC)[reply]

This is a good question. In a blog post earlier this year, Adrianne Wadewitz noted that in all the press coverage she had read re: the category controversy, "not one single female Wikipedian has been interviewed or quoted." I began to look at existing studies re: gender and Wikipedia and Wikipedians' motivations and noticed that, although academics and researchers had been perhaps more diligent and rigorous, there was certainly a lack of female Wikipedians' voices in the literature. Judd Antin and a few others have done some qualitative work re: the Wikipedia community, but the vast majority of research is quantitative and, I would argue, can be somewhat reductionist in nature. I propose to carry out a qualitative, interview-based study that incorporates feminist research methods. Mssemantics (talk) 20:15, 10 December 2013 (UTC)[reply]

Without citing specific examples of the questions you intend to ask (that presumably have not been asked before), the larger research questions you intend to answer or citations to specific examples of previous work not asking/answering these questions, it's hard to see where you intend to make novel contributions. I'm also curious what you mean when you say "feminist research methods". Which methods occur in this category? --EpochFail (talk) 20:22, 12 December 2013 (UTC)[reply]

The Handbook of Feminist Research: Theory and Praxis edited by Sharlene Nagy Hesse-Biber is a good place to start. Essentially, feminist research methods start from a feminist epistemology (e.g. one that isn't positivist or post-positivist and generally allows for an understanding and acceptance of subjective truths and leans more toward an interpretative approach to research).

Regarding novel contributions, is establishing that requirement a part of the review process? If so, I would need to know which guidelines (e.g., how novel contributions are being defined here) are being used to answer that question. Collier and Bear (2012) explore conflict, criticism, competence, and discretionary time as factors that contribute to the differences in participation among men and women, but they consider only new editors, and their work is strictly quantitative. Antin et al (2011) in "Gender Differences in Editing Wikipedia" too consider new editors and focus on edit counts and contributions. Lam et al (2011) note "Taken together, our results for RQ3: Gender-Conflict hint at a culture that may be resistant to female participation. More research, including interviews, surveys, and focus groups, is needed to determine the underlying causes of the problems evidenced in our findings, and to determine what can be done to improve the situation." As for as research re: motivations, I've yet to find a study that differentiates between male and female Wikipedians. Therefore, I believe my proposed study can contribute by: 1) using a qualitative approach grounded in a feminist epistemology 2) considering edit counts and contributions within the greater contexts of how active female Wikipedians view their work and community. Mssemantics (talk) 21:47, 12 December 2013 (UTC)[reply]

It's not "contributions" that I'm concerned about so much as demonstrating that you aren't planning to ask the same old motivation questions that Wikipedians are periodically pestered with. Really, I just was hoping to get a statement like the one you've given me. So thanks. :) --EpochFail (talk) 23:43, 12 December 2013 (UTC)[reply]

Survivorship bias?[edit]

My first question would be; what are you guys doing to compensate for survivorship bias? The interviews will certainly be helpful, but I worry that by focusing on people who have survived in the current environment, the traits and motivations the research identifies will be ones that aid people in doing that. Obviously, that's a good thing, but it leads to the potential perpetuation of the culture that has alienated, say, less aggressive people, be they male or female. Are there any thoughts on reaching out to people who didn't end up participating, having tried? —The preceding unsigned comment was added by Okeyes (WMF) (talk • contribs) .

The scope of this study is limited to women who have "survived" the current Wikipedia climate particularly because they are outliers. Future work may include longitudinal case studies with new female editors and interviews with women who have left the community. Mssemantics (talk) 21:31, 12 December 2013 (UTC)[reply]

Procedural digression
Jumping in quickly for a point of order. Part of what occurs on these talk pages constitutes a semi-formal review process (albeit an undocumented one, and not really what you would call "binding") similar to that of a human-subjects review board. So we should make it clear, when we ask questions, whether we're raising community impact issues that probably need to be addressed before the research proceeds, or whether we're simply providing informal feedback, as peers, that is intended to help the researcher under review get the most out of their study. So questions like this one, which pertain primarily to the scientific validity of the study design, are valid and important, but don't need to be specifically addressed in order for the study to be "approved". Since our process is largely undocumented, it's probably not self-evident to the researcher under review which questions represent "issues to be fixed" and which ones are simply "points to be discussed". Perhaps in future we can just lump them all under two different standard headers: "Review questions" and "Peer feedback" or something? I don't in any way mean to pick on Oliver (tho he's thick-skinned enough to take it in stride, even if I did ;). I just want to raise the issue: if we're going to continue to vet proposed research studies, we should make it clear that our role in doing so is more akin to that of an Institutional Review Board (concerned with ethics, costs, and consequences) rather than, say, a review committee for an academic journal. Jtmorgan (talk) 18:55, 12 December 2013 (UTC)[reply] Jtmorgan, if you have process concerns, I'd recommend bringing them up with RCom generally via the mailing list (rcom-l). FWIW, I disagree with your assessment of the relevance of Oliver's question. If anyone shows up here and has questions about the study, the researcher is obligated to address them. I am not the only valid inquisitor and we are not a university IRB. --EpochFail (talk) 21:51, 12 December 2013 (UTC)[reply] I felt the need to jump in and clarify because Okeyes posting his question from his staff account, rather than his personal account. In the context of a proposal review, comments made by a staff member may be taken differently than those of a fellow researcher participating in a volunteer capacity--especially for participants who are unfamiliar with the review process. The question itself was a good one. I've hidden this procedural comment in order to avoid side-tracking the discussion. Jtmorgan (talk) 23:49, 12 December 2013 (UTC)[reply] I think there are three dimensions not two. Is this ethically right, does it make academic sense and is it going to piss off the community. I think this project is pretty much ready to go, but I will raise some practical matters. WereSpielChequers (talk) 00:20, 20 December 2013 (UTC)[reply]

Procedural digression

Jumping in quickly for a point of order. Part of what occurs on these talk pages constitutes a semi-formal review process (albeit an undocumented one, and not really what you would call "binding") similar to that of a human-subjects review board. So we should make it clear, when we ask questions, whether we're raising community impact issues that probably need to be addressed before the research proceeds, or whether we're simply providing informal feedback, as peers, that is intended to help the researcher under review get the most out of their study. So questions like this one, which pertain primarily to the scientific validity of the study design, are valid and important, but don't need to be specifically addressed in order for the study to be "approved".

Since our process is largely undocumented, it's probably not self-evident to the researcher under review which questions represent "issues to be fixed" and which ones are simply "points to be discussed". Perhaps in future we can just lump them all under two different standard headers: "Review questions" and "Peer feedback" or something?

I don't in any way mean to pick on Oliver (tho he's thick-skinned enough to take it in stride, even if I did ;). I just want to raise the issue: if we're going to continue to vet proposed research studies, we should make it clear that our role in doing so is more akin to that of an Institutional Review Board (concerned with ethics, costs, and consequences) rather than, say, a review committee for an academic journal. Jtmorgan (talk) 18:55, 12 December 2013 (UTC)[reply]

Jtmorgan, if you have process concerns, I'd recommend bringing them up with RCom generally via the mailing list (rcom-l). FWIW, I disagree with your assessment of the relevance of Oliver's question. If anyone shows up here and has questions about the study, the researcher is obligated to address them. I am not the only valid inquisitor and we are not a university IRB. --EpochFail (talk) 21:51, 12 December 2013 (UTC)[reply]

I felt the need to jump in and clarify because Okeyes posting his question from his staff account, rather than his personal account. In the context of a proposal review, comments made by a staff member may be taken differently than those of a fellow researcher participating in a volunteer capacity--especially for participants who are unfamiliar with the review process. The question itself was a good one. I've hidden this procedural comment in order to avoid side-tracking the discussion. Jtmorgan (talk) 23:49, 12 December 2013 (UTC)[reply]

I think there are three dimensions not two. Is this ethically right, does it make academic sense and is it going to piss off the community. I think this project is pretty much ready to go, but I will raise some practical matters. WereSpielChequers (talk) 00:20, 20 December 2013 (UTC)[reply]

Screening survey on user talk[edit]

Hi Mssemantics. I saw that you posted a set of screening survey questions. Thanks! They look great. However, it appears that you're planning to provide the survey through a post on a user's talk page. I'm not sure it's a good idea to ask a personal question like "Do you self-identify as female offline?" in a public place. It seems like it would be better to ask this question through a more private medium. --EpochFail (talk) 15:23, 19 December 2013 (UTC)[reply]

Hi EpochFail. I plan to post a link to the survey (e.g., not the survey itself or the questions) so that only the individual and myself can see the questions/answer. This ("Do you self-identify as female offline?") is an important question to ask in the screening process as I'm interviewing women only. If you've advice as to a better way to proceed, however, I'm open. Mssemantics (talk) 21:13, 19 December 2013 (UTC)[reply]

Nope. That sounds great. Just so long as you're not posting the question directly on their talk page, I think it is fine. Upon re-reading my topic post, I can see how that wasn't clear. --EpochFail (talk) 21:17, 19 December 2013 (UTC)[reply]

Straw poll: Is this research project ready for RCom's approval?[edit]

The topic says it all. Mssemantics, note that you're welcome to participate in this discussion and respond to votes. My goal in starting this straw poll is to make sure that there's nothing I missed in my review of the project, so I'll be inviting other RCom members and discussion participants to come back to record their "vote". (ping User:Ironholds (aka User:Okeyes_(WMF)) & DarTar)

approve - This study is well described. Unlike many other Wikipedian motivation surveys, this study's methods and intents differ in that an interview will be aimed at Women and the experience of being a female Wikipedia editor. Most importantly, the study will affect a minimal amount of Wikipedians (~20) so it is unlikely to result in any disruption. --EpochFail (talk) 21:24, 19 December 2013 (UTC)[reply]
approve I have some concerns that I will raise above, but nothing to merit an oppose. WereSpielChequers (talk) 00:20, 20 December 2013 (UTC)[reply]
approve No reason to oppose and the topic is very relevant
approve Enjoyed reading the proposal and I think it's critical that we have more women doing more research about a subject that directly impacts them, and features qualitative research as well. SarahStierch (talk) 18:43, 23 December 2013 (UTC)[reply]
approve Important topic, well thought-through proposal (thanks EpochFail for shepherding the discussion) and expected low disruption of the recruitment method and sample size. DarTar (talk) 22:45, 23 December 2013 (UTC)[reply]

It looks to me like the consensus is clear, so I'm closing this poll as approve. I advise Mssemantics to continue with the proposed study when ready. Thanks for your patience, folks. --EpochFail (talk) 22:54, 23 December 2013 (UTC)[reply]

approve I am especially in favor of this kind of qualitative approach about motives and opinions. Ziko (talk)

Thanks![edit]

Many thanks to EpochFail, Jtmorgan, and others who have asked good questions, provided insightful feedback, and helped me get my first Wikipedia research project underway. I'm excited to start...and to share what I learn. Mssemantics (talk) 01:38, 31 December 2013 (UTC)[reply]

I'm looking forward hearing more! Jmorgan (WMF) (talk) 21:33, 2 January 2014 (UTC)[reply]

IEG Survey[edit]

Hey Jmorgan (WMF)! Can you please review the section I've just added re: doing a large scale (hopefully!) survey for my related IEG project? Thanks!--Mssemantics (talk) 17:10, 23 August 2014 (UTC)[reply]

Looks great to me, Mssemantics. I've set up your Qualtrics account. Thanks for checking in! Cheers, Jmorgan (WMF) (talk) 16:15, 25 August 2014 (UTC)[reply]

Open data[edit]

Will the (anonymized, of course) dataset be available for other researchers? --Piotrus (talk) 03:31, 15 October 2014 (UTC)[reply]

Only if interview participants consent to having their transcripts made available. Their safety, privacy, and confidentiality is of primary concern. Given the relatively small community of core editors, even a mention of a specific interaction or edit could be identifying data. --Mssemantics (talk) 04:48, 13 February 2015 (UTC)[reply]