Research talk:Finding a Collaborator

Dissemination

I'm here after seeing this note at WT:VG on en-wp. Unfortunately I can't expect to have an uninterrupted block of time in which to participate myself. However, it'd be very handy if the results could also be linked to at the same places where notifications (such as the WT:VG one) were made. Thanks, and I hope it goes well. -- Trevj (talk) 10:38, 17 September 2013 (UTC)

Review

Thanks for posting this research proposal. I have a number of questions that I'd like to see addressed in the proposal:

Goal of the study

The goal of the study is too generic. There is no description of what the "new visualization tool" is designed for and how it is expected to support editor decisions other than a vague reference to visualizing a "credibility" in Wikipedia". There is no explicit research question or research hypothesis that the study aims to test, which makes it hard to assess whether the subject recruitment request is appropriate. Can you expand the project description and provide more information on the goals of the study?

The new visualization tool is designed to enhance searches of potential collaborators within a Wikipedia community. The history of edits for each wikiproject will be used to visualize the level of editors. Research questions are as follows: How will our visualization tool enhance searches of potential collaborators? How will our visualization tool be used during the search process? Would this tool support users to find potential collaborators when editing Wikipedia? What changes to the tool would make it more helpful to users?

The project description with more information on the goals of the study is as follows. The purpose of this study is to understand how Wikipedia editors find their collaborators, and to measure the usability of a visualization tool for Wikipedia collaborator finder that enables users to know about potential collaborators and their status. This study will help us determine the nature of Wikipedia collaborator finding, and also give us how the usability of the tool can be improved. If participants decide to participate in the study, we will start to have a session via online video. This appointment will last about one hour. During the study participants will be asked some basic questions about experiences of the participants and their feelings about our visualization tool. And the participants will perform tasks that are related to collaborator finding in Wikipedia. The participants must be an experienced and registered editor to participate in the study. At any point the participants may pause or stop the study by simply letting the researcher know of their wishes. We will do everything possible to maintain the confidentiality of information of the participants. Videos, software data and survey data will be linked with a session number. Video recordings will be kept for no longer than five years. All study data will be maintained on password-protected computers associated with the research team. Session numbers rather than user names will be used to identify data files.

Please wrap this content into the project description. --EpochFail (talk) 19:48, 30 September 2013 (UTC)]

I wrapped this content into the project description. Wkmaster (talk) 11:49, 2 October 2013 (UTC)

State of the art

There is a large body of works on tools to surface content credibility/trustworthiness in Wikipedia (see for example w:WikiTrust), how does this proposal situate itself with respect to the literature? What's original in this study that hasn't been covered in past research? What benefits do you expect this study, if successful, will provide to the community?

This proposal situates itself clearly on the topics of credibility/trustworthiness with respect to the literature. The original in this study that has not been covered in past research is that our new visualization tool represent four distinguished metrics to support decision-making process to identify potential collaborators. If successful, we expect the we understand behavior of editors in finding collaborators. In addition, we will also learn how the new visualization is used in identifying potential collaborators.

It could be that we're struggling to understand what the under interface looks like and how it works. Can you please add a screenshot and description to the project write-up? --EpochFail (talk) 19:48, 30 September 2013 (UTC)

I added a screenshot and description to the project write-up. Wkmaster (talk) 11:50, 2 October 2013 (UTC)

Sample size

What is the minimum participant sample size for this study to be successfully carried out?

We expect that the minimum participant sample size is 15.

Please include this in the project description. --EpochFail (talk) 19:48, 30 September 2013 (UTC)

I included this in the project description. Wkmaster (talk) 11:49, 2 October 2013 (UTC)

Recruitment strategy

What is the target population for this study? Is it experienced Wikipedians? New editors? The proposal mentions an eligibility check based on whether "the person currently edits Wikipedia". Can you make this more explicit so as to better target the recruitment strategy and avoid collecting personal information from a large number of ineligible participants?

The target population for this study is experienced Wikipedians. The research study will be announced via public postings on Wikiproject websites. The announcement will include information about who will be invited to participate in user studies with the researchers. When individuals voluntarily contact the researchers to indicate interest in participating, the researchers will verify they are active Wikipedia editors via publicly available information in Wikipedia. If individual is qualified, the researchers will contact them via electronic message (email) with a description of the study, information about their right to not answer any question(s) and/or to discontinue the interview-based study at any time without penalty. They will be informed that they must be at least 18 years-of-age in order to participate. The message will also inform potential participants that the study will be recorded, but that their actual identities will not be used in any research report.

This sounds good. It should be wrapped up in the project description. Please include what the "description of the study" and consent form will be sent with the emails as well. --EpochFail (talk) 19:48, 30 September 2013 (UTC)

I wrapped this up in the project description as you suggested. Wkmaster (talk) 11:49, 2 October 2013 (UTC)

Personal data collection

What is the rationale for collecting participants' phone numbers? How can prospective participants verify the identity of the researchers posting the recruitment message? Have you considered using a talk page to verify the participant eligibility to avoid the need of collecting personal information by email?

The telephone number will be used only for scheduling individuals for participation in the user study. Prospective participants can verify the identity of the researchers posting the recruitment message by communicating with researchers via email with organization domain (uw.edu). We plan to use a talk page to verity the participant eligibility, but we still need to collect personal information by email for scheduling individuals for participation in the user study as mentioned above.

My apologies, but I'm still confused about the need to collect phone numbers. Will you be holding an interview over the phone? --EpochFail (talk) 19:48, 30 September 2013 (UTC)

We will not need to collect phone numbers. Wkmaster (talk) 11:49, 2 October 2013 (UTC)

Ethics committee approval

Have the researchers applied and obtained an IRB approval for this study?

Yes.

Was it approved? Please post details in the project description. --EpochFail (talk) 19:48, 30 September 2013 (UTC)

Yes, it has been approved. I posted details in the project description. Wkmaster (talk) 11:49, 2 October 2013 (UTC)

Most IRBs that I'm familiar with will provide a study identifier number and a phone number that can be called for verification. Please add this information to the project description. See Peer mentorship and snuggle for an example. --EpochFail (talk) 14:12, 2 October 2013 (UTC)

I added additional IRB information to the project description. --Wkmaster (talk) 23:26, 3 October 2013 (UTC)

Data and research output publication

Should the study be carried out and published, we would like to see the data/preprint of any research output made available as open access as per Research:Subject_recruitment.

We will show the data/preprint of any research output made available as open access as per Research:Subject_recruitment. Wkmaster (talk) 01:44, 22 September 2013 (UTC)

Thanks, DarTar (talk) 05:31, 18 September 2013 (UTC)

Just to add, whatever happens, if the proposal gets green light, one should not send requests from the IP address, only from a registered account.--Ymblanter (talk) 07:34, 18 September 2013 (UTC)

Mini-straw poll

Is this project ready for RCom approval? (ping DarTar, Ymblanter)

• ready The project is well described. Their recruitment strategy (posts in VillagePump, no direct contact to editors) will cause minimal disruption. --EpochFail (talk) 00:04, 4 October 2013 (UTC)

What is the RCom approval? --Wkmaster (talk) 20:41, 4 October 2013 (UTC)

Good question. The Research Committee (RCom) is a collection of Wikipedians, Researchers and Foundation staff working to support wiki research. We organize the subject recruitment reviews that you are currently participating in. "Approved" projects are vetted by one or more uninvolved RCom members. We're not signing off that a project is "good" so much as that it is well specified and non-disruptive to Wikipedians. We don't have any official say as to who can and cannot recruit research subjects on Wikipedia, but this approval process has become the de facto equivalent of a rule over time. See Category:Projects reviewed by RCom. For a relevant example, see Research:Re:Flex User Studies. --EpochFail (talk) 22:51, 4 October 2013 (UTC)

• Ready. I think the questions got answered in a satisfactorily manner, and I do not have any objections.--Ymblanter (talk) 17:16, 5 October 2013 (UTC)
• ready. Wkmaster, thanks for expanding the proposal and clarifying the above issues, that's really appreciated. The relatively small sample also makes me feel better about the recruitment targeting experienced Wikipedians. I don't have any other objections. I recommend that you create a user page on Meta and link to en:User:Wkmaster, so your username doesn't appear as a redlink --DarTar (talk) 04:21, 7 October 2013 (UTC)

Thank you for your advice. It seems that I created a user page on Meta here. Questions are: 1) how can I link the user page on Meta to en:User:Wkmaster? 2) what is the difference between User:Wkmaster and en:User:Wkmaster? --Wkmaster (talk) 12:52, 7 October 2013 (UTC)

It looks like we have clear agreement here. I've advised Wkmaster to proceed. --EpochFail (talk) 13:44, 9 October 2013 (UTC)

Thank you. I will then proceed. Am I allowed to post a recruitment message on the talk of any wikiproject pages? --Wkmaster (talk) 20:18, 9 October 2013 (UTC)

IRB verification

Please provide a link on the University's web site which contains contact information for the Institutional Review Board, so those wishing to verify the legitimacy of this study can do so. The linked-to page should be one that is obviously directly under the control of either the IRB or the University, not one under the control of a student, faculty member, employee, or department involved in this study. Thank you. Davidwr/talk 07:10, 9 January 2014 (UTC)

I included the following link under the IRB verification section of our wiki page at m:Research:Finding_a_Collaborator: http://www.washington.edu/research/hsd/ Wkmaster (talk) 22:04, 9 January 2014 (UTC)

Timeline is vague and ambiguous

Please do not use climatic seasons as substitutes for actual dates. Seasons vary across the world, some places, such as the tropics don't have four distinct seasons and of course summer in Europe is at the same time as winter in Australia. Please use months or quarters, which are the same everywhere. Dodger67 (talk) 09:10, 17 March 2014 (UTC)

The project template lists out:

* Start: 2013-07

* End: 2014-04

Is there some serious concern about the status of this project? --EpochFail (talk) 18:22, 17 March 2014 (UTC)

I'm concerned with adverse selection here

Many a valued Wikimedia contributor is unable or unwilling to use tools which require cameras, microphones, etc. , whether for physical or ideological or stylistic reasons. You are already pre-screening for the more audiovisually-minded and technologically sophisticated, not mention just plain wealthy enough to afford all this crap. Some of us are textually-oriented, and see no reason why we should have our experience deprecated or ignored. --Orange Mike (talk) 21:13, 31 March 2014 (UTC)

And...

Anything happened? Rich Farmbrough 14:43 13 April 2014 (GMT).

Current status? (July 2014)

The description of this study says that it was supposed to have been completed in April 2014. That was three months ago, yet I see no indication that the study has even begun. What is the current status, please? (attn: Wkmaster) — Jaydiem (talk) 20:10, 23 July 2014 (UTC)

I just sent Wkmaster an email to ask him to come here to comment. --EpochFail (talk) 23:41, 23 July 2014 (UTC)

Thank you for your comments. I modified the end date to make sure that the work is still in progress. Wkmaster (talk) 22:05, 5 August 2014 (UTC)