Research:Evaluating article recommendations for content translation/Interview protocol

This is a semi-structured interview protocol. Not all questions will be asked in every interview, and the investigator may ask additional probe questions if interesting things come up. When possible, the investigator will ask the participant to share links to pages that they talk about or diffs that point to particular actions or events which are referred to over the course of the interview.

Protocol

Thanks for agreeing to speak with me today. The purpose of this interview is to evaluate a new system for recommending articles to translate from one Wikimedia project to another. I'm taking with people like you, who regularly translate articles, in order to increase the quality of these recommendations. During the course of our discussion today, I'm going to ask you about the translation work you do now, and the way you perform that work. Then, I will share with you a link to a prototype of the article recommendation system, ask you to use it a little bit, and then ask you about your impressions of the recommendations and the system as a whole.

Any feedback you provide is valuable. There are no 'right' or 'wrong' answers--I'm testing the system, not you. If something abut the system confuses you, tell me about it, and I will share your responses with the system developers so that we can (hopefully!) make the next version less confusing.

You can stop the interview at any time, for any reason. If you do not want to answer a question I ask, you don't have to. If you have any questions for me, you can ask at any time. Ready to get started?

Background

First, I'd like to learn a little bit about your background as a contributor to Wikimedia Projects

• When did you first start contributing to Wikimedia projects?
• What Wikimedia projects do you contribute to most?
• What Wikimedia projects do you read most?
• What kind of work do you do on Wikimedia projects (other than translating)?

Now I'd like to ask you some questions about your translation work on Wikimedia projects

• What kind of translation work do you do on Wikimedia projects?
  • Have you done other translation work, like translating messages via translatewiki.net, or using the translation extension on multilingual wikis?
• What languages do you translate to/from?
• What Wikimedia projects do you translate to/from?

Translation process

Now let's talk about your article translation work specifically. I'd like to learn more about your background with translation.

• When did you first start translating articles for Wikimedia projects?
• Why did you start translating articles?

Now I'd like to talk specifically about the process you go through when you translate articles

• Tell me a little about how you get started with a new translation. What do you do first?
  • How do you decide whether an article is important to translate?
  • How do you decide whether an article is ready to be translated?
  • What kind of articles do you enjoy translating the most?
• What tools do you use in your translation work: for example, to find articles to translate or to perform translation?
  • ContentTranslation, labs/toolserver tools, etc?
• Can you show me an example of an article you translated recently?
  • why did you translate this article?
  • have you translated articles like this before?
• What are you thinking about translating next?
• Have you ever started translating an article, but then decided to abandon that translation? Why?

Prototype

Now I'm going to share with you a prototype of a new tool that's designed to help translators like you decide find articles to translate. The way this tool works is that you enter a title of an article that exists on a 'source' wiki (in this case, English Wikipedia), and then you specify a 'target' language that you are interested in translating articles to. Then the tool gives you a list of articles that are similar to the article you entered, that (it thinks!) don't exist in your target language. Here's the URL to the tool...

A couple things to keep in mind while you use this tool.

1. The recommendations are real, but the page you are on is not what the final interface will look like. So don't worry too much about the way it looks (although you're welcome to give feedback on that, too).
2. What we're most interested in is whether the articles that the tool is recommending are useful to you. So I'm going to ask you a series of different questions about the recommendations you see here.

• I'd like you to think about how these articles are related to the source article
• I'd like you to tell me about how important/appropriate are these articles are to the target wiki?
• Are these articles ready for translation?
  • what would they need to be ready for translation?
• Are there articles that you expected to be on this list, that aren't?
• are any of these articles one's that you'd like to work on next? How likely?