Talk:Community Insights

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search

Costs for this survey[edit]


how many money does this survey cost. And why do you use a external company for this survey. I think the most questions of this survey are multiple choice questions and there it does not take a long time to know how many persons have chosen what for a answer and how were the questions translated. I think there have been translated some things into German where it doesnt make sense. For example Village pumps was translated with Dorfpumpen. This is a translation but does not exist in the German Wikipedia as a Diskussion site. Is it possible to get the text of the questions in German so that I can check the questions and tell you when I think that another translation of a word were good. -- Hogü-456 (talk) 18:30, 10 September 2019 (UTC)

As Hogü said, the translations are terrible. For example there is a question about how I deal with new users and one option is "I post a message on their talk page, using a barnstar or a predefined message" while the german translation is "Ich poste eine Nachricht auf ihrer Talkseite mit einem Barnstar oder einer vorformulierten Nachricht." The translation is very close to the original, however there is one problem: Barnstars don't exist in the German Wikipedia. Another example: Ihe part where I am asked where I am from I can select a country from a list which is sorted alphabetically in english - but obviously is a mess when viewed in german. "Deutschland" can be found under "G", "Österreich" can be found under "A". It works in english but not if every entry in the list is translated individually. It's obvious that this survey is made by enwiki users for enwiki users. I hope that the surveys taken are not all grouped together but are instead analyzed in seperate groups depending on the language the survey-taker used. On another note: The final part asking about demographic data is pretty weird. For example the question about gender has the options "male", "female", "transgender", "non-binary". Sooo apparently trans people are neither man nor woman? And when asked about age the minimum available is 18. Are people below 18 not allowed to participate on Wikimedia projects? Both of these don't affect me (I am neither trans nor below 18) but I wanted to point this out anyways. ーŒ̷͠²ð·¨´´̢́̕͘³͏¯̞̗ 【🅱alk】 17:44, 21 September 2019 (UTC)
Hello Œ̷͠²ð·¨´´̢́̕͘³͏¯̞̗! We are aware that there are some issues with the translations-- they were completed by a third-party service. We had hoped to have volunteers make corrections to the survey translations, but as this is such a large request many of the languages did not get reviewed, and German is one of them. We are manually fixing errors (like the one mentioned at the top of this page) as we can, but some we are stuck with--in particular the problem of alphabetization you refer to. We intended to make the list of regions and countries alphabetical in each survey language, but it is not possible due to the survey infrastructure. Please know that this is a known issue, and we are looking into ways to fix it for next year--same goes for overall translation quality. As for the gender question, we ask respondents to select all that apply in order to make the question as inclusive as possible. For the age question, we are bound by legal restrictions on surveying people under 18 without parental consent. I appreciate your feedback! I do have one question for you-- is there another "vorformulierten Nachricht" that we might use to substitute for "Barnstar" in the German version of the survey? I'd like to fix the error you are pointing out, but I'm wondering if there is an example of a "predefined message" I can substitute that is more relevant to German Wikipedia. Thank you again for your time and feedback! --RMaung (WMF) (talk) 22:06, 27 September 2019 (UTC)
@RMaung (WMF): Thanks for your reply and sorry for my late answer. There are several templates in dewiki to greet new users, see de:WP:Begrüßung#Standardisierte Begrüßungsvorlagen. I think the phrase could be translated as: "Ich hinterlasse eine Nachricht auf seiner/ihrer Benutzerdiskussionsseite unter Verwendung einer Begrüßungsvorlage." (that would be the literal translation of "I leave a message on his/her user talk page using a greetings-template.")
Oh and about the age thing: if you are not allowed to survey people under 18, why don't you just add the following option: "I am below 18 and my parents or guardians are answering this survey for me" That way kids their parents of course know what to pick and you got your age data in a totally legal way. :P Then again adding options to questions after some people already answered them is not fair, however in the next survey this could be an option. ーŒ̷͠²ð·¨´´̢́̕͘³͏¯̞̗ 【🅱alk】 15:57, 5 October 2019 (UTC)
If the software does not allow to use different alphabetical orders for different languages you might want to use international codes in future, like "DE - Germany", "DE - Alemania" etc. Respondents should be able to find their own international code in a medium (internet) where such a code is commonly used. → «« Man77 »» [de] 16:38, 27 October 2019 (UTC)


If you want your data to be representative, maybe you should choose another service provider. Qualtrics has sanctioned all those living in Iran. 4nn1l2 (talk) 03:56, 10 October 2019 (UTC)

@4nn1l2: I realize this response is coming incredibly late, but it is still relevant to this year's survey: Qualtrics does not block Iran, it is the result of the political relationship and resulting data embargoes between the governments in Iran and the US (and thus any survey that originates in the US, no matter which provider we use). We are hopeful that the situation will change in the future. RMaung (WMF) (talk) 15:16, 9 October 2020 (UTC)

Survey design[edit]

The failure of this survey to explain to ordinary editors what "...implementing your project or scheme..." actually means made it virtually impossible to answer one whole set of questions. Having then moved forward through the survey to try to understand the context of those questions, I found there was no back button (just before the personal info section.) It was thus impossible to go back and give proper answers. I'm now feeling I've spent 25 minutes answering a survey that's not applicable to my level of editing activity on en-wiki only. Nick Moyes (talk) 20:08, 17 October 2019 (UTC)

2019 Survey?[edit]

What is the current status of the 2019 Survey? Antrocent (talk) 06:16, 13 February 2020 (UTC)

Confusion in questions[edit]


@RMaung (WMF): I'm not sure where to provide this feedback, so I'll be putting it here. As I'm answering this gigantic survey, it's sad to see that again there's confusion on terminology - some on rather important questions. Especially:

  • "For the Wikimedia projects you participate in, how much do you agree or disagree with the following statements about user conduct policies?" It is unclear what is being referred to. I would guess community policies, but the question is asked right after a question about WMF's policies ( Code of Conduct for Wikimedia technical spaces, Trust and Safety) and right before Wikimedia Foundation’s Friendly Space Policies. It's likely that different respondents interpret this question differently, with a bias based on how much they dislike WMF's policies.
  • When talking about actions towards newcomers, one line states: "I post a message on their talk page, using a barnstar or a predefined message" which is a super wide array of possible interpretations. The context suggests you imagine this to be positive, but the most common 'predefined message' would probably be templates giving them a reprimand/warning. Some people may interpret this question as such, and some may think you're only talking about positive messages.
  • For the question "About how often have you done each of the following:" there's a huge gap between "A few times per year" and "once or twice per week". With such vague scales, it's likely that more modest people round down by a lot, and people in less modest cultures round up a lot. This may make a discrepancy in support seem apparent that is not there (or make a discrepancy the other way around disappear). This is especially relevant for 'rare events' such as "Mentored someone closely to help them implement their Wikimedia program, event, campaign, or partnership". If you do that once a month, that is HUGE. But it would still possibly disappear in the noise of 'a few times a year' which may be also selected by the also admirable volunteer that closely mentors someone once every two years (because it's more than never).

In general, the survey assumes that the respondee fits one single profile. That I 'lead' one project. That I contribute to one community. This is maybe true for some, but many people fill multiple roles. It is unclear to me how I should fill out questions like 'Does the community you organize or lead have established policies and/or procedures for group decision-making?'. Is that always the same community as I responded about before? Sometimes a response can be multiple communities. I imagine you could give instructions to pick one and stick with it, but I totally didn't see that kind of instruction this time.

The number of questions should really come down a *lot* :) This is just an exhaustion exercise and it's impossible for me to stay focused for this number of questions. It's definitely longer for me than the 40 minutes max that you promised (I think it may have taken about an hour). Maybe you could warn people more fairly about this, and allow them to only fill out certain sections (choose at the start) and then allow to continue to other sections at the end.

Finally, I would have appreciated a note that states the obvious: the WMF is running this survey, but no individual responses will be shared outside the survey-data-folks (even within the organization). This is a bit implied in the privacy policy that was hidden behind a link, but not explicit. This is the equivalent situation of the WMF being landlord (runs the projects), employer (runs the grants programs), supervisor (runs the movement process) and also runs the survey. I definitely presume a firewall situation, but can't realistically rely on it. Even when you remove personal identifiers, there is plenty of data points in the demographic questions to identify any community member based on their responses alone (combination of language, country of residence and start year would be plenty in my case).

I appreciate the efforts, but I wish that these surveys were much much shorter, and easier to digest to make them more accessible.

Effeietsanders (talk) 22:10, 23 September 2020 (UTC)

Yes - I agree with much of the above. About an hour in and I found myself answering questions as if I were major lead in a whole host of teams and projects, rather than a humble contributor. Maybe the fact that I did try to answer them was unhelpful? - I don't know. Yes, the timescale breakdown for things like frequency of mentoring were ridiculous and bound to produce wildly inaccurate results. Nothing asked whether the respondee helps new editors in any help fora, either. And terms were bandied around as if we all know what they are - not a hyperlink in sight (for me to work out what an affiliate is, for example, or to check whether or not I had read or remember the relevant policy being referred to.) Over all - far too long-winded. Nick Moyes (talk) 22:23, 23 September 2020 (UTC)
Hi @Effeietsanders: @Nick Moyes: Thank you for your feedback! We are working to improve the survey's length, reliability, and validity every year. The majority of respondents do take between 20 and 40 minutes to respond to the survey, but some take less and some more, depending on how a respondent participates in the movement. I apologize that you feel you saw questions that were not relevant to your participation, @Nick Moyes:, you likely saw those because you had indicated that you lead a project, event, campaign, or group. I'd love to hear more about how we might better phrase that question so you don't see irrelevant sections.
I appreciate the concerns about data interpretation-- please know that we keep this in mind when we report and interpret data. For example, the question about how often the respondent has "mentored someone closely." I agree-- just doing that once is a big deal, and it is not lost in the data. We ask 'how often' so that we can better track change over time, both in the proportion of organizers who mentor others at all, but also how frequently they do so.
For future iterations of the survey, we will add hyperlinks for those terms which might not be apparent to all. I will also work with our Legal department to make sure that section of the survey's privacy policy is even more clear. Thanks again to you both. [[User:|RMaung (WMF)]] (talk) 15:09, 9 October 2020 (UTC)
Thanks for the response. Just a sidenote: this nuance is lost in the data, when a not-appropriate scale is chosen that confuses the respondent, and people err on the side of caution, or the side of bragging (not sure how else to phrase this), depending on their cultural background or social status. I think in general it is sub-optimal to have respondents interpret what your scale means, but that is a more subjective discussion. Either way, by now it's moot to try to improve anything, even if you wanted to.
I hope for a significant improvement on both length and reliability - as these introduce biases that will then be relied upon for many strategic decisions. This is a sustained problem for quite a few years now. Effeietsanders (talk) 16:29, 9 October 2020 (UTC)
@RMaung (WMF): Darn - I drafted a long yesterday, but lost it all in an edit conflict. No time to rewrite it today, but, in essence, I'd suggest that because you guys probably think much bigger and more strategically than the majority of us mere mortals, that you create a hierarchy of questions from simple involvement to more deeply embedded in WMF work. Then the right questions only need kick in at the relevant point. So, if I've helped participate in or helped to run a local editathon/meetup/online training sessions once or twice, I don't get marked as some major project organiser, as if I ran Women in Red or a Wikimania project. Yes, hyperlinks would be good. Having terms defined or explainable as you go along is essential to respondees understanding what it is that you are asking, and for you to get the right answers. Do you have a panel of non-WMF staff from various worldwide projects on whom you trial these surveys before they're rolled out? If not, getting a few volunteers together might help you iron out problems or gaps in understanding. Regards, Nick Moyes (talk) 10:15, 10 October 2020 (UTC)

How to add a translation into Russian?[edit]

Hello! How to add a translation into Russian? I wrote it, see: Community_Insights/ru Зайва Игорь Леонидович (talk) 06:30, 26 September 2020 (UTC)

@Зайва Игорь Леонидович: Hi! what in particular are you translating? This meta page, or the survey, or something else? Thank you! RMaung (WMF) (talk) 15:11, 9 October 2020 (UTC)
@RMaung (WMF): Hi! --Зайва Игорь Леонидович (talk) 18:37, 9 October 2020 (UTC)

Access Denied[edit]

It's a little bit sad, given the stated aims of this research and other things coming from the WMF, that Qualtrics prohibits anyone responding to the survey, or in fact accessing any part of their site including their privacy policy and ToS, over the Tor anonymity network. And no nice message either, just a raw 403. -- zzuuzz (talk) 20:22, 28 September 2020 (UTC)

@Zzuuzz: Thanks for letting me know-- I'll reach out to Qualtrics to see if there are any ways around this. Qualtrics has a variety of benefits and functionality, but I acknowledge that there are some drawbacks. For this year it was the best available option for us to use. RMaung (WMF) (talk) 15:13, 9 October 2020 (UTC)