How to move forward
- What was this session about?
How survey can be used to understand better the communities you serve.
- What are the next steps to be taken?
- Form a list of people who have survey experience in the movement to increase knowledge exchange around surveys
- Build up a list of surveys that can be borrowed and re-used by affiliates, program and other community leaders (see also Surveys/Learning) .
- Who is the person to reach out to?
This session documentation was approved by the facilitator.
- Original Description
- Listening to community voices in a balanced way is key to making decisions that are collaborative and legitimate, important qualities for our movement. Surveys are one way to bring in the voices that may otherwise not be heard. They are not just about getting input of data, but are a way to gather insights about needs, challenges, or opportunities of communities. In this session, participants will learn about how organizations are applying new approaches to making decisions through surveys, using case studies to highlight the potential of surveys. More importantly, participants would be able to learn from each other about when surveys are used, when other approaches to community listening are appropriate, and what should be avoided.
- Desired Outcome
- Participants will have a better understanding for when to use surveys and how to use surveys
- Create a detailed best practice list about what participants would like to see about surveys (design elements & SWOT), which can be used by participants.
- Participants will get to know survey resources and people with survey experience
- Session Format
- Conversing & Working (Panel & workshop. We will have 4 presenters (5 minutes each) with discussion groups following)
- Edward Galvez, with the panelists Felix Nartey (Wikimedia Ghana User Group), Sandra Rientjes (Wikimedia Nederland) and Anna Torres (Wikimedia Argentina)
- Summary of the session
Edward Galvez opened the session on how to understand communities using surveys. He gave a short introduction explaining that surveys mainly can be used as “flashlight in the dark” to understand certain aspects of communities. You see some aspects highlighted, but you don’t necessarily see the context. Edward pointed out that trying to learn from a large group of people was definitely a challenge.
After his short introduction, Edward first handed over to Felix Nartey (Wikimedia Ghana User Group), Anna Torres (Wikimedia Argentina) and then Sandra Rientjes (Wikimedia Nederland).
Felix told the audience the experience of his user group doing a professional survey (with the support of WMF) with the goal to sum up activities of a pilot project for scaling up offline Kiwix in Ghanaian schools. The main goal (reasons) of the project was to find alternative/auxiliary means to increase the reach and (or) use of Wikipedia and its sister projects. The group learned a lot doing this survey, e.g. that the sample size didn't necessarily reflect the entire population that used the resource (see the slides).
After Felix’ presentation, Anna took over and presented Wikimedia Argentina’s experience with surveys. They conducted a survey to meet their community of editors to expand their activities outside of Buenos Aires. The main goal of the survey, which was created together with its board and WMF staff, was to “localize” the editors around Argentina and to be aware of their knowledge and activities. Two findings of the survey were important: 41% didn't want to participate in Wikimedia Argentina’s activities and a great number of editors believed that the main support from Wikimedia Argentina should be providing information and resources to editors.
Sandra from Wikimedia Nederland explained her chapter’s experience with a big survey they had conducted last year. There were signals that many Wikipedians had seen problems in the working atmosphere; dealing with newbies, gender gap, etc. Wikimedia Nederland felt, on the other hand, that Wikipedians were difficult to reach, as the chapter is only in contact with around 200 Wikipedians. This was a reason to do a proper survey and focus on community health issues. As the chapter did not want to create and conduct the survey completely by itself, an expert was hired to create the survey and to process the results. The questions were created together with the staff and community members. The survey was announced (and conducted) using a SiteNotice for about ten days. Finally, 452 of around 1200 completed the survey. The idea, Sandra explained, was to do such a survey in ~ 2 years with a different focus.
After the lightning talks, the audience was split up into four groups. After the group discussions, they summarized the following important points for creating, conducting and processing surveys:
- Make clear whom you are targeting with your survey
- Clear and exact questions → no language problems, everyone understands the same
- Problem: sometimes you set an agenda with the questions with ask (bias in the survey itself)
- You can also bias the results with the method (e. g. Google survey: some people don’t want to use Google)
- It is helpful to have an expert who knows how to design a survey to get meaningful results
- Make sure you have not too many questions and not the only one type of questions → balance
- Central point: make clear why you are doing the survey → also makes it easier to explain and communicate the survey
- Pretest the survey: show it to the board, community members or other stakeholders; possibly refine it
- Check the questions in detail: is each single question really useful?
- Don’t do it too often
At the end, Edward mentioned further resources around surveys
- Learning Patterns
- Survey Support Desk
- [[Category:Survey skills]] (people who have experience with surveys)
- Qualtrics, the online survey platform provided by the Wikimedia Foundation
- surveyswikimedia.org for help
The groups wrote down the following aspects:
- Think what the results will influence
- Have questions relate to your expectation
- Test the survey with people
- Board members + Wikipedians
- Clear + Distinct
- Why should WMNL do the survey → may get negative feedback about the survey
- Challenged why are you spending money on this
- Posting on village pump + newsletter
- People complain “I did not hear about it” / “I did not want to hear about this” / Related to topics they did not like / Do not contact
- List of questions are specific related to where there was a problem (Biased in this way)
- For proper survey you need to watch out for our biases
- Doing survey about community for replication in other communities
- Ask communities for input about approaches
- Communication + Conflict training + Gender
- Maybe offline workshops
- Pretesting with board / community members → set up a test data set
- Know and communicate the “why” of your survey
- Be clear about how answers can lead to actions
- Communicate in the right channels → e.g. local “village pump”
- Don’t do it too often
- Focus on questions you really need → do a check for each question
- Understanding Demography
- Getting Information about interests
- Not enough responses
- Not representative results
- Targeting the wrong audience
- Right timing (close in time to a specific event)
- Test group
- Innovating & user-friendly design
- Bad survey design
- “Survey fatigue”
- Lacking motivation and polarized answers
||Community & Process (How)
- Need to target the people intended
- Know who to target and how – did not know how to use site notice
- Questions must be clear and exact
- Still agenda to what you ask
- Survey by WMF years ago: Questions can skew results if leading questions → leading to answers by method, including inclusion participants, Google surveys, some don’t want to use Google
- Analysis, profession, sometimes need to ask / have someone who knows what they are doing
- Double ck questions → ask experienced outsiders
- Pilot with small set of people
- Not too many questions or all same type of questions
- Open-ended questions are difficult to analyze
- Can cause fatigue
- Need balance
- Complex language → Response bias
- Community Involvement
- Discussion on list to see what we should ask from whom
- Learn what questions communities need to answer
- Allow to review + comment on designed draft
- Be cautious if surveys are for research in which knowing in advance may alter future answers