Survey best practices

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search

Chapters, the Wikimedia Foundation and other organisations have conducted surveys of Wikipedia editors. This page collects best practices for those conducting surveys. These best practices are taken from governments and international organisations.

Demographic information[edit]

To aid with systemic bias and to understand the needs of Wikimedia contributors, some surveys have asked demographic information questions, on areas like age, gender, marital status, sexual orientation and household income.

Age[edit]

If considered necessary, use the question: What is your age?

The UK Office of National Statistics provides four general use age bands:

Harmonised Standard 1 Harmonised Standard 2 Harmonised Standard 3 Harmonised Standard 4
0-24
25-64
65-74
75+
0-24
25-44
45-64
65-74
75+
0-15
16-24
25-34
35-44
45-54
55-64
65-74
75-84
85+
0-4
5-10
11-15
16-19
20-24
25-29
30-34
35-39
40-44
45-49
50-54
55-59
60-64
65-69
70-74
75-79
80-84
85+

Use of narrower age bands with small sample sizes is not recommended as it has the potential for personal identification.[1]

Sexual orientation[edit]

If considered necessary, use the question: Which of the following terms best describes your sexual orientation?

Provide as possible answers:

  • Heterosexual
  • Gay/lesbian (or "Homosexual")
  • Bisexual
  • Other
  • Prefer not to say[2]

Sex/Gender identity[edit]

First approach[edit]

If considered necessary, use the following questions:

What is your gender?

  • Male
  • Female
  • Other (please specify)

Are you transgender?

  • Yes
  • No
  • Other (please specify)

If you are transgender, what was your (assigned) gender status at birth?

  • Male
  • Female
  • Not applicable
  • Other please specify

Taken from Gay and Gender research website survey (Social and Political attitudes of LBTQ individuals)

Second approach[edit]

The ONS 'Trans Data Position Paper' (Released in 2011)' [3] suggests some alternative questions

Is your gender identity the same as the gender you were assigned at birth?

  • Yes
  • No

Do you live and work full time in the gender role opposite to that assigned at birth?

  • Yes
  • No

(With an additional suggested follow up question)

Do you sometimes adopt the gender role opposite to that assigned at birth?

  • Yes
  • No

The briefing paper clarifies under no circumstance should the data ever be attributable to an individual (page 16) - so these questions would have to be anonymous, and notes that at the time, the ONS decided that data collection of gender or transgender status was not be included in standard household surveys. The second set of questions suggested are similar to those recommended by Unison [4]

Third approach[edit]

The recommendation of the Equality and Human Rights commission [5] in this regard are quite nuanced. These are undoubtedly the most thorough questions about gender, and were developed ...with a sample of the general public who identified with a range of gender identities and with the explicit aim of replacing the question 'Are you male/female?' on UK monitoring forms, national surveys or census.

Education[edit]

In the UK the ONS measures levels of qualification in the population between 16 - 64 years of age. This measure doesn't take into account type of degree, and therefore for example doesn't measure the difference between undergraduate, postgraduate and doctoral degrees[6]

The levels of qualification against which data is collected are:

  • Degree or equivalent
  • Higher education
  • A Level or equivalent
  • GCSEs grades A*-C or equivalent
  • Other qualifications
  • No qualification
  • Don't know

Household Income[edit]

Household Composition[edit]

The Office of National Statistics defines 'Family' as 'A married, civil partnered or cohabiting couple with or without children, or a lone parent with at least one child. Children may be dependent or non-dependent.' and a household as 'A person living alone, or a group of people living at the same address who have the address as their only or main residence and either share one main meal a day or share living accommodation (or both).'

The latest data can be found here [7]

Families[edit]

The Labour Force Survey defines family by the following criteria:

  • Married Couple family
  • Civil partner Couple family
  • Opposite sex cohabiting couple family
  • Same sex cohabiting couple family
  • Lone parent family

Against which criteria respondents are asked to specify if they have One child, Two children and Three or more children.

Households[edit]

Are defined by single residences and number of occupants as follows:

  • One person
  • Two people
  • Three people
  • Four people
  • Five people
  • Six or more people

Tools[edit]

Online survey tools run by third parties may have different privacy policies, and these policies may get in conflict with Wikimedia users privacy expectations. The initial list of tools was defined at Phab:T94807.

Recommended services[edit]

Not recommended but allowed[edit]

  • Google Forms is sometimes used for events feedback and by the community because it's very easy to set up, sometimes widely advertised e.g. id.wikipedia.sitenotice. Google's privacy policy is not consistent with Wikimedia's since they collect much more information from users. That being said, if we clearly indicate to users that the survey they are taking will be on Google and thus subject to the Google privacy policy (preferably before they click on the link to the survey), you can use it to administer the services.

Not evaluated[edit]

  • LimeService seems rather good but should be verified. They don't need Safe Harbor because they already are in EU (with Canada option).
  • SurveyMonkey has some privacy policy, as well as certifications and EU branches for EU users plus special articles for Canada, Japan, Australia and Brasil users.

References[edit]