Jump to content

Learning and Evaluation/Archive/Learning modules/3Construct Validity

From Meta, a Wikimedia project coordination wiki

Jump to page

Part 1: Introduction

Why Survey?
Why Surveys Are Useful
Survey instruments
Types of information
Attributes - a special case
Survey Objective and Planning

Part 2: Reliability & Validity

Reliability & Validity
Face Validity
Content Validity
Criterion Validity
Construct Validity

Part 3: Question Construction

Writing Good Questions
Questions from Existing Surveys
Constructing your own Questions
Be Specific
Be Concise
Avoid Double Negatives
Minimize Social Desirability Bias
Avoid Double-barreled questions
Avoid abbreviations, jargon, technical terms, or slang
Avoid leading questions
Avoid loaded questions
Use appropriate wording
Ask useful questions
Rely on second-hand data sparsely
Use caution when asking personal questions

Part 4: Response Options

Question types
Dichotomous pairs
Multiple choice
Check all that apply
Choosing response options

Part 5: Questionnaire structure

Important considerations
Questions order
Additional Resources

  Wikimedia Training Designing Effective Questions Menu

Construct Validity

Does a question measure what it is really supposed to measure?

For example, does an intelligence test really measure intelligence?

Objective: A program leader wants to know what the quality level is of their images from Wiki Loves Monuments that just ended last month.

Very poor contruct validity: Number of images uploaded to Commons as a measure of quality for WLM implementation.
The number of images is not an accurate measure of photo quality; many of the photos might be duplicates or have poor quality. Thus this is not an accurate measure of quality.

Poor construct validity: Measuring the quality rating one week after the event to determine if the photos quality.
The process of reviewing all photos and nominating those of the highest quality for the quality photo rating takes much longer than one week after the event. Thus, this measure is not accurately measuring the number of photos uploaded.

Better construct validity: Measuring the quality rating three months after the event to determine the proportion of photos with a Quality rating.
This third option has better construct validity because obtaining the Quality rating involves not only the label but the nomination process, which takes a lot of time. Three months after the event is a more reasonable time frame for counting the number of Quality photos.