Learning and Evaluation/Archive/Learning modules/3Criterion Validity

From Meta, a Wikimedia project coordination wiki
Jump to navigation Jump to search
Jump to page

Part 1: Introduction

Welcome!
Why Survey?
Why Surveys Are Useful
Constructs
Operationalize
Survey instruments
Types of information
Attributes - a special case
Survey Objective and Planning

Part 2: Reliability & Validity

Reliability & Validity
Reliability
Validity
Face Validity
Content Validity
Criterion Validity
Construct Validity

Part 3: Question Construction

Writing Good Questions
Questions from Existing Surveys
Constructing your own Questions
Be Specific
Be Concise
Avoid Double Negatives
Minimize Social Desirability Bias
Avoid Double-barreled questions
Avoid abbreviations, jargon, technical terms, or slang
Avoid leading questions
Avoid loaded questions
Use appropriate wording
Ask useful questions
Rely on second-hand data sparsely
Use caution when asking personal questions

Part 4: Response Options

Question types
Fill-in-the-blank
Dichotomous pairs
Multiple choice
Check all that apply
Ranking
Scales
Choosing response options

Part 5: Questionnaire structure

Important considerations
Questions order
Additional Resources
Feedback

  Wikimedia Training Designing Effective Questions Menu

Criterion Validity

Does the measure produce results that correspond with a superior one?


Does the measure correspond with another known/valid measure?


Objective: A program leader wants to know the quality of articles that students produce in a Wikipedia Education Program.


1. Very poor criterion validity: Program leader relies only on self-report of respondents.
When someone self-assesses the quality of their own work, they typically have a poor idea how they compare to others. This makes for very poor criterion validity since individuals can only compare the quality of their work to themselves.


2. Poor criterion validity: Program leader measures the number of good articles as the only measurement for quality.
Relying only on the number of good articles is not the best measure for quality. Different Wikimedia projects have different measures for "good articles."


3. Better criterion validity: Number of good articles and compares this to peer or teacher assessment of student's work.
The third option has better criterion validity because even if students edit Wikipedia in more than one language, the teacher or peer-review might help to normalize the differences across Wikimedia projects.