Teacher page: designing data collection

The problem with this sub-topic

There is nothing to practise until a student notices it is a skill. They will read the page, agree with all of it, and then write "it is a bad question" in the exam and score nothing. The entire lesson is about turning agreement into named faults, so make them say the words out loud this lesson rather than nodding at them.

The one thing to do with the targets

Cover the captions and have them sort the four pictures themselves. Then ask which one they would least like to be using without knowing it. The answer is the top right: tight grouping, wrong place. A broken scale reading 2 kg heavy is perfectly reliable, and its consistency is precisely what makes it convincing.

Follow with: how would you discover it? They will suggest weighing again, which gives the same answer and proves nothing. Repeating a measurement tests reliability and can never test validity. That exchange is worth more than the definitions.

Answers

Question	Answer
1. The scale 2 kg heavy	B. Reliable but not valid.
2. "Clear and an improvement?"	B. Double-barrelled.
3. Rewriting "regularly"	C. How many of the last 7 days, at least 30 minutes.

What each wrong answer tells you

They choose	What it means
"Reliable and valid" (Q1)	They are treating consistency as correctness, which is the exact confusion the targets exist to break. Go back to the picture.
"Valid not reliable" (Q1)	They have the two words the wrong way round. A vocabulary problem rather than a thinking one, and quick to fix.
"It is leading" (Q2)	Defensible, since "improvement" does lean. Worth crediting the observation and then pushing: ask them to answer it as someone who found it clear but worse. They cannot, and that is a harder failure than a lean.
"Do you exercise often?" (Q3)	They have swapped one undefined word for another. Ask what "often" means to them, then to the person next to them.
"Rarely, sometimes, often" (Q3)	Closer, and the most interesting wrong answer. Fixed options feel rigorous. Ask two students to define "sometimes" in days per week and let the class hear the gap.

Where the marks go

Name the fault. Leading, loaded, double-barrelled, vague, unbalanced options. "It is a bad question" is worth nothing however correct it is, and it is what most of them write.

Improve means rewrite. Give the question in full with its answer options. Describing an improvement is not making one.

Reliability and validity are checked. If they remember one sentence, make it: repeating a measurement tests reliability and tells you nothing about validity.

A possible order

	What is happening
1	The four targets, sorted by them with the captions covered, then the broken scale discussion.
2	The five faults. Do not just read them: put a bad question on the board and have the class name the fault before you say it.
3	Rewriting. Give them three bad questions and have them produce the replacements in full, including options. This is the part that earns marks.
4	Categories and expected frequencies above 5, and degrees of freedom dropping when a parameter is estimated. Link forward to the chi-squared lessons.
5	The three questions, then the standing instruction: in the exam, name the fault.

Two things not to say

Do not use reliable and valid loosely in ordinary speech this week, including about anything unrelated to statistics. They are listening, and the distinction is fragile for about a fortnight after it is taught.

Do not accept "it is biased" as an answer here. Bias was 4.1 and it is about who you ask. This sub-topic is about what you ask and how you ask it, and conflating the two loses marks in both.

Questions to set

Three tiers, ramping the way practice should: the method on its own, then the method inside something real, then a challenge. Set the tier the class in front of you needs rather than one undifferentiated sheet. Answers are given so these can go straight onto a board.

1Fluency

The method on its own, with friendly numbers. Set these first and move on quickly once they are secure.

Name the faultA questionnaire asks: 'Was the new timetable clear and an improvement?' State the fault.
It is a double question, so a single answer cannot be interpreted. It is also leading, since 'improvement' invites agreement.
Reliability or validityA bathroom scale reads 3 kg heavy every time. Which does it lack?
Validity. It is perfectly reliable, giving the same reading every time, and consistently wrong.

2In context

The same skill inside a real situation, where the first job is working out what is being asked.

Rewrite a questionRewrite the timetable question so it can be analysed.
Two questions: 'How clear was the new timetable?' and 'Compared with last year, is the new timetable better, the same or worse?', each on a stated scale.
Categories before the testWhy must categories for a χ² test be chosen before looking at the data?
Choosing boundaries after seeing the values lets you select the split that gives significance, which invalidates the probability the test reports. The 5% is only 5% if the test was fixed in advance.

3Challenge

Reasoning, working backwards, or spotting an error. These are where the top grades are decided.

Reliability without validityGive an example from school data of a measure that is reliable but not valid, and explain.
Counting how many times a student puts a hand up, as a measure of understanding. Two observers would agree closely, so it is reliable, but it mostly measures confidence, so it is not valid for what it claims.
Design a checkYou suspect students answer a sensitive survey question dishonestly. Suggest a design that reduces that, and state its cost.
Make responses anonymous and avoid any question that identifies the student. The cost is that you can no longer link answers to other data about the same person, so some analysis becomes impossible.

Practicalities

Works on a phone. No interactive to break: the targets are plain drawings, so this page is also the one to print if the wifi fails. Nothing a student types is saved or sent anywhere.