Designing data collection

This is the sub-topic with no calculations in it, which is why it is the one people revise last and lose marks on. Everything here is a decision you make before any data exists.

Reliable and valid are not the same word

Four attempts at hitting the same target. Reliable means the shots land together. Valid means they land where you aimed.

The top right one is the trap. A measurement that is reliable but not valid gives you the same answer every time, which feels like evidence it is correct. A broken scale reading two kilograms heavy is perfectly reliable. Repeating a measurement tests reliability and can never test validity.

Testing reliability. Measure twice and compare. Give the same survey to the same people later (test-retest), or give two versions written to be equivalent (parallel forms).

Testing validity. Ask whether the thing measures what it claims. Does it cover the whole of what it says it covers (content), and does it agree with a measure already trusted (criterion-related)?

Questions that answer themselves

Leading. "Do you agree that the canteen should serve healthier food?" You have told them the answer you want.
Loaded. Contains an assumption. "How much time do you waste on your phone?" assumes it is wasted.
Double-barrelled. Two questions in one. "Was the lesson clear and interesting?" cannot be answered by someone who found it clear and dull.
Vague. "Do you exercise regularly?" Regularly means different things to different people, so the answers are not comparable.
Unbalanced options. Offering excellent, good, fair with no negative option forces the result upwards.

A good test: could somebody read your question and tell which answer you were hoping for? If so, rewrite it. The fix is almost always to ask for a fact instead of an opinion. Rather than "do you exercise regularly", ask "on how many of the last seven days did you exercise for at least 30 minutes?"

Choosing categories before you test

If the data is going to a chi-squared test, the grouping is a decision you make, and it is examinable.

Expected frequencies should be above 5. If a category is too thin, combine it with a neighbour. Doing so after seeing the results is something you must declare, because choosing categories to suit the answer is the statistical equivalent of moving the goalposts.

Degrees of freedom drop when you estimate. Every parameter you estimate from the data itself, such as using the sample mean as the model's mean, costs one more degree of freedom. The data cannot both choose the model and independently confirm it.

Your turn

1. A scale always reads 2 kg heavier than the true mass. How would you describe it?

Reliable and valid Reliable but not valid Valid but not reliable Neither reliable nor valid

2. "Was the new timetable clear and an improvement?" What is the main fault?

It is leading. It is double-barrelled: someone who found it clear but worse cannot answer. It is vague. There is nothing wrong with it.

3. Rewrite "do you exercise regularly?" so the answers can be compared. Which is best?

Do you exercise often? Do you agree that regular exercise is important? On how many of the last 7 days did you exercise for at least 30 minutes? How regularly do you exercise: rarely, sometimes, often?

Where the marks go

When asked to criticise a question, name the fault. Leading, loaded, double-barrelled, vague, unbalanced. "It is a bad question" earns nothing however correct it is.

When asked to improve one, give the rewritten question in full, with its answer options if it has them. Describing how you would improve it is not the same as doing it.

Reliable and valid are not interchangeable and examiners check. If you can only remember one thing, remember that repeating a measurement tests reliability and tells you nothing about validity.

Want a verdict on your own draft?

These pages are free and stay free, but they are general and your IA is not. Send me your research question, or whatever exists so far, and I will tell you in writing whether the topic has a ceiling on it, where the marks are going, and what to change first. That costs nothing and it comes back within 24 hours.

Written by a serving IB Diploma and Career-related Programme Coordinator and Head of Mathematics, who reads internal assessments across every subject group every year. If you then want the whole draft reviewed properly against all five criteria, that is the paid one, and it is refunded if it does not name at least three specific things to fix.

Get a free verdict Full written review, $99

I never write any part of it. Not a sentence, not a calculation, not your data. Under 18: a parent buys this and the thread is with them. I do not work with students at my own school.