6 Questions to ask when evaluating a research claim

1. Is prediction testable like experimental hypothesis or is their prediction untestable because:

1. It is too vague

2. It is subjective rather than objective

3. They are trying to prove null hypothesis (It's hard to prove a negative, but easy to fail to prove a negative).

2. How consistently do the results occur? (if there are results!)

1. Beware of argument by example--the lesson of the case study & also the survey.

2. Look for statistical significance (our tool for ruling out coincidence),

but realize that statistical significance does not necessarily mean large or important

3. Ask about control groups (lesson from the experiment)

1. Is there one --or are they assuming they know what the control group would do?

Is it a group?

2. Is it fair?

a. Was it established by random assignment?

b. Is it treated like the treatment group in every respect except that it doesn't get the treatment?

c. Were double-blind techniques used? If not, the results could be due to a placebo effect.

4. Are they trying to insinuate that correlation means causation? Remember, A-->B is only one of several possibilities for a statistical relationship between A and B.

5. Are they reporting what they actually saw

or

are they reporting their interpretation of what they saw?

(The lesson of observation: objective observation is difficult)

6. Are their results based on a sample that would apply to you?

(The lesson of the survey: the sample matters)

---