Bonus Appendix C: Conducting a Study
Copyright 2000-2007 Mark L. Mitchell & Janina M.Jolley. All rights reserved.
You have reviewed the literature, developed a hypothesis,operationalized your variables, and given sound reasons for testing yourhypothesis. However, your preliminary work is still not done. You must nowdecide exactly what specific actions you will take. In other words, althoughyou probably have decided on the general design (such as a simple experiment ora 2 X 2), your plan is not complete until each detail has been thought throughand written down.
You shouldwrite down exactlywhat procedures you will follow. For example, what instructions, word for word,will you give participants? Where will you test them? Will participants be runin groups or individually? In answering these questions, you must take intoaccount issues of validity. However, your paramount concern must always beethics. You do not have the right to harm another.
EthicalConsiderations: Human Research
Ethics should be the foundation of your research plan. Therefore, youshould read Appendix A (Ethics) before conducting a study.
In additionto reading Appendix A, you must be extremely careful not to harm yourparticipants. Ideally, your participants should feel just as well when theyleave the study as they did when they began the study. Unfortunately, even inthe most innocuous studies, protecting your participants from discomfort ismuch easier said than done.
Weighing the Risks
Realize that any experience may be traumatic to someparticipants. Trauma can occur from things you would never think of as beingtraumatic. Because any study has risks and because you will not know all of therisks, do not run a single participant without your professor’s permission.
To begin tosensitize yourself to the risks involved in your proposed study, list the 10worst things that could possibly happen to participants. If you are using humanparticipants, be aware that not all participants will react in the same way.Some may experience trauma because the study triggers some painful memory. Someparticipants may feel bad because they think they did poorly. Otherparticipants may feel bad because they think their behavior ruined your study.Realize that some of your participants may be mentally unbalanced and anyattack on their self-esteem might lead to disastrous consequences. Becauseparticipants are often fragile, you should list some serious consequences inyour worst-case scenario.
Because any study has the potential for harm, the possibility of severeconsequences does not mean that your professor will not allow you to do thestudy. However, you and your professor should think about ways to minimize therisks.
One method of minimizing risks isto screen out “vulnerable participants.” For instance, if there isany reason to believe that your study may increase heart rate or bloodpressure, you may want to make sure that only people in good health participatein your study. If your study might harm people with low self-esteem, you maywant to use only well-adjusted participants who have high levels ofself-esteem. Therefore, you might give a measure of self-esteem to potentialparticipants to eliminate those with low self-esteem.
Not only should you screenparticipants, but you should also let participants screen themselves. That is,participants should be volunteers who give their informed consent:
Howinformed is informed consent? Very informed, when it comes to tellingparticipants about any unpleasant aspect of the study. If participants aregoing to get shocked or exposed to loud noises or extreme cold, they should beinformed of this before they volunteer. Consequently, if your study doesinvolve unpleasantness, you may have difficulty getting participants tovolunteer.
Informedconsent is considerably less informed when it comes to more innocuous aspectsof the study. After all, the study would be ruined if participants kneweverything that would happen (and why it happened) before it happened. So,although participants are usually told the truth, they are not always told thewhole truth. For example, a memory experiment’s description would mentionthat participants have to memorize words, but might omit the fact that theresearcher is looking at the order in which facts are recalled or that there isa surprise recall of all the lists at the end of the study.
Becauseparticipants are not fully informed about your study, there may be some thingsabout it that they dislike. For example, suppose a participant finds the tasktoo difﬁcult or finds it upsetting to try the surprise recall task. Whatcan you do?
Oneprotection against these unexpected problems is to make sure participantsunderstand that they can quit the study at any time. So, before theparticipants begin your study, tell them that if they find any aspect of thestudy uncomfortable, they can and should escape this discomfort by quitting the study.Assure them that it is their duty to quit if they experience discomfort andthat they will still get full credit.
Modifying the Study
You have seen that you can minimize ethical problems by lettingparticipants know what they are in for and by letting participants gracefullywithdraw from the study. You should also minimize harm by making your study ashumane as possible. You can make your study more ethical by reducing thestrength of your treatment manipulation, carefully selecting stimulusmaterials, and by being a conscientious researcher.
Reducing the Treatment Strength
Although using extreme levels ofyour predictor variable may help you get a significant change in the criterionvariable, extreme levels may harm your participants. For example, 24 hours offood deprivation is more likely to cause hunger than 12 hours. However, 24hours of deprivation is more stressful to the participant. If you plan anunpleasant manipulation, remember your participants’ welfare and minimizethe unpleasant consequences as much as possible. Consider using levels of thepredictor variable that are less severe than you originally intended.
Modifying Stimulus Materials
By modifying your stimulusmaterials, you may be able to prevent them from triggering unpleasant memories.For instance, if you were interested in the effects of caffeine on memory forprose, you would not want the prose passage to cover some topic like death,divorce, alcoholic parents, or rape. Instead, you would want to use a passagecovering a less traumatic topic such as sports. If the sports article referredto someone’s death or hospitalization, you might want to delete thatsection of the article.
The Conscientious Researcher
Often, it is not the study that causes ethical problems,
First, whenscheduling your research sessions, make sure you leave a 10-minute gap betweenthe end of one session and the beginning of the next session. Someinvestigators feel that, like a physician, they should efficiently schedulepeople one after another. Their attempt at efficiency results in participantshaving to wait for the investigator, the investigator having to rush throughthe formalities of greeting participants, or—even worse—theinvestigator rushing through debriefing. Thus, the overly efficient investigator,like the overly efficient physician, is seen as unconcerned. Although thisconduct does not become physicians, it is intolerable for psychologicalresearchers. After a research participant has given an hour of his or her time,you should be more than willing to answer any questions the participant has.Furthermore, if you rush through greeting or debriefing each participant, theparticipants will see you as uncaring. Consequently, participants will be less likelyto tell you about any psychological discomfort they felt and less likely toaccept any aid you might offer. Thus, the first step is to walk, rather than torun, participants through your study.
Second,give the participants power. That is, allow participants to rate your study ona scale such as the one in Table 1–1 (below). Give each participant’srating sheet to your instructor. Following this simple procedure helps you tobe a conscientious and courteous researcher.
SampleDebriefing Rating Scale
Being aparticipant in psychology studies should provide you with a firsthand look atresearch. On the scale below, please indicate how valuable or worthless youfound being in today’s study by circling a number from +3 to –3.
WORTHLESS:–3 –2 –1 +1 +2 +3 :VALUABLE
NOTE: Thisscale is a slightly modified version of a scale that has been used at The OhioState University.
Although you should try to anticipate and prevent every possible badreaction a participant may have to being in your study, you will fail. .Inevitably, your procedures will still cause some unpleasantness. After thestudy is over, you should try to remove this unpleasantness by informingparticipants about the study, reassuring them that their reactions were normal,and expressing your appreciation for their participation.
You shouldalso listen to participants and be sensitive to any unexpected, unpleasantreactions to your study. By being a good listener, you should be able to undoany damage you have unwittingly done. This process of informing yourparticipants about the study and removing any harm done is called debriefing.Occasionally,ordinary debriefing will not undo the harm caused to the research participant.In those cases, there are several steps you may take to alleviate distress. Forparticipants who are upset with their responses, you should ask them whetherthey want you to destroy their data. For participants that you cannot calmdown, you should take them to talk to a professor, counselor, orfriend—even if this means canceling a research session you had scheduled.
In summary,you should be very concerned about ethics. Since ethics involves weighing thecosts of the study against the potential benefits, you should do everything youcan to minimize the risk of participants becoming uncomfortable. If, despiteyour efforts, a participant experiences discomfort, you should try to reducethat discomfort during debriefing.
EthicalConsiderations: Animal Research
With animal participants, you incur the same responsibilities that youdid with human participants—you must protect animal participants fromundue stress and discomfort. In many ways, you have even more responsibility toanimal participants because they depend on you for their mere existence. Youmust keep them fed, clean, warm, and comfortable—24 hours a day. Tofulfill your responsibility to animal participants, you must follow APA’sguidelines for proper housing, food and water, and handling (see Appendix A).
Furthermore,because your animal subjects n have neither the power to give their informedconsent, nor the power to quit the study, you must carefully question the valueof your study. Ask yourself and your professor this question: “Is thepotential knowledge gained from the study worth the cost to the animals?”Finally, if you must euthanize (kill) your animal participants at the end ofyour study, follow APA’s guidelines to ensure that you euthanize youranimals in the most humane way.
Maximizethe Benefits: The Other Side of the Ethics Coin
We havediscussed ways of minimizing harm to participants. However, minimizing harm isnot enough to ensure that your study is ethical. For your study to be ethical,the potential benefits must be greater than the potential harm. Thus, anextremely harmless study can be unethical if the study has no potentialbenefits. In other words, just as you owe it to your participants to reducepotential harm, you owe it to your participants to maximize the potentialbenefits of your study. You maximize that potential by making sure your studyprovides accurate information. To provide accurate information, your studyneeds to have power and validity.
One of themost serious obstacles to obtaining accurate information is lack of power.Remember, null results do not prove the null hypothesis. They only make peoplewonder about the study’s power. There is no point in doing a study thatis so powerless that it will lead to inconclusive, null results.
To havepower, you should use a strong manipulation, a sensitive dependent measure,well-standardized procedures, a sensitive design, and enough participants.
SampleSize: There Is Power in Numbers
Perhapsyour most important obstacle to finding a significant effect is a lack ofparticipants. As a general rule, you should have at least 16 participants ineach group. However, thenumber of participants you need in each group will be affected by thesensitivity of your design, the heterogeneity of your participants, the numberof observations you get from each participant, the size of the difference youexpect to find between conditions, and the sensitivity of your dependentmeasure.
If you havea within-subjects design, a reliable and sensitive dependent variable, andexpect a rather large difference between your conditions, you may be able touse fewer than 16 participants per group. If, on the other hand, you are usinga simple, between-subjects design, heterogeneous participants, a manipulationthat may have little effect, and a relatively insensitive dependent measure,you may want at least 100 participants per condition.
Gettingenough participants to have enough power can be a challenge, especially if youare at a small school. However, even at a large school, getting enough studentsto volunteer to be in your study can be difficult.
Findingadult participants can be even more challenging than finding children. Forexample, one of your textbook’s authors wanted an adult population forher doctoral dissertation. Her first thought was to contact a major company andgain access to its employees. This tactic failed. Next, she tried to run anewspaper ad asking for volunteers. One newspaper refused to print it. Anotherwould only run it in the “Personal” section. Thus, her appeal forparticipants appeared with ads for astrological advice, massage services, andpeople wanting dates. Although a few “volunteers” called, mostwanted either a date or an obscene conversation. We do not recommend newspaperads—especially if your goal is to get a representative sample of theadult population.
OlderAdults The authors have had greater success recruiting elderly participants.Nutrition centers, retirement communities, friendship networks, and nursinghomes have been fruitful sources of participants. In addition, we recommend the“grandmother connection”: having an older relative or friendintroduce you to other prospective participants.
Obviously,finding human participants will take planning, perseverance, and luck. Once youcontact prospective participants, you should explain your study to them. Beforethey participate, they must sign a permission form. Signing the permissionform—called an informed consent form—indicates that the participantunderstands what the study is about and has voluntarily agreed to participate.If you properly design the form (by following the instructions on page 475 of Research designexplained andfollowing the model presented in Box A-2, page 476 of Research design explained
Reducing Threats to Construct Validity
After ensuring that your study has adequate power,we would like to be able to tell you that you can take it easy and relax.Unfortunately, however, you cannot relax. Power is not your only concern whenconducting psychological research. You must also ensure that the constructvalidity of your results is not destroyed by
If you use more than one investigator, you may be able todetect researcher effects by including the researcher as a factor in yourdesign. In other words, randomly assign participants to both a condition and toa researcher. For example, if you have two treatment conditions (
Using ANOVAto detect researcher effects can be useful. However, there are at least tworeasons why using ANOVA may not eliminate researcher effects. First, thisstatistical approach will only tell you if one researcher is getting differentresults than other researchers. If all your researchers are biased, you may notget a significant researcher effect. (Besides, if you are the only researcher,you cannot use researcher as a factor in an ANOVA.) Second, and moreimportantly, detecting researcher effects is not the same as preventingresearcher effects.
To preventresearcher effects, you must address the three major causes of researchersfailing to conduct studies in an objective and standardized manner. What arethese causes? First, researchers may not know how to behave because you havenot spelled out the exact procedures that the researchers should follow. Second,researchers may not follow those procedures. Third, the researchers maystrongly expect participants to behave in certain ways.
Loose-Protocol Effect: TheImportance of Developing a Protocol Often, the researchers are not behaving inan objective and standardized way because of the loose-protocol effect:
Before youstart your study, carefully plan everything out. As a first step, you shouldwrite out a set of instructions that chronicles the exact procedure for eachparticipant. These procedures should be so specific that by reading andfollowing your instructions, another person could run your participants thesame way you do.
To makeyour instructions specific, you might want to write a computer program based onthese instructions. Since computers do not assume anything, writing such aprogram forces you to spell out everything down to the last detail. If youcannot program, just write the script as if a robot were to administer thestudy. Write out each step, including the actual words that researchers willsay to the participants. The use of such a script will help standardize yourprocedures, thus reducing threats to validity.
Once youhave a detailed draft of your protocol, give it a test run. For example, toensure that you are as specific as you think you are, pretend to be aparticipant and have several different people run you through the study using only
At the endof your test runs, you should have a detailed set of instructions that you andany co-investigator can follow to the letter. To double-check your protocol,see Table 1–2 (below).
How willyou manipulate your treatment variables?
How willyou measure your dependent (criterion) variables?
How manyparticipants will you need?
Do you haveyour professor’s permission to conduct the study?
Do you havea suitable place to run your participants?
How willyou get your participants?
If you areusing animals, how will they be cared for?
What willyou do with your animals after the study?
If you areusing human participants, how will you make your sign-up sheets available topotential participants?
Have youincluded a description of the study (including how long it takes) on thesign-up sheet?
Willparticipants be rewarded for volunteering to be in your study (such as money orextra credit)?
If you areconducting an experiment, how will you assign participants to condition?
Have youwritten out a detailed research protocol?
If you areusing human participants, have you developed a consent form?
If you areusing human participants, have you written out the oral instructions you willgive your participants?
If you areusing human participants, have you written out what you will say duringdebriefing?
Will you informparticipants about the outcome of your study? How?
Inspiring the Troops to AvoidResearcher Effects.
Unfortunately, even if you writeout your protocol (procedures) in detail, you or yourco-investigators may still fail to follow that protocol. To avoid the
To makesure investigators learn the procedures, you should hold training sessions.Supervise investigators while they practice the procedures on each other and onpractice participants.
Onceresearchers know the right way to run the study, the key is to make sure thatthey are motivated to run the study the same way every time. To increaseresearchers’ motivation to be consistent, you might have them work inpairs. While one researcher runs the participants, the other will listen inthrough an intercom or watch through a one-way mirror. You may even wish torecord research sessions.
If yourresearchers still have trouble following procedures, you may need to automateyour study. For instance, you might use a computer to present instructions,administer the treatment, or collect the dependent measure. Computers have thereputation for following instructions to the letter, so using a computer mayhelp standardize your procedures. Of course, computers are not the onlymachines that can help you. Some of the machines that could help you giveinstructions and present stimuli include automated slide projectors, taperecorders, and videotape players. Countless other devices could help you recorddata accurately, from electronic timers and counters to noise-level meters.
Researcher-Expectancy Effect Thefinal source of researcher bias is the researcher-expectancy effect:
Forexample, if you present stimuli in booklets, you can design your booklets sothat booklets for different conditions look very similar. In that way, aninvestigator running a group of participants might not know what condition eachparticipant is in. For some studies, you may be able to use a secondinvestigator who does nothing except collect the dependent measure. You couldeasily keep this second investigator in the dark about what treatment theparticipant received.
Review ofResearcher Effects.
Unfortunately,in psychological research, you must be aware not only of researcher effects,but also of participant effects: Participants may see through the study and try to playalong with what the investigator wants. Fortunately, there are various ways ofpreventing participants’ expectancies from biasing your results.
PreventingParticipants’ Expectancies For starters, you might make your researcherblind to reduce the chance that the participant will get any ideas from theresearcher. Thus, the techniques for reducing research effects that we justdiscussed may also reduce the effects of participants’ expectancies.
Inaddition, you may also be able to prevent participants’ expectancies byskillfully choosing your research design. In experimental investigations, forexample, you might use a between-subject design rather than a within-subjectdesign because participants who are exposed to only one treatment condition areless likely to guess the hypothesis than participants who are exposed to alltreatment conditions.
Anotherdesign trick you can use to reduce the impact of participants’expectancies is to use placebo treatments. Placebo treatments preventparticipants from knowing that they are in the “no-treatment”condition. Therefore, if you have comparison condition(s), use placebotreatment(s) rather than no-treatment condition(s). That way, all groups thinkthey are receiving the treatment. Thus, any treatment effect you find will notbe due to participants changing their behavior because they expect thetreatment to have an effect.
Participantsare less likely to know the hypothesis if they do not know what you aremeasuring. Obviously, if, as in some field experiments, participants do noteven know you are observing them, participants will not know what you aremeasuring. Thus, if your hypothesis is an obvious one, you might consider afield study. Although field studies lend themselves to unobtrusive recording,unobtrusive recording can even occur in a laboratory study. That is,participants will assume that if you are not in the room with them, you are notobserving them. However, thanks to one-way mirrors and intercoms, you canmonitor participants’ behavior from the next room.
Rather thantrying to obscure or confuse participants as to the purpose of the study, youmight try to prevent participants from thinking about the purpose of the study.One way to stop participants from thinking about the study’s purpose isto design a study that has a high degree of experimental realism
Before now,you might have been surprised to see experimental realism and other strategiesfor reducing participant effects in a section on ethics. However, you now knowthat planning an ethical study involves taking into account many factors. Notonly must you ensure the safety of your participants, but you must alsodemonstrate the validity of your methods. To avoid overlooking an importantethical consideration, consult Table 1-2, Table 1-3 (below), Appendix A, andyour professor.
Is aphysically unpleasant stimulus going to be used in your study? If so,
1. Isthis fact clearly stated
2. Haveyou considered alternatives that would be less unpleasant?
3. Haveyou limited the intensity of this stimulus?
4. Haveyou taken steps to reduce potential harm to your participants caused by aphysically unpleasant stimulus?
Are you goingto use stress of some sort (such as sense of insecurity or failure, assaultupon values, fatigue, or sleep deprivation) in your study? If so,
1. Isthis fact clearly stated
2. Haveyou considered alternatives that would be less stressful?
3. Haveyou limited the intensity of this stimulus?
4. Haveyou taken steps to reduce potential harm to your participants caused by apsychologically unpleasant stimulus?
What willyou do if participants exhibit signs of harm (for instance, crying, disorientedbehavior)?
Are youprepared to describe the purpose and nature of your study to your participantsduring debriefing?
Will youuse deception in your study? If so, what will you tell participants duringdebriefing?
Are youaware that participants can quit your study at any time? If a participant doesdrop out, will you give your participants credit for participating? Is thisfact stated on the informed consent form? Is this fact part of yourinstructions to the participants?
Whateducational gain do you think participants will obtain from participating inyour study?
Beyondthe Proposal: The Pilot Study
Even afteryou have carefully designed your study, modified it based on comments from yourinstructor, and been given your professor’s go-ahead to run it, you maystill want to run several participants (friends, family members, other membersof the class) just for practice. By running practice participants, you will getsome of the “bugs” out of your study. Specifically, by running anddebriefing practice participants, you will discover
2. whether you canperform the study the same way every time or whether you need to spell out yourprocedures in more detail;
3. whether you areproviding the right amount of time for each of the research tasks and whetheryou are allowing enough time in between tasks;
4. whether yourinstructions were clear;
5. whether your coverstory was believable;
6. whether you need torevise your stimulus materials;
7. how participantslike the study; and
In short,running practice participants helps you to fine-tune your study. Becauserunning practice participants is so useful, many professional investigators runenough practice participants to constitute a small study—what researcherscall a pilot study.
Conductingthe Actual Study
The dressrehearsal is over. You have made the final changes in your procedures and yourproposal. Now you are ready for the real thing—you are ready to conductyour study. This section will show you how.
As you may imagine, some of your prospective participants may beapprehensive about the study. Participants often are not sure whether they arein the right place, or even whether the researcher is a Dr. Frankenstein.
To put yourparticipants at ease, let them know they are in the right place, and becourteous. You should be both friendly and businesslike. The expertinvestigator greets the participant warmly, pays close attention to theparticipant, and seems concerned that the participant knows what will happen inthe study. The expert investigator is obviously concerned that each participantis treated humanely and that the study is done professionally.
Beingprofessional does not hurt how participants view you. Why? First, mostparticipants like knowing that they are involved in something important.Second, some will view your professionalism as a way of showing that you valuetheir time—which you should.
So, how canyou exude a professional manner? Some novice investigators think that theyappear professional when they act aloof and unconcerned. Nothing could be lessprofessional. Participants are very turned off by a disinterested attitude. Theyfeel that you do not care about the study and that you do not care about them.
To appearprofessional, you should be neatly dressed, enthusiastic, well-organized, andprompt. “Prompt” may be an understatement. You should be ready andwaiting for your participants at least 10 minutes before the study is scheduledto begin. Once your participants arrive, concentrate exclusively on the job athand. Never ask a participant to wait a few minutes while you socialize withfriends.
What do youlose by being a “professional” investigator? Problem participants.If you seem enthusiastic and professional, your participants will also becomeinvolved in doing your study—even if the tasks are relatively boring.Thus, if you are professional in your manner and attitude, you will probablynot even have to ask the participants to refrain from chatting throughout thestudy. Similarly, if you are professional, participants will stop askingquestions about the study if you say, “I will explain the purpose at theend of the study.”
After youhave established rapport, you need to give your participants instructions. Toget participants to follow instructions to the letter, you might
Once thestudy has begun, try to follow the procedure to the letter. Consistentlyfollowing the same procedures improves power and reduces the possibility ofbias. Therefore, do not let participants change your behavior by reinforcing orpunishing you. For instance, imagine you are investigating long-term memory.
Once the study is over, you should debrief your participants. Indebrieﬁng, you should ﬁrst try to ﬁnd out whether theparticipants suspected the hypothesis. Simply ask participants what theythought the study was about. Then, explain the purpose of your study.
If youdeceived your participants, you need to make sure they are not upset about thedeception. You also need to make sure that they understand why deception wasnecessary. Participants should leave the study appreciating the fact that therewas one and only one reason you employed deception: It was the only way to getgood information about an important issue.
Making sureparticipants accept your rationale for deception is crucial for three reasons.First, you do not want your participants to feel humiliated or angry. Second,if they get mad, they may not only be mad at you, but also at psychologists ingeneral. Perhaps that anger or humiliation will stop them from visiting apsychologist when they need help. Third, the unhappy participant may spread theword about your deception, ruining your chances of deceiving otherparticipants.
Afterexplaining the purpose of the study, you should answer any questions theparticipants have. Although answering questions may sometimes seem like a wasteof time, you owe it to your participants. They gave you their time, now it isyour turn.
Afterparticipants’ questions and doubts have been dealt with, give them anopportunity to rate how valuable they felt the study was. Letting participantsrate your study encourages you to be courteous to your participants, ( lets youknow whether your study is more traumatic than you originally thought, and
Afterparticipants rate your study, you should assure participants that their responsesduring the study will be kept conﬁdential. Tell them that no one but youwill know their responses. Then, ask the participants not to talk about thestudy because it is still in progress. For example, you might ask them not totalk about the study until next week. Finally, you should thank yourparticipants, escort them back to the waiting area, and say goodbye.
You might think that once a participant leaves the study, yourresponsibilities to that participant end. Wrong! You are still responsible forguaranteeing the participant’s privacy. Knowledge about a givenparticipant is between you (the investigator) and the participant—noone else. Neverviolate this conﬁdentiality. To ensure conﬁdentiality, you shouldtake the following precautions:
5. Watchyour mouth. There is rarely a reason to talk casually about aparticipant’s behavior. Even if you do not mention any names, otherpeople may guess or think they have guessed the identity of your participant.We realize that it is hard to keep a secret. But to talk freely about someonewho participated in your study is to betray a trust. Furthermore, keepingsecrets will, for many of you, be an important part of your professional role:Therapists, researchers, consultants, lawyers, and physicians all must keeptheir clients’ behaviors confidential.