How To Do Research

How To Do Research
The first thing to say is that if it has to be done, it's you who's got to do it.
•
You. There's no one else to do the project for you. Very likely, there's no one who will give you
any help. You're the one who cares, the drive has got to come from within you. You're the one
who knows how to answer the question, no one else knows as much.
The above is not the whole truth - as your project evolves, you'll find yourself
receiving technical help, wisdom, and emotional support from a great many
people - but you need to be aware that self-reliance is vital.
There may be other types of research, e.g., in which a neophyte is fitted into a slot in the God
Professor's program. I don't have experience of them.
•
Do. It is important that you start the main activities of the research - experimentation, or survey, or
simulation, or data analysis - sooner rather than later. A common mistake is to think and plan and
read, and never get round to the main game. When you start, you may find that the real difficulties
and issues are quite different from what you thought in the abstract.
Research has several components, for example: thinking of the question, answering the question,
communicating the answer. It may be worth mentioning that not all of these are essential.
•
•
•
Even without much of a question, one can collect some data, summarize it, and report the results.
Certainly if you're the first to think of this kind of data, that's all you need to do. Certainly if the
data keeps changing from year to year (e.g., accidents, and many other society-level phenomena),
this type of study is always useful to a greater or lesser extent.
Even without an answer, the question is sometimes such a good one that it is worth telling others
about. (It might be that you don't have time to answer it, or are uninterested, or not competent in
the necessary techniques.)
Sometimes you don't want to communicate your question and answer. They might have turned out
to be less interesting than you thought when you started. Or you might want to wait until you've
done something additional.
Nevertheless, most research has all three components. The answering and the communicating can be
dismissed quite briefly.
•
•
The fact that you're doing research is strong evidence that you have the skills necessary to answer
the question. Not all of them, perhaps, but enough to work out a way of attacking the research and
to serve as a foundation for additional skills to be learnt along the way. Incidentally, as a rule of
thumb, if there is a course on something, people think of it as a low-level skill - it is not
necessarily below your dignity as a researcher, but you don't get any merit badges for mastering it.
(I am a little concerned at the increasing amount of structured studies that universities are
prescribing for their first-year research students. To my mind, the overwhelming priority at that
stage is to do a doable piece of research, not attend lectures on bibliographic tools and the
institution's policy on intellectual property.) As to non-technical aspects of answering the question,
there is some advice below.
There are lots of textbooks and courses to aid you in your scientific writing. Essentially, though, it
is all a matter of practice.
One of the reasons I recommend psychology to students is that you can't escape
either the use of numbers or the use of words. (Economics is another good
subject from this point of view, but it doesn't have much of an experimental
component.)
As soon as you've got a message to communicate, you should write a report on it, not wait until
the whole project is finished. That probably means writing 1000 words per week, every week. Just
1
occasionally, supervisors forget to tell new writers the obvious - the easiest sections to write are
the descriptions of the results, and of what you did (because these sections are highly constrained
by the facts), and you should write these first; what the results mean, and why you did what you
did, are altogether more problematic, so the Discussion and Introduction sections should be
written later. If you're desperate, imagine you're speaking to a friend about your work, and write
down what you would say. (Then, as always, revise and edit, revise and edit, revise and edit.)
The important issue is that of what to study. If you have ideas, there's no problem. But, is there anything
you can do to prompt ideas to come to you? Or, in the absence of ideas, how can you do something
worthwhile? I don't suppose there's a complete answer to these questions, but the following comments are
intended to be helpful.
To get ideas, expose yourself to them.
• Talk to someone about your research. Your mum or dad, for example. Or (though I've never tried
this) the top stream of 13-year-olds at your local selective high school. Brainstorming with your
colleagues is better than nothing, but you're not really forced to decode your talk sufficiently.
Anyway, there are other uses for your colleagues, such as being rude to you.
It is tremendously valuable if there's someone in your circle who can bring off
the trick of being rude without causing offence. Suppose you are a cognitive
psychologist, and in a seminar you claim that “activation flows through the
levels” of your model. You benefit if someone from another tradition confronts
you with the claim that “activation”, “flows”, and “levels” are all meaningless. It
is generally up to the young to do this.
Informal seminars, where you listen to others, and talk yourself, are an important part of this.
•
Read widely. (But remember what I said earlier: reading is not what research is about. Indeed, I
would say there is a type of personality for whom too much reading is a major danger to their ever
doing anything worthwhile.)
• How to read: read the title, skim the abstract, look at the pictures and maybe the tables,
and if there's anything interesting, then consult the text, looking for that specific point.
(No-one starts at the beginning of a paper and works their way through to the end.)
• What to read: many of the “prestigious” journals are best avoided, as they have got into
the habit of attending to a lot of details that everyone knows are unimportant, thus
diluting the real message. The papers are polished and bland, and the reader is sucked
into taking them on their own terms. You need something rougher, that you can get to
grips with.
• Comments and letters in any journal. Whatever the point is, it is made quickly.
Furthermore, an area of controversy may be highlighted for you.
• Journals from outside your main discipline that are relevant to your topic. For
example, if your own discipline is psychology, see what the management,
sociology, medicine, and engineering journals are saying about your topic. They
may very well take sufficiently different an attitude to provoke thought. And
interdisciplinary journals, e.g., on the borders of music and psychology, or law
and statistics.
• For the same reason, journals from countries that Americans have never heard of
(such as Germany, India, and Japan).
• Low prestige journals sometimes publish simple data on unusual questions - if
the topic attracts you, ideas for improving the method or generalizing the results
will soon come.
• For the same reason, you can often find mental stimulation in journals from 60
or 80 years ago.
• Your attitude: sympathetic criticism. So much junk is published that a degree of skeptical
hostility is appropriate when approaching most papers (unless the author is on the
Approved List). But temper this with sympathy: a paper may be useless to you, and it
2
may even be clear that by any reasonable standard the research was a waste of time, but
yet the method may be useful to one reader, a detail in the results may be just what a
second reader is looking for, and a third reader may find her thoughts clarified by a point
in the discussion.
If someone says your paper is trash, retort that anyone who
writes a decent paper can get it published, but it takes real
talent to get rubbish into print.
•
•
•
•
•
•
•
Forget harsh realities for a moment, and think seriously about what question really interests you,
gets you passionate. Even though you're trying to keep your mind on the ideal, you will find
practicalities obtrude themselves. Some of these practical problems may turn into research
projects.
Look to a leisure interest of yours for inspiration. Many students are interested in at least one of
music, sport, and politics; well, if their academic subject is psychology, surely this has some
interesting things to say about all of these? The same goes for many other academic subjects.
Many of us are lucky enough to have easy access to computerized databases. Try searching them
with an unusual combination of words, e.g., statistics and music, or laughter and Wagner, or
measurement and theology.
Ask successful researchers in your department whether they have any particular methods for
getting ideas to come.
I find making lists, and then organizing them, often useful. Lists of what? Of questions, possible
experiments, possible surveys, sources of data, ways of operationalising a concept, ways in which
your supervisor could be more helpful,...
Draw an analogy. (Theoretical, or method of analysis.)
Thought experiments. If the results were to be ..., then we would conclude .... On the other hand, if
they were to be ..., our conclusion would be ....
Suggestions for what to do if you don't have any ideas.
• Criticize a paper by someone else.
• Can I improve the experiment?
• Can I improve the theory? In statistical modeling, one might replace a discrete variable
with a continuous variable, or vice versa - e.g., if “blue” and “white” types have been
proposed, replace them with a trait (“blueness”).
• Can I refine the definition?
• Can I broaden the circumstances in which the effect occurs?
• Can I simulate this on the computer?
• Can I improve the statistical analysis? (“Two of a trade never agree”, it is said, and
certainly no two statisticians have agreed with each other this century.) Improvement
does not necessarily mean additional complication. Sometimes, simple descriptive results
are overlooked in the rush to perform a complicated hypothesis test. If nothing better
comes to mind, try confidence intervals in place of hypothesis tests, or nonparametric
methods in place of parametric, or Bayesian weight of evidence in place of a p-value.
If you do comment on a published paper, you're actually paying it quite a high compliment. (If you
come across 24-carat dross, you should follow the convention of politely ignoring it.)
• You can always write a paper on:
• The usability of university Calendars.
• The comprehensibility of ergonomics textbooks.
• The rise and fall of the use of particular cliches in the titles of learned articles.
• Do readers want more raw data in journal articles?
• Bayesian weight of evidence in psychology journals, 1970-1996.
• How first-year students view research participation.
• Comparison of several fields in respect of their tolerance of low survey response rates.
• Why do students hate statistics?
Get the idea?
3
•
•
•
Think of an experiment that no one else has done, and do it. (The emphasis here is on doing
something because it can be done, not because you actually understand the implications.)
The different packaging of something unoriginal.
Go to a journal that is less quantitative than you are, and teach the readers something (be tactful,
now).
General advice.
• If you have a rigid deadline to work to, it is vital that you are not reliant on someone else for
anything important. Among the possible areas of difficulty are: administrative approval from an
outside body, construction of equipment, computing. You must have a workable plan that can be
implemented if approval is refused/the equipment isn't delivered/the computing expertise is
unavailable.
• Most people beginning research underestimate the importance of the physical aspects - going
through 500 files a second time because you didn't extract a particular item of information the first
time, coming in to the lab on Sunday afternoon to try out an idea you've just had, driving at 4am to
another city because the evening news told you of a “natural experiment” there that day, taking 60
journal volumes off the shelf to skim through for the key paragraph you wish you had made a note
of, and so on.
• Write down an idea when you have it, because you may easily have forgotten it by the time you
get to the office. Some people even keep an ideas book for this purpose; but the trouble with this is
that it really need to be physically very small, so that it can be always with you.
• Work around the problem, if you can't solve it.
• Sometimes a problem is best handled by defining it away.
• There is some research that simply isn't worth doing. It may be obvious to the drover's dog
roughly what the problem is... and that the institutional barriers to solving it are insurmountable.
Surveys, especially, can be a substitute for actually doing anything about the problem.
• Importance of flexibility.
• Luck. Have several projects going - diversification - they won't all be disasters.
• Don't be afraid to admit your ignorance. The nature of academic research is that one is continually
venturing into areas where one is an ignoramus. It is unpleasant to admit this. But the earlier you
do, the sooner you get an explanation and the quicker you can put that little bit of extra learning to
use.
• There can be a danger in overmuch personal involvement. Some people love people with
Alzheimer's disease, or autism, or depression, and want to research the condition. For a beginning
researcher, the problem of separating scientific evidence about the mass of people with the
condition from the loved one's personal experience is one problem too many - it is better to select
another subject for research.
• Perfectionism and attention to detail is a virtue. Taken to excess, it is a vice.
• Some researchers seem to suffer from a “fear of success”, which I think is different from obsessive
perfectionism. I think what happens is that they perceive the successful completion of their
research as the end of one stage in their life and the beginning of another, and they are reluctant to
move on to the new stage.
• Universities have become very bureaucratic about documenting the progress of research students.
One can appreciate the reasons for this, and yet there is a sense in which it is all a waste of time if the student is successful, there's no need to have a file of routine documentation; if he or she
isn't, no amount of paperwork will rescue the situation.
The supervisor may feel the student is wasting his or her time, and yet be
reluctant to say this (or say it forcefully enough) because it may actually be
wrong and because in any case it will be discouraging. The supervisor may
proffer clear advice, and yet the student be stubborn. If I had an answer, I'd give
it here...
•
All sorts of stuff has been written about the role of the supervisor. Much of it is sensible. But it's
all a long-winded way of saying that it is reasonable for the student to expect the supervisor to
4
•
•
•
listen to him or her for something like 20-50 hours per year. Anything more than that is taking a
responsibility away from where it properly belongs (i.e., with the student).
Don't miss any deadlines that have been agreed with your supervisor. If you do, then (depending
on the nature of your excuse) you'll go on to either the list of people who probably won't make it,
or the list of people who probably won't make it but who deserve sympathy. You do not want to be
on either of these lists. And don't be late for meetings with your supervisor.
Don't expect much praise from your supervisor. If your supervisor thought about it, he or she
would realize that a few words of praise were both justified (by the brilliance of your work) and
desirable (for the benefits of positive reinforcement). Sad to say, he or she doesn't think about it:
from about your second week of research, you've been accepted as a grown-up and been judged by
the same standards as the professors.
You might find it worthwhile reading How to get a PhD. Managing the peaks and troughs of
research by E. M. Phillips and D. S. Pugh (Milton Keynes: Open University Press, 1987).
Here are three points that Phillips and Pugh make, all of which I mildly disagree
with. (i) Intelligence gathering (i.e., description of the facts) is not research. (ii)
Research of the testing-out type (as contrasted with exploratory research and
problem-solving research) is much the most appropriate for a PhD student. (iii)
It is necessary for your thesis to have a “thesis”, in the sense of a connected
argument and message.
It may be that Phillips and Pugh are substantially correct in the context of
relatively mature fields of study; and that my opinion that (i) intelligencegathering may be research, (ii) exploratory research is useful and appropriate,
and (iii) the research may be too messy to have much of a “thesis” to it, is
formed by my experience being mostly in relatively immature fields of study.
•
•
•
•
Taste. Is it permissible to use phrases like “glass ceiling” or “evidence-based medicine” or “gold
standard” in reporting your research? Fastidious writers avoid cliches most of the time, but not
every use of them should be condemned. (In some allegedly scholarly fields of study, they seem to
be compulsory.)
Don't worry about the possibility of someone else duplicating your research. It is rare for this to
happen, and, when it does, it is stimulating to examine the similarities and the differences.
Don't despair if it becomes necessary to tear up two years' work. This is quite common, and often
means you are so expert and can see things so clearly that you can finish the whole project in only
six more months.
If you do happen to discover something important, be sure to do it at the right time and place.
Perhaps you remember the inventor of the Infinite Improbability Drive: “just after he was awarded
the Galactic Institute's Prize for Extreme Cleverness he got lynched by a rampaging mob of
respectable physicists who had finally realized that the one thing they really couldn't stand was a
smartass” (The Hitch Hiker's Guide to the Galaxy, Chapter 10).
Applying to do post-graduateresearch.
There is quite a lot of advice available on the WWW about how to choose which departments to apply to
(for a research degree), and how to increase your chances of success. I'm a bit skeptical about the need for
this - I think that by the end of their undergraduate careers, most students know in what sub-field of the
subject they want to work, or what style of approach they want to use. Furthermore, they know which
departments at which universities study that sub-field or take that style of approach. Consequently, their list
of desirable departments is already a short one; if they have made personal contacts, the list will be very
short indeed. However, if you think you need advice, here is mine.
• You should appreciate that there are an enormous number of specializations’s available in the
scholarly world. Flick through one of the directories of scholarly societies if you don't believe me.
For example, there is a Colour Society of Australia, an International Association for CrossCultural Psychology, and a Stress and Anxiety Research Society. If some specialization appeals to
you (surely you'll like something in all your undergraduate studies!), then sometime about the
middle of your undergraduate career, you should stop thinking of yourself as (for example) an
5
embryo psychologist, and start thinking of yourself as (for example) an embryo colour
psychologist. You will then only be interested in departments that are strong in colour studies.
If you are in this position, do not limit yourself to departments of psychology.
You might find the most suitable place is in a group within a chemistry or
physiology department.
•
•
Make personal contacts. If colour psychology really does fascinate you, make contact with people
in the field. Academics are enthusiasts for their own little patch, and will be delighted to find
someone who shares that enthusiasm. If possible, get some lowly job in their department during
the vacation. But be warned - I would say it is impossible to successfully fake this. Alleged
enthusiasm without an appropriate level of specialist knowledge and native intelligence will get
you nowhere.
If you really are unsure - if you just feel you'd like to do research in “psychology” or “chemistry” you may do better to delay your entry into research until your ideas have matured a little further.
(You cannot be expected to have a fully-thought-out research project at this stage; what I mean is
that you should know (for example) that your interest is in cognitive psychology studied by
computer-controlled experiments on normal humans.)
A little advice of a statistical or technical nature.
This is not the real theme of the present document, but a few comments of this type may be worthwhile.
You might also like to read “Pitfalls of data analysis” by Helberg.
• There is often tension between the respective advantages of standardizing on one particular
method of answering a question, and of using a variety of different methods. Think of measuring
unemployment, or television viewing, for instance. On the one hand, one wants a standard,
reproducible, method - comparisons from month to month are important, and one has reason to
think that whatever defects are present remain pretty constant over time. On the other hand, any
single practicable method is bound to have peculiarities and biases, and the “real, true” answer
should probably be synthesized from several different methods, even if each of them makes use of
only a small sample size.
When controversies break out, one factor is often the intrusion of a new player
into the system: the existing interests have learned to co-exist with each other
and with the defects of a question-answering method, but this doesn't suit
someone new.
The following is very much a broad-brush statement, to which there must be numerous exceptions:
too often, a standard method is adopted too early in the development of a subject. In research, one
usually wants the “real, true” answer, whereas for administration, one may prefer a standardized,
reproducible, answer.
“Figures often beguile me, particularly when I have the arranging of them
myself” (Mark Twain). Presumably sociologists have studied the use of statistics
in the wielding of power?
•
•
•
Meta-analysis: this term refers to quantitatively combining the results of previous studies of a
subject, in order to arrive at the right answer. I'm not terribly keen on the idea, but this may be
because I have not worked in fields where there have been lots of previous studies of the same
question. There are some difficult principles and practicalities involved, and inadequate handling
of these led critics to refer to meta-analysis as being “meta-silliness”. I think there have been
substantial improvements in the methodology, and these were winning me over, but then I read a
paper by Sohn (Clinical Psychology Review, 16, 1996, 147-156), which argues that publication
bias is so serious as to vitiate all conclusions from literature reviews in the field he examines,
effectiveness of psychotherapy (and I think the implication of the paper is that publication bias has
a more serious impact on a meta-analysis than on a review in traditional narrative format).
Contrast between exploring data, and using data for hypothesis testing.
Maybe you want to do both. One easy technical solution is to randomly split your data into two
portions, explore one half and generate hypotheses from it, and then test the hypotheses using the
other half.
6
•
Importance of a random sample, or random allocation. I have the feeling that often it is foolish to
prefer a rubbishy sample of 200 to a random sample of 20, yet often this is done.
It might be possible to argue that a rubbishy sample of 200 is the better for
hypothesis generation, whereas a random sample of 20 is the better for
hypothesis testing.
•
•
My impression is that the subject which is leading the way as regards strictness of methodology is
medicine. Many people will be familiar with the basics. In order to measure the effect of
something, it is necessary to compare it with something else. That is, we have a treatment group
and a control group. Patients are assigned at random to the one group or the other. It is desirable
that neither the patient nor the doctor who evaluates the patient's condition know whether the
patient received the drug or not. (This is termed a double-blind experiment.) But did you know
this idea is being taken so far that Gotzsche (Controlled Clinical Trials, 17, 1996, 285-293)
seriously suggests that the data should be blinded during statistical analysis? That is, two reports
are written. The first assumes A to be the treatment and B to be the control, and the second is
based on the reverse assumption. Both are completed before the code is broken. This is to avoid
bias creeping in during the data analysis and writing.
Moderate-sized effects imply large studies. What may turn out to be the most important statistical
paper of 1996 was published in the Oxford Textbook of Medicine, of all places: the convincinglywritten article by Collins et al. (1996) will surely encourage tens of thousands of medical
professionals towards good statistical practice in their research. One of the messages is that there
may be a number of treatments that are only moderately beneficial, but yet would be very
worthwhile because the disease is common or the treatment is cheap or both; and which, because
the treatment vs. control difference is not great, require large sample sizes in order to establish
statistical significance of the difference. These sample sizes are so large that they imply
collaboration between many investigators. Collins et al. give clear and interesting examples of
both large single clinical trials (e.g., aspirin in treating heart attacks), and of meta-analyses of
many small trials (e.g., adjuvant therapy for early breast cancer treated surgically). The effect sizes
in these studies were large enough to be well worth having - a reduction in 35-day mortality from
11.8% to 9.4% among patients with acute myocardial infarction who were treated with aspirin,
and a reduction in 10-year mortality from 48% to 38% among women aged 50+ with stage II
breast cancer who had tamoxifen included in their treatment - yet were also small enough that
clinical trials of the usual sizes were too small to detect them.
It would not surprise me if there were to be a trend towards simple, large,
collaborative studies in many areas of scientific enquiry besides medicine. Now,
consider that very many students conduct some sort of small research project as
part of their studies. I suggest that if different students in different universities
together decided on a common topic and worked out a common methodology
(and stuck to it), then (i) the results would be of much greater value than results
from a single institution, and (ii) the experience of collaborative work would
itself be valuable training, if indeed we are entering an era when this will be
increasingly common.
Collins R., Peto R., Gray, R., Parish S. (1996). Large-scale randomized
evidence: Trials and overviews. In Weatherall D. J., Ledingham J. G. G.,
Warrell D. A. (Editors), Oxford Textbook of Medicine. 3rd Edition, pp. 21-32.
Oxford: Oxford University Press.
•
Don't get carried away with the statistical analysis. Tell the story of your research using the tools
from your first course in statistics, not your last - that is, using nothing more complicated than
well-chosen descriptive techniques plus the concept of standard error. As much as is practicable,
present your results in such a way that the reader can see that the conclusions from your more
complicated calculations are probably correct.
Suppose your research is experimental; and involves obtaining a measurement
(e.g., an accuracy score) from each subject in each of four conditions; that the
four conditions consist of each combination of two levels (e.g., big and small) of
7
one factor with two levels (e.g., light and dark) of another factor; that the
subjects are in two groups (e.g., males and females); and that interest centres on
whether the average interaction effect is different for males from what it is for
females. Such an experimental question would be recognised by some people as
being about the three way interaction in a three-factor experiment, one factor
being a between-subjects factor and two of the factors being within-subjects
factors. Such people would know that a concise way of analysing the data and
presenting the results would be via Analysis of Variance. I recommend: (a) Yes,
do that. (b) But also, present your results in a way that enables them to be
roughly checked. (Otherwise, your readers will presume you've got them
wrong.) In the example, the interaction can be calculated for each subject. It is
the difference between two differences, [(big, light) - (big, dark)] - [(small, light)
- (small, dark)]. So calculate this for each subject, and summarise the values for
the males by their mean and s.d., and the values for the females likewise. It will
then be clear how similar or different are the interactions for the two genders.
It might be worth going to quite some trouble to make sure you, your supervisor,
and your examiners agree on this one. Having gone to the trouble of really
understanding your data and then explaining it comprehensibly, it would be
vexing to come across an examiner who wants to see 20 pages of ANOVA
tables. Or vice versa.
•
Just occasionally, it may happen that careful attention to notation in writing an equation will
reveal that you really did not understand something you thought you did (or even that everyone
else did not understand it).
This can happen with any type of mathematical equation, not necessarily one
involving random variables. But, to give a simple example of this latter type,
there might be an averaging process somewhere, and one can sometimes get
confused about what domain it is that one is averaging over.
8