Why we need pre-registration

There has been a chorus of disapproval this week at the suggestion that researchers should 'pre-register' their studies with journals and spell out in advance the methods and analyses that they plan to do. Those who wish to follow the debate should look at this critique by Sophie Scott, with associated comments, and the responses to it collated here by Pete Etchells. They should also read the explanation of the pre-registration proposals and FAQ  by Chris Chambers - something that many participants in the debate appear not to have done.



Quite simply, pre-registration is designed to tackle two problems in scientific publishing:

• Bias against publication of null results


• A failure to distinguish hypothesis-generating (exploratory) from hypothesis-testing analyses

Either of these alone is bad for science: the combined effect of both of them is catastrophic, and has led to a situation where research is failing to do its job in terms of providing credible answers to scientific questions.

Null results

Let's start with the bias against null results. Much has been written about this, including by me. But the heavy guns in the argument have been wielded by Ben Goldacre, who has pointed out that, in the clinical trials field, if we only see the positive findings, then we get a completely distorted view of what works, and as a result, people may die. In my field of psychology, the stakes are not normally as high, but the fact remains that there can be massive distortion in our perception of evidence.



Pre-registration would fix this by guaranteeing publication of a paper regardless of how the results turn out. In fact, there is another, less bureaucratic, way the null result problem could be fixed, and that would be by having reviewers decide on a paper's publishability solely on the basis of the introduction and methods. But that would not fix the second problem.

Blurring the boundaries between exploratory and hypothesis-testing analyses

A big problem is that nearly all data analysis is presented as if it is hypothesis-testing when in fact much of it is exploratory.



In an exploratory analysis, you take a dataset and look at it flexibly to see what's there. Like many scientists, I love exploratory analyses, because you don't know what you will find, and it can be important and exciting. I suspect it is also something that you get better at as you get more experienced, and more able to see the possibilities in the numbers. But my love of exploratory analyses is coupled with a nervousness. With an exploratory analysis, whatever you find, you can never be sure it wasn't just a chance result. Perhaps I was lucky in having this brought home to me early in my career, when I had an alphabetically ordered list of stroke patients I was planning to study, and I happened to notice that those with names in the first half of the alphabet  had left hemisphere lesions and those with names in the second half had right hemisphere lesions. I even did a chi square test and found it was highly significant. Clearly this was nonsense, and just one of those spurious things that can turn up by chance.



These days it is easy to see how often meaningless 'significant' results occur by running analyses on simulated data - see this blogpost for instance. In my view, all statistics classes should include such exercises.



So you've done your exploratory analysis, got an exciting finding, but are nervous as to whether it is real. What do you do? The answer is you need a confirmatory study. In the field of genetics, failure to realise this led to several years of stasis, cogently described by Flint et al (2010). Genetics really highlights the problem, because of the huge numbers of possible analyses that can be conducted. What was quickly learned was that most exciting effects don't replicate. The bar has accordingly been set much higher, and most genetics journals won't consider publishing a genetic association unless replication has been demonstrated (Munafo & Flint, 2011). This is tough, but it has meant that we can now place confidence in genetics results. (It also has had a positive side-effect of encouraging more collaboration between research groups). Unfortunately, those outside the field of genetics are unaware of these developments, and we are seeing increasing numbers of genetic association studies being published in the neuroscience literature, with tiny samples and no replication.



The important point to grasp is that the meaning of a p-value is completely different if it emerges when testing an a priori prediction, compared with when it is found in the course of conducting numerous analyses of a dataset. Here, for instance, are outputs from 15 runs of a 4-way Anova on random data, as described here:

Each row shows p-value for outputs (main effects then interactions) for one run of 4-way Anova on new set of random data. For a slightly more legible version see here

If I approached a dataset specifically testing the hypothesis that there would be an interaction between group and task, then the chance of a p-value of .05 or less would be 1 in 20  (as can be confirmed by repeating the simulation thousands of times - in a small number of runs it's less easy to see). But if I just looked for significant findings, it's not hard to find something on most of these runs. An exploratory analysis is not without value, but its value is in generating hypotheses that can then be tested in an a priori design.



So replication is needed to deal with the uncertainties around exploratory analysis. How does pre-registration fit in the picture? Quite simply, it makes explicit the distinction between hypothesis-generating (exploratory) and hypothesis-testing research, which is currently completely blurred. As in the example above, if you tell me in advance what hypothesis you are testing, then I can place confidence in the uncorrected statistical probabilities associated with the predicted effects.  If you haven't predicted anything in advance, then I can't.



This doesn't mean that the results from exploratory analyses are necessarily uninteresting, untrue, or unpublishable, but it does mean we should interpret them as what they are: hypothesis-generating rather than hypothesis-testing.



I'm not surprised at the outcry against pre-registration. This is mega. It would require most of us to change our behaviour radically. It would turn on its head the criteria used to evaluate findings: well-conducted replication studies, currently often unpublishable,  would be seen as important, regardless of their results. On the other hand, it would no longer be possible to report exploratory analyses as if they are hypothesis-testing. In my view, unless we do this we will continue to waste time and precious research funding chasing illusory truths.

References

Flint, J., Greenspan, R. J., & Kendler, K. S. (2010). How Genes Influence Behavior: Oxford University press.



Munafo, M, & Flint, J. (2011). Dissecting the genetic architecture of human personality Trends in Cognitive Sciences, 15 (9), 395-400 DOI: 10.1016/j.tics.2011.07.007

Ten things than can sink a grant proposal: Advice for a young psychologist

 © cartoonstock.com

So you’ve slaved away for weeks giving up any semblance of social or family life in order to put your best ideas on paper. The grant proposal disappears into the void for months during which your mental state oscillates between optimistic fantasies of the scientific glory that will result when your research is funded, and despair and anxiety at the prospect of rejection. And then it comes: the email of doom: “We regret that your application was not successful.” Sometimes just a bald statement, and sometimes embellished with reviewer comments and ratings that induce either rage or depression, depending on your personality type.



There are three things worth noting at this point. First, rejection is the norm: success rates vary depending on the funding scheme, but it’s common to see funding rates around 20% or less. Second, resilience in the face of rejection is a hallmark of the successful scientist, at least as important as intelligence and motivation. Third, there is a huge amount of luck in the grants process: just as with the journal peer review process, reviewers and grant panel members frequently have disparate opinions, and rejection does not mean the work is no good. However, although chance is a big factor, it's not the only thing.



This week I participated in a workshop on “How to get a grant” run by my colleague Masud Husain. We are both seasoned grant reviewers and have served on grants panels. Masud prepared some slides where he noted things that can lead to grant rejection, and I dug out an old powerpoint from a similar talk I’d given in 2005. There was remarkable convergence between the points that we highlighted, based on our experiences of seeing promising work rejected by grants panels. So it seemed worth sharing our insights with the wider world. These comments are tailored to postdocs in psychology/neuroscience in the UK, though some will have broader applicability.

1. Lack of clarity

The usual model for grant evaluation is that the proposal goes to referees with expertise in the area, and is then considered by a panel of people who cover the whole range of areas that is encompassed by the funding scheme. The panel will, of course, rely heavily on expert views, but your case can only be helped if the other panel members can understand what you want to do and why it is important. Even if they can't follow all the technical details, they should be able to follow the lay abstract and introduction.



It's crucial, therefore, that you give the draft proposal to someone who is not an expert in your research topic - preferably not a close friend, but someone more likely to be critical. Ask them to be brutally honest about the bits that they don't understand. When they give you this feedback, don't argue with them or attempt verbal explanations; just rewrite until they do understand it.

2.Badly written proposal

In an ideal world, funders should focus on the content of your proposal rather than the presentation, right? Things like spelling, formatting, and so on are trivial details that only inferior brains worry about, right?



Nope. Wrong on both counts. The people reading your grant are busy. They may have a stack of proposals to evaluate. I have, for instance, been involved in evaluating for a postdoctoral fellowship scheme where my task was to select the top five from a heap of forty odd proposals. The majority of proposals are very good, and so this is a task that is both difficult and important. You  can end up feeling like one of Pavlov's dogs forced to make ever-finer discriminations, and this can put you in a grumpy and unforgiving mood. You take a dim view of proposals where there are typos, spelling errors and missing references. I've seen grant proposals where the applicant failed to turn off 'track changes', or where 'insert reference here' is in the text. In this highly competitive context, there's a high chance that these will go on the 'reject' heap. Even if there are no errors in the text, a densely packed page of verbiage is harder for the reviewer to absorb than a well laid-out document with spacing and headings. You will usually feel that the word limit is too short, and it is tempting to pack in as many words as possible, but this is a mistake. Better ditch material than confront your reviewer with an intimidating wall of words. Judicious use of figures can make a huge difference to the readability of your text, and readability is key. I personally dislike it when numbers are used to indicate references, especially if the reference list then omits titles of referenced papers: people commonly do this to save space, but I like to be able to readily work out what references are being referred to.



Anyone can improve the presentation of a grant. Use of a spell-checker is obvious, but if possible, you should also look at examples of successful applications to see what works in terms of layout etc. You can also Google "good document layout" to find websites full of advice.

3. Boring or pointless proposal

This is a difficult one, because what one person finds riveting, another finds tedious. But if you find your proposal boring, then there's close to zero chance anyone will want to fund it. You should never submit a grant proposal unless you are genuinely excited by the work that you are proposing. You need to ask yourself "Is this what I most want to spend my life doing over the next 2-3 years?" If the answer is no, then rethink the proposal. If yes, then it's crucial to convey your enthusiasm.

4. Lack of hypotheses

This is a common reason for rejection of grant proposals. The phrase 'fishing expedition' is often used to dismiss research that involves looking at a large number of variables in an unfocussed way. As an aside, I remember an exasperated colleague saying that a fishing expedition was an entirely sensible approach if the aim was to catch fish! But funding bodies want to see clear, theoretically-driven predictions with an indication of how the research will test these. A hypothesis should have sufficient generality to be interesting, and usually will be tested by a variety of methods.  For instance, suppose I think that dyslexia may be caused by a particular kind of sensory deficit, and I plan to test children on a range of visual and auditory tasks. I could say that my hypothesis is that there will be differences between dyslexics and controls on the test battery, but this is too vague. It would be better to describe a particular hypothesis of, say visual deficit, and make predictions about the specific tasks that should show deficits. Better still one would set out a general hypothesis about links between the putative deficit and dyslexia, and specify a set of experiments that tested the predictions using a range of methods.



Also, ask yourself, is your hypothesis is falsifiable, and will it yield interesting findings even if it is rejected. If the answer is no, rethink.

5. Overambitious proposal

This is another common reason for rejection of proposals, particularly by junior applicants. In psychology, people commonly overestimate how many participants can be recruited (especially in clinical and longitudinal studies) and how much testing experimenters can do. Of course, you do sometimes see cases where the proposal does not contain enough. But that is much less common that the opposite.



If you are working with human participants, you need to demonstrate that you have thought about two things:

a) Participant recruitment

• Where will you recruit from?


• Have you liaised with referral sources?


• How many suitable people exist?


• What proportion will agree to take part?


• Overall, how many participants will you be able to include in a given period (e.g. 3 months/ 1 year)?


• Have you taken into account the time it will take to get ethics approval?


• Have you costed proposal to take into account reimbursements to partipants and travel?

b) Is your estimate of research personnel realistic?

• How long does it take to test one participant?


• Have you taken into account  the fact that researchers need to spend time on :
  
  - Scheduling appointments
  
  - Travelling
  
  - Scoring up/entering/analysing data
  
  - Doing other academic things (e.g. reading relevant literature, attending seminars)

If you are working with fancy equipment, then you need to consider things like whether you or your research staff will need training to use it, as well as availability.



For more on this, see my previous blogpost about an excellent article by Hodgson and Rollnick (1989): "More fun, less stress: How to survive in research", which details the mismatch between people's expectations of how long research takes and the reality.

6. Overoptimistic proposal

An overoptimistic proposal assumes that results will turn out in line with prediction and has no fall-back position if they don't. A proposal should tell us something useful even if the exciting predictions don't work out. You should avoid multi-stage experiments where the whole enterprise would be rendered worthless if the first experiment failed.

7. Proposal depends on untried or complex methods

You're unlikely to be funded if you propose a set of studies using a method in which you have limited experience, unless you can show that you have promising pilot data. If you do want to move in a new direction, try to link up with someone who has some expertise in it, and consider having them as a collaborator. Although funders don't want to take risk with applicants who have no experience in a new method, they do like proposals to include a training component, and for researchers to gain experience in different labs, even if just for a few months.

8. Overcosted (or undercosted) proposal

This one is easy: Ask for everything that you do need, but don't ask for things you don't need. This is not the time to smuggle in funding for that long-desired piece of equipment unless it is key to the proposal.



The committee will also be unimpressed if you ask for things the host institution should provide. But don't omit crucial equipment because of concerns about expense: just be realistic about what you need and explicitly justify everything.

9. Proposal is too risky

This is much harder one to call. Most funding bodies say they don’t want to fund predictable studies, but they are averse to research where there is high risk of nothing of interest emerging. A US study of NIH funding patterns came to the depressing conclusion that researchers who did high-impact but unconventional research often missed out on funding (Nicholson & Ioannidis, 2012). Funders often state that they like multidisciplinary research, but that runs the risk that, unless methodologically impeccable in all the areas that are covered, it will get turned down.



If you want to include a high-risk element to the proposal, take advice from a senior person whose views you trust - their reaction should give you an indication of whether to go ahead, and if so which aspects will need most justification. And if you want to include a component from a field you are not an expert in, it is vital to take advice from someone senior who does know that area.



It is usually sensible to be up-front about the risky element, and to explain why the risk is worth taking. If you are planning a high-risk project, always have a safety net - i.e. include some more conventional studies in the proposal to ensure that the whole project won't be sunk if the risky bit doesn't pan out.

10. Statistics underspecified or flawed

You need to describe the statistical analysis that you plan, even if it seems obvious to you - if only to demonstrate to the panel that you know what you are doing and have the competence to do it. If you are planning to use complex statistics, get advice from a statistician, and make it clear in the proposal that you have done so. If you don't have adequate statistical skill, consider having a statistician as consultant or collaborator on the grant. And do not neglect power analysis: underpowered studies are a common reason for grants to be rejected in biomedical areas.



Most grants panels are multidisciplinary, and there can be huge
cultural differences in statistical practices between disciplines. I've
seen cases where a geneticist has criticised a psychology project for
lack of statistical power (something geneticists are very hot on), or
where a medic criticises an experimental intervention study for not
using a randomised controlled design. Don't just propose the analysis that you usually do: find out what is best practice to ensure you won't be shot down for a non-optimal research design or analytic approach.

***********************

Finally, remember that the proposed research is one of three elements that will be assessed: the others are the candidate and the institution. There's no point in applying for a postdoctoral fellowship if you have a weak CV: you do need to have publications, preferably first-authored papers. There's a widespread view that you don't stand a chance of funding unless you have papers in high impact journals, but that's not necessarily true, especially in psychology. I'm more impressed by one or two solid first-authored papers than by a long string of publications where you are just one author among many, and (in line with Wellcome Trust policy) I don't give a hoot about journal impact factors. Most funding agencies will give you a steer on whether your CV is competitive if you ask for advice on this.



As far as the institution goes, it helps to come from a top research institution, but the key thing is to have strong institutional support, with access to the resources you need and to supportive colleagues. You will need a cover letter from your institution, and the person writing it should convey enthusiasm for your proposal and be explicit in making a commitment to providing space and other resources.



Good luck!



Reference

Nicholson JM, & Ioannidis JP (2012). Research grants: Conform and be funded. Nature, 492 (7427), 34-6 PMID: 23222591

Genetic variation and neuroimaging: some ground rules for reporting research

Those who follow me on Twitter may have
noticed signs of tetchiness in my tweets over the past few weeks. In the course
of writing a review article, I’ve been reading papers linking genetic variants
to language-related brain structure and function. This has gone more slowly than I expected for
two reasons. First, the literature gets ever more complicated and technical:
both genetics and brain imaging involve huge amounts of data, and new methods
for crunching the numbers are developed all the time. If you really want to understand
a paper, rather than just assuming the Abstract is accurate, it can be a long,
hard slog, especially if, like me, you are neither a geneticist nor a
neuroimager. That’s understandable and perhaps unavoidable. The other reason,
though, is less acceptable. For all their complicated methods, many of the
papers in this area fail to tell the reader some important and quite basic
information. This is where the tetchiness comes in. Having burned my brains out
trying to understand what was done, I then realise that I have no idea about
something quite basic like the sample size. The initial assumption is that I’ve
missed it, and so I wade through the paper again, and the Supplementary Material, looking
for the key information. Only when I’m absolutely certain that it’s not there,
am I reduced to writing to the authors for the information. So
this is a plea – to authors, editors and reviewers. If a paper is concerned
with an association between a genetic variant and a phenotype (in my case the
interest is in neural phenotypes, but I suspect this applies more widely) then
could we please ensure that the following information is clearly reported in
the Methods or Results section

1. What genetic variant are we talking about?
You might think this is very simple, but it’s not: for instance, one of the
genes I’m interested in is CNTNAP2, which has been associated with a range of
neurodevelopmental disorders, especially those affecting language. The evidence
for a link between CNTNAP2 and developmental disorders comes from studies that
have examined variation in single-nucleotide polymorphisms or SNPs. These are
segments of DNA that are useful in revealing differences between people because
they are highly variable. DNA is composed of four bases, C, T, G, and A in
paired strands. So for instance, we might have a locus where some people have
two copies of C, some have two copies of T, and others have a C and a T. SNPs
are not  necessarily a functional part of
the gene itself – they may be in a non-coding region, or so close to a gene that
variation in the SNP co-occurs with variation in the gene. Many different SNPs
can index the same gene. So for CNTNAP2, Vernes et al (2008)tested 38 SNPs,
ten of which were linked to language problems. So we have to decide which SNP
to study – or whether to study all of them. And we have to decide how to do the
analysis. For instance, SNP rs2710102 can take the form CC, CT or TT. We could
look for a dose response effect (CC < CT < TT) or we could compare CC/CT with TT, or we could compare CC with CT/TT. Which of these we do may depend on whether prior research suggests the genetic effect is additive or dominant, but for brain imaging studies grouping can also be dictated by practical considerations: it’s usual to compare just two groups and to combine genotypes to give a reasonable sample size. If you’ve followed me so far, and you have some background in statistics, you will already be starting to see why this is potentially problematic. If the researcher can select from ten possible SNPs, and two possible analyses, the opportunities for finding spuriously ‘significant’ results are increased. If there are no directional predictions – i.e. we are just looking for a difference between two groups, but don’t have a clear idea of what type of difference will be associated with ‘risk’ – then the number of potentially ‘interesting’ results is doubled.

For CNTNAP2, I found two papers that had
looked at brain correlates of SNP rs2710102. Whalley et al (2011) found that adults
with the CC genotype had different patterns of brain activation from CT/TT
individuals. However, the other study, by Scott-van Zeeland et al (2010), treated
CC/CT as a risk genotype that was compared with TT. (This was not clear in the
paper, but the authors confirmed it was what they did).

 Four studies looked at another SNP -
rs7794745, on the basis that an increased risk of autism had been reported for
the T allele in males. Two of them (Tan et al, 2010; Whalley et al, 2010) compared TT vs TA/AA and two (Folia et al, 2011; Kos et al, 2012) compared
TT/TA with AA. In any case, the ground is rather cut from under the feet of
these researchers by a recent failure to replicate an association of this SNP
with autism (Anney et al, 2012).

2. Who are the participants? It’s not very
informative to just say you studied “healthy volunteers”. There are some types
of study where it doesn’t much matter how you recruited people. A study looking
at genetic correlates of cognitive ability isn’t one of them. Samples of
university students, for instance, are not representative of the general
population, and aren’t likely to include many people with significant language
problems.

3. How many people in the study had each type
of genetic variant? And if subgroup analyses are reported, how many people in
each subgroup had each type of genetic variant? I've found that papers in top-notch journals often fail to provide this basic
information.

Why is this important? For a start, likelihood
of showing significant activation of a brain region will be affected by sample
size. Suppose you have 24 people with genotype A and 8 with genotype B. You
find significant activation of brain region X in those with genotype A, but not
for those with genotype B. If you don’t do an explicit statistical comparison
of groups (you should - but many people don’t) you may be misled into concluding that brain
activation is defective in genotype B – when in fact you just have low power to
detect effects in that group because it is so small.

In addition, if you don’t report the N, then
it’s difficult to get an idea of the effect size and confidence interval for
any effect that is reported. The reasons why this is optimal are
well-articulated here. This issue has been much discussed in psychology, but seems not to have
permeated the field of genetics, where reliance on p-values seems the norm. In
neuroimaging it gets particularly complicated, because some form of correction
for ‘false discovery’ will be applied when multiple comparisons are conducted. It’s
often hard to work out quite how this was done, and you can end up staring at
a table that shows brain regions and p-values, with only a vague idea of how
big a difference there actually is between groups.

 Most of the SNPs that are being used in brain studies are ones that
were found to be associated with a behavioural phenotype in large-scale genomic
studies where the sample size would include hundreds if not thousands of
individuals, so small effects could be detected. Brain-based studies often use
sample sizes that are relatively small, but some of them find large, sometimes
very large, effects. So what does that mean? The optimistic interpretation is
that a brain-based phenotype is much closer to the gene effect, and so gives
clearer findings. This is essentially 
the argument used by those who talk of ‘endophenotypes’ or ‘biomarkers’.
There is, however, an alternative, and much more pessimistic view, which is
that studies linking genotypes with brain measures are prone to generate false
positive findings, because there are too many places in the analysis pipeline
where the researchers have opportunities to pick and choose the analysis that
brings out the effect of interest most clearly. Neuroskeptic has a nice blogpost illustrating this well-known problem in
the neuroimaging area; matters are only made worse by uncertainty re SNP classification
(point 1).

A source of concern here is the
unpublishability of null findings. Suppose you did a study where you looked at,
say, 40 SNPs and a range of measures of brain structure, covering the whole
brain. After doing appropriate corrections for multiple comparisons, nothing is
significant. The sad fact is that your study is unlikely to find a home in a
journal. But is this right? After all, we don’t want to clutter up the
literature with a load of negative results. The answer depends on your sample
size, among other things. In a small sample, a null result might well reflect
lack of statistical power to detect a small effect. This is precisely why
people should avoid doing small studies: if you find nothing, it’s
uninterpretable. What we need are studies that allow us to say with confidence
whether or not there is a significant gene effect.

4. How do the genetic/neuroimaging results relate to cognitive measures in your sample?  Your notion that ‘underactivation of brain area
X’ is an endophenotype that leads to poor language, for instance, doesn’t look
very plausible if people who have such underactivation have excellent language skills. Out
of five papers on CNTNAP2 that I reviewed, three made no mention of cognitive measures,
one gathered cognitive data but did not report how it related to genotype or
brain measures, and only one provided some relevant, though sketchy, data.

5. Report negative findings. The other kind of
email I’ve been writing to people is one that says – could you please clarify
whether your failure to report on the relationship between X and Y was because
you didn’t do that analysis, or whether you did the analysis but failed to find
anything. This is going to be an uphill battle, because editors and reviewers
often advise authors to remove analyses with nonsignificant findings. This is a
very bad idea as it distorts the literature.

And last of all....

A final plea is not so much to journal
editors as to press officers. Please be aware that studies of common SNPs aren't the same as studies of rare genetic mutations. The genetic variants in the
studies I looked at were all relatively common in the general population, and so
aren't going to be associated with major brain abnormalities. Sensationalised
press releases can only cause confusion:

This release on the Scott van-Zeeland (2010) study described neuroimaging
findings from  CNTNAP2 variants that are found in over 70% of the population. It claims that: 

• “A gene variant tied to autism rewires the
  brain"

• "Now we can begin to unravel the mystery
  of how genes rearrange the brain's circuitry, not only in autism but in many
  related neurological disorders."

• “Regardless of their diagnosis, the children
  carrying the risk variant showed a disjointed brain. The frontal lobe was
  over-connected to itself and poorly connected to the rest of the brain”

• "If we determine that the CNTNAP2
  variant is a consistent predictor of language difficulties, we could begin to
  design targeted therapies to help rebalance the brain and move it toward a path
  of more normal development."

Only at the end of the press release, are we
told that "One third of the population [sic: should be two thirds] carries this variant in its DNA.
It's important to remember that the gene variant alone doesn't cause autism, it
just increases risk." 

References

Anney, R., Klei, L.,
Pinto, D., Almeida, J., Bacchelli, E., Baird, G., . . . Devlin, B. .
Individual common variants exert weak effects on the risk for autism spectrum
disorders. Human Molecular Genetics, 21(21), 4781-4792. doi: 10.1093/hmg/dds301(2012)

V. Folia, C. Forkstam, M.
Ingvar, P. Hagoort, K. M. Petersson, Implicit artificial syntax processing:
Genes, preference, and bounded recursion. Biolinguistics 5,  (2011).

M. Kos et al., CNTNAP2
and language processing in healthy individuals as measured with ERPs. PLOS One
7,  (2012).

Scott-Van Zeeland, A., Abrahams, B., Alvarez-Retuerto, A., Sonnenblick, L., Rudie, J., Ghahremani, D., Mumford, J., Poldrack, R., Dapretto, M., Geschwind, D., & Bookheimer, S. (2010). Altered Functional Connectivity in Frontal Lobe Circuits Is Associated with Variation in the Autism Risk Gene CNTNAP2 Science Translational Medicine, 2 (56), 56-56 DOI: 10.1126/scitranslmed.3001344

G. C. Tan, T. F. Doke, J.
Ashburner, N. W. Wood, R. S. Frackowiak, Normal variation in fronto-occipital
circuitry and cerebellar structure with an autism-associated polymorphism of
CNTNAP2. Neuroimage 53, 1030 (2010).

Vernes, S. C., Newbury,
D. F., Abrahams, B., Winchester, L., Nicod, J., Groszer, M., . . . Fisher, S.  A functional genetic link between distinct developmental language
disorders. New England Journal of Medicine, 359, 2337-2345. (2008).

H. C. Whalley et al.,
Genetic variation in CNTNAP2 alters brain function during linguistic processing
in healthy individuals. Am. J. Med. Genet. B 156B, 941 (2011).

Psychology: Where are all the men?

There's a lot of interest in under-representation of women in certain science subjects, but in psychology, there's more concern about a lack of men. A quick look at figures from UCAS (Universities & Colleges Admissions Service) shows massive differences in gender ratios for different subjects. In figure 1 I’ve plotted the percentage of women accepted for subjects that had at least 6000 successful applicants to degree courses in 2011.

Fig. 1. % Females accepted on popular UK degree courses 2011

Given the large sample sizes, the sex differences are statistically
significant for all subjects except Media Studies, which is bang on 50%.
As a psychologist, I found the most surprising thing about this plot
was the huge preponderance of women in psychology. This didn’t square
with my experiences: my colleagues include a good mix of men and women,
so I was keen to find the explanation for the mismatch. There seemed to be several possible explanations, which aren’t mutually exclusive, namely:

• Oxford University, where I work, may be biased in favour of men


• The proportions of women decline with career stage


• The proportion of women in psychology may have increased since I was a student


• The proportion of women may vary with sub-area of psychology

So I set off to track down the evidence for these different explanations.

Is Oxford University biased against women?

I’m leading our department’s Athena SWAN panel, whose remit is to identify and remove barriers to women’s progress in scientific careers. In order to obtain an Athena SWAN award, you have to assemble a lot of facts and figures about the proportions of women at different career stages, and so I already had at my fingertips some relevant statistics. (You can find these here). Over the past three years, our student intake ranged from 66% -71% women: rather lower than the UCAS figure of 78%. However, acceptance rates were absolutely equivalent for men and women. The same was true for staff appointments: the likelihood of being accepted for a job did not differ by gender. So with a sigh of relief I think we can exclude this line of explanation.

Does the proportion of women in psychology decline with career stage?

I have a research post and so don’t do much teaching. Have I got a distorted view of the gender ratios because my interactions are mostly with more senior staff? This looks believable from the data on our department. Postgraduate figures ranged from 65%-70% women. Ours is a small department, and so it is difficult to be confident in trends, but in 2011 there were 16/27 (59%) female postdocs, 6/11 (55%) female lecturers, 6/13 (46%) senior researchers and 4/11 (36%) female professors. This trend for the proportion of women to decline as one advances through a career is in line with what has been observed in many other disciplines. We also obtained data from other top-level psychology departments for comparison, and similar trends were seen.

Has the proportion of women in psychology increased over time?

My recollection of my undergraduate days was that male psychology students were plentiful. However, I was an undergraduate in the dark ages of the early 1970s when there were only five Oxford colleges that accepted women, and a corresponding shortage of females in all subjects. So I had a dig around to try to get more data. The UCAS statistics go back only to 1996, and the proportion of women in psychology hasn’t changed: 78% in 1996, 78% in 2011. However, data from the USA show a sharp increase in the proportion of women obtaining psychology doctorates from 1960 (18%) through 1972 (27%) to 1984 (50%). This, of course, is in part a consequence of the increase of women in higher education in general. But that isn’t a total explanation: Figure 2 compares proportions of female PhDs over time in different subject areas, and one can see that psychology shows a particularly pronounced increase compared with other disciplines.

Fig 2. Percentages of PhDs by women in the USA: 1950-1984

Does the proportion of women in psychology vary with sub-area?

The term ‘psychology’ covers a huge range of subject matter with different historical roots. Most areas of academic psychology make some use of statistics, but they vary considerably in how far they require strong quantitative or computational skills. For instance, it would be difficult to specialise in the study of perception or neuroscience without being something of a numbers nerd: that’s generally less true for developmental, clinical, interpersonal or social psychology, which require other skills sets. I looked at data from the American Psychological Association (APA), which publishes the numbers of members and fellows in its different Divisions. The APA is predominantly a professional organisation, and non-applied areas of psychology are not strongly represented in the membership. Nevertheless, one can see clear gender differences, which generally map on to the expectation that women are more focused on the caring professions, and men are more heavily represented in theoretical and quantitative areas. Figure 3 shows relevant data for sections with at least 700 members. It is also worth noting that the graph illustrates the decrease in the proportions of women going from membership to fellowship, a trend bucked by just one Division.

Fig 3. Data from American psychological association: Division membership 2011

What, if anything, should we do?

The big question is how far we should try to manipulate gender differences when we find them. I’ve barely scratched the surface in my own discipline, psychology, yet it’s evident that the reasons for such differences are complex. Figure 2 alone makes it clear that women in Western societies have come a long way in the past half-century: far more of us go to university and do PhDs than was the case fifty years ago. Yet the proportion of women declines as we climb the career ladder. In quantifying this trend, it’s important to compare like with like: those who are in senior positions now are likely to have trained at a time when the gender ratio was different. But it's clear from many surveys that demographics changes can't explain the dearth of women in top jobs: there are numerous reasons why women are more likely than men to leave an academic career – see, for instance, this depressing analysis of reasons why women leave chemistry. In our department we are committed to taking steps to ensure that gender does not disadvantage women who want to pursue an academic career, and I am convinced that with even quite minor changes in culture we can make a difference.



The point I want to stress here, though, is that I see this issue - creating a female-friendly environment for women in psychology-  as separate from the issue of subject preference. I worry that the two issues tend to get conflated in discussions of gender equality. My personal view is that psychology is enriched by having a mix of men and women, and I share the concerns expressed here about difficulties that arise when the subject becomes heavily biased to one gender. However, I am pretty uncomfortable with the idea of trying to steer people’s career choices in order to even out a gender imbalance.



Where this has been tried, my impression is that it's mostly been in the direction of trying to encourage more girls into male-dominated subjects. In effect, the argument is that girl's preferences  are based on wrong information, in that they are unduly influenced by stereotypes. For instance, the Institute of Physics has done a great deal of work on this topic, and they have shown that there are substantial influences of schooling on girls’ subject choices. They concluded that the weak showing of girls in physics can be attributed to lack of inspirational teaching, and a perception among girls that physics is a boys’ subject. They have produced materials to help teachers overcome these influences, and we’ll have to wait and see if this makes any appreciable difference to the proportions of girls taking up the subject (which according to UCAS figures has been pretty stable for 15 years: 19% in 1996 and 18% in 2011).



It's laudable that the Institute of Physics is attempting to improve the teaching of physics in our schools, and to ensure girls do not feel excluded. But if they are right, and gender stereotyping is a major determinant of subject choices, shouldn’t we then adopt similar policies to other subjects that show a gender bias, whether this be in favour of girls or boys?



Interestingly, Marc Smith has produced relevant data in relation to A-level psychology, which is dominated by girls, and perceived by boys as a ‘girly’ subject. So should we try to change that? As Smith notes, the female bias seems linked to a preference for schools to teach A-level psychology options that veer away from more quantitative cognitive topics. Here we find that psychology provides an interesting test case for arguments around gender, because within the subject there are consistent biases for males and females to prefer one kind of sub-area to another. This implies that to alter the gender balance you might need to change what is taught, rather than how it is taught, by giving more prominence to the biological and cognitive aspects of psychology. If true, it might be easier to alter gender ratios in psychology than in physics, but only by modifying the content of the syllabus.



One of the IOP's recommendations is: "Co-ed schools should have a target to
exceed the current national average of 20% of physics A-level students
being girls." But surely this presumes an agenda whereby we aim for
equality of genders in all subjects, with equivalent campaigns to
recruit more boys into nursing, psychology and English? I'm not saying
this would necessarily be a bad thing, but I wonder at the automatic assumption that it has to be a good thing - or even an achievable thing. There are obvious disadvantages of gender imbalances in any subject area - they simply reinforce stereotypes, while at the same time creating challenges at university and in the workplace for those rare individuals who buck the trend and take a
gender-atypical subject. But the kinds of targets set by the IOP make me uneasy nonetheless. The downside of an insistence on gender balance is a sense of coercion, whereby children are made to feel that their choice of subject isn't a real choice, but is only made because they  have been brainwashed by gender stereotypes. Yes, let's do our best to teach boys and girls in an inspiring and gender-neutral fashion, but, as the example of psychology demonstrates, we are still likely to find that females and males tend to prefer different kinds of subject matter.



References
 

Smith, M (2011). Failing boys, failing psychology The Psychologist, 24 (5), 390-391 Other: WOS:000290745000037
 



Howard, A., & et al, . (1986). The changing face of American psychology: A report from the Committee on Employment and Human Resources. American Psychologist, 41 (12), 1311-1327 DOI: 10.1037//0003-066X.41.12.1311