How to bury your academic writing

Inappropriate use of journal impact factors has been much in
the spotlight. The impact factor is not only a poor indicator of research
quality but it is also blamed for delaying publication of good science, and even encouraging dishonesty.  My own
experience is in line with this: some of my most highly-cited work has appeared
in relatively humble journals. In the age of the internet, there are three
things that determine if a paper gets noticed: it needs to be tagged so that it
will be found on a computer search, it needs to be accessible and not locked
behind a paywall, and it needs to  be
well-written and interesting.

While I'm not a slave to metrics, I am, like all academics
these days, fascinated by the citation data provided by sources such as Google
Scholar, and pleased when I see that something I have written has been cited by
others. The other side of the coin is the depression that ensues when  I find that a paper into which I have
distilled my deepest wisdom has been ignored by the world. Often, it's hard to
say why one article is popular and another is not. The papers I'm proudest of
tend to be those that required the greatest intellectual effort, but these are
seldom the most cited. Typically, they are the more technical or mathematical
articles; others find them as hard to read as I found them to write.  Google Scholar reveals, however, one factor
that exerts a massive impact on whether a paper is cited or not: whether it
appears in a journal or an edited book.

I've had my suspicions about this for some time, and it has
made me very reluctant to write book chapters. This can be difficult. Quite often, a chapter for the proceedings is the
price one is expected to pay for an expenses-paid invitation to a conference.
And many of my friends and colleagues get overtaken by enthusiasm for
editing a book and are keen for me to write something. But statistical analysis of citation
data confirms my misgivings.

Google Scholar is surprisingly coy in terms of what it
allows you to download. It will show you citations of your papers on the
screen, but I have not found a way to download these data.  (I'm a recent convert to data-scraping in R,
but you get a firm rap over the knuckles for improper behaviour if you attempt
to use this approach to probe Google Scholar too closely). So in what follows I treated rank order of citations, rather than absolute citation level
as my dependent variable. I
downloaded a listing of my papers, ranked by citations, and
coded them according to whether the article appeared in a journal or as a
book chapter. Book chapters tend not to be empirical – they are more often
review papers, or conceptual pieces - so to control for that I subdivided the
journal articles into empirical and theoretical/review pieces. I also excluded papers published after 2007,
to allow for the fact that recent papers haven't had a chance to get cited
much, as well as any odd items such as book reviews. To make interpretation more intuitive, I inverted the rank order, so that
a high score meant lots of citations, and the boxplots showing the results are
in the Figure below.

Citation rank by Publication type. High rank indicates more citations. There is no significant difference between journal reviews and empirical papers, both of which have significantly higher citation rank than book chapters (p < .001)

Because I'm nerdy about these things, I did some stats, but
you don't really need them. The trend is very clear in the boxplot: book
chapters don't get cited. Well, you
might say, maybe this is because they aren't so good; after all, book chapters
aren't usually peer reviewed. It could be true, but I doubt it. My own
appraisal is that these chapters contain some of my best writing, because they
allowed me to think about broader theoretical issues and integrate ideas from
different perspectives in a way that is not so easy in an empirical article. Perhaps,
then, it's because these papers are theoretical 
that they aren't cited. But no: look at the non-empirical pieces published
in journals. Their citation level is just as high as papers reporting empirical
data. Could publication year play a part? As mentioned above, I excluded papers
from the past five years;  after doing
this, there was no overall correlation between citation level and publication
year.



Things may be different for other disciplines, especially in
humanities, where publication in books is much more common. But if you publish in
a field where most publications are in journals, then I suspect the trend I see
in my own work will apply to you too. Quite simply, if you write a chapter for an edited book, you might as
well write the paper and then bury it in a hole in the ground.



Accessibility is the problem. However
good  your chapter is, if readers don't
have access to the book, they won't find it. In the past, there was at least a
faint hope that they may happen upon the book in a library, but these days,
most of us don't bother with any articles that we can't download from the
internet. 

I'm curious as to whether publishers have any plans to tackle this issue. Are they still producing edited collections? I still get asked to contribute to these from time to time, but perhaps not so often as in the past. An obvious solution would be to put edited books online, just like journals, but there would need to be a radical rethink of access costs if so. Nobody is going to want to pay $30 to download a single chapter. Maybe publishers could make book chapters freely available one or two years after publication  - I see no purpose in locking this material away from the public, and it seems unlikely this would damage book sales. If publishers don't want to be responsible for putting material online, they could simply return copyright to authors, who would be free to do so.



My own solution would be for editors of such collections to take matters into their own hands, bypass publishers altogether, and produce freely downloadable, web-based copy. But until that happens, my advice to any academic who is tempted to write a chapter for an edited collection is don't.

Reference

Eve Mardera, Helmut Kettenmann, & Sten Grillner (2010). Impacting our young Proceedings of the National Academy of Sciences, 107 DOI: 10.1073/pnas.1016516107

The bewildering bathroom challenge

The tap is a simple but genius piece of design. You turn a
handle one way and water flows. You turn it the other way, and the water stops.
The bathplug is even simpler. You find a pliable, waterproof substance and cut
it to fit exactly into the hole out of which the water flows, and you equip it
with handle or chain on top that you can grasp to remove it.

Hotels the world over, however, are not satisfied with such
simplicity. They conspire to make the task of producing water and containing it
ever more baffling. I had wondered whether they were just more focused on
appearance than function, but this website
makes it clear that it's deliberate: It's worth quoting the blurb in full:

"A lot of attention in the design world is focused on
creating products that are intuitive and easy to use, but sometimes a little
ambiguity can be a good thing. Designed for use in restaurant and hotel
bathrooms these taps embrace ambiguity to create a sense of intrigue to provide
a more engaging interaction."

Hmm. Ambiguity is not really what I'm seeking in a bathroom.
And engaging isn't the word I'd use for my interaction, as I try turning, pressing,
pulling levers and dials and waving my hands around under taps. It usually ends
in quite a bit of swearing. And that's in the cases where I can actually find something to push, pull or turn.

I wonder if there is some secret competition, known only to
hoteliers, scored as follows:

• 1 point for each room where a pool of water in basin
  indicates they haven't mastered the plug


• 2 points for each guest who gets a wet head when trying to
  turn on the bath tap


• 3 points for each guest who has to get someone from
  reception to explain how to turn on the tap


• 4 points for each guest too shy to ask reception so doesn't
  wash during their stay

Hotels in the old Soviet Union
had a simpler approach to frustrating their guests - they just didn't provide a
bath plug.

The devaluation of low-cost psychological research

Psychology encompasses a wide range of subject areas,
including social, clinical and developmental psychology, cognitive psychology
and neuroscience. The costs of doing different types of psychology vary hugely.
If you just want to see how people remember different types of material, for
instance, or test children's understanding of numerosity, this can be done at very
little cost. For most of the psychology I did as an undergraduate, data
collection did not involve complex equipment, and data analysis was pretty
straightforward - certainly well within the capabilities of a modern desktop
computer. The main cost for a research proposal in this area would be for staff
to do data collection and analysis. Neuroscience, however, is a different
matter. Most kinds of brain imaging require not only expensive equipment, but
also a building to house it and staff to maintain it, and all or part of these
costs will be passed on to researchers. Furthermore, data analysis is usually
highly technical and complex, and can take weeks, or even months, rather than
hours. A project that involves neuroimaging will typically cost orders of
magnitude more than other kinds of psychological research.

In academic research, money follows money. This is quite
explicit in funding systems that reward an institution in proportion to their
research income. This makes sense: an institution that is doing costly research
needs funding to support the infrastructure for that research. The problem is
that the money, rather than the research, can become the indicator of success. Hiring
committees will scrutinise CVs for evidence of ability to bring in large
grants. My guess is that, if choosing between one candidate with strong
publications and modest grant income vs. another with less influential
publications and large grant income, many would favour the latter.
Universities, after all, have to survive in a tough financial climate, and so
we are all exhorted to go after large grants to help shore up our institution's
income. Some Universities have even taken to firing people who don't bring in
the expected income. This means that cheap cost-effective research in
traditional psychological areas will be devalued relative to more expensive
neuroimaging.

I have no quarrel, in principle, with psychologists doing
neuroimaging studies - some of my best friends are neuroimagers -  and it is important that if good science is to be done in
this area that it should be properly funded. I am uneasy, though, about an
unintended consequence of the enthusiasm for neuroimaging, which is that it has
led to a devaluation of the other kinds of psychological research. I've been
reading Thinking Fast and Slow,
by Daniel Kahneman, a psychologist who has the rare distinction of
being a Nobel Laureate. This is just one example of a psychologist who has made major advances without using brain scanners. I couldn't help thinking that Kahneman would not fare
well in the current academic climate, because his experiments were simple,
elegant ... and inexpensive.

I've suggested previously that systems of academic rewards
need to be rejigged to take into account not just research income and
publication outputs, but the relationship between the two. Of course, some
kinds of research require big bucks, but large-scale grants are not always
cost-effective. And on the other side of the coin, there are people who do
excellent, influential work on a small budget.

I thought I'd see if it might be possible to get some hard
data on how this works in practice. I used data for Psychology Departments from
the last Research Assessment Exercise (RAE), from this website, and matched
this up against citation counts for publications that came out in the same time
period (2000-2007) from Web of Knowledge. The latter is a bit tricky, and I'm
aware that figures may contain inaccuracies, as I had to search by address,
using the name of the institution coupled with the words Psychology and UK. This will miss articles that don't have these words in the address. Also when double-checking the numbers, I  found that for a search by address, results can fluctuate from one occasion to the next. For these reasons, I'd urge readers to treat the results with caution, and
I won't refer to institutions by name. Note too that though I restrict consideration to articles between 2000-2007, the citations extend
beyond the period when the RAE was completed. Web of Knowledge helpfully gives
you an H-index for the institution if you ask for a citation report, and this
is what I report here, as it is more stable across repeated searches than the citation count. Figure 1 shows how research income for a department
relates to its H-index, just for those institutions deemed research active,
which I defined as having a research income of at least £500K over the reporting
period. The overall RAE rating is colour-coded into bandings, and the symbol denotes
whether or not the departmental submission mentions neuroimaging as an
important part of its work. 

Data from RAE and Web of Knowledge: treat with caution!

Several features are seen in these data, and most are
unsurprising:

• Research income and H-index are positively correlated, r =
  .74 (95%CI .59-.84) as we would expect. Both variables are correlated with the
  number of staff entered in the RAE, but the correlation between them remains
  healthy when this factor is partialled out, r = .61 (95%CI .40-.76).


• Institutions coded as doing neuroimaging have bigger grants: after taking into account differences in number of staff, the mean income
  for departments with neuroimaging was £7,428K and for those without it was
  £3,889K (difference significant at p = .01).


• Both research income and H-index are predictive of RAE
  rankings: the correlations are .68 (95% CI .50-.80) for research income and .79
  (95% CI .66-.87) for H-index, and together they account for 80% of the variance
  in rankings. We would not expect perfect prediction, given that the RAE committee
  went beyond metrics to assess aspects of research quality not
  reflected in citations or income. And in addition, it must be noted that the
  citations counted here are for all researchers at a departmental address, not
  just those entered in the RAE.

A point of concern to me in these data, though, is the wide
spread in H-index seen for those institutions with the highest levels of grant
income. If these numbers are accurate, some departments are using their
substantial income to do influential work, while others seem to achieve no more
than other departments with much less funding. There may be reasonable
explanations for this - for instance, a large tranche of funding may have been
awarded in the RAE period but not had time to percolate through to
publications. But nevertheless, it adds to my concern that we may
be rewarding those who chase big grants without paying sufficient attention to
what they do with the funding when they get it.

What, if anything, should we do about this? I've toyed in
the past with the idea of a cost-efficiency metric (e.g. citations divided by
grant income), but this would not work as a basis for allocating funds, because
some types of research are intrinsically more expensive than others. In
addition, it is difficult to get research funding, and success in this arena is
in itself an indicator that the researchers have impressed a tough committee of
their peers. So, yes, it makes sense to treat level of research funding as one indicator
of an institution's research excellence when rating departments to determine
who gets funding. My argument is simply that we should be aware of the
unintended consequences if we rely too heavily on this metric. It would be nice
to see some kind of indicator of cost-effectiveness included in ratings of
departments alongside the more traditional metrics. In times of financial
stringency, it is particularly short-sighted to discount the contribution of
researchers who are able to do influential work with relatively scant
resources.

Communicating science in the age of the internet

© www.CartoonStock.com

Here's an interesting test for those on Twitter. You see a
tweet giving a link to an interesting topic. You click on the link and see it's
a YouTube piece. Do you (a) feel pleased that it's something you can watch or
(b) immediately lose interest. The answer is likely to depend on content, but
also on how long it is. Typically, if I see a video is longer than 3 minutes,
I'll give up unless it looks super-interesting.

Test #2 is for those of you who are scientists. You have to
give a presentation about a recent piece of work to a non-specialist audience.
How long do you think you will need? (a) one hour; (b) 20 minutes; (c) 10
minutes; (d) 3 minutes.

If you're anything like me, there's a disconnect between
your reactions to these different scenarios. The time you feel you need to
communicate to an audience is much greater than the time you are willing to
spend watching others. Obviously, it's not a totally fair comparison: I'm
willing to spend up to an hour listening to a good lecture (but
no more!); though to tell the truth, it's an unusual lecturer who can keep
me interested for the whole duration.

Those who use the internet to communicate science have
learned that the traditional modes of academic communication are hopelessly
ill-suited for drawing in a wider audience. TED
talks have been a remarkably successful phenomenon, and are a million miles
from the normal academic lecture: the ones I've seen are typically no longer than
15 minutes and make minimal use of visual aids. The number of site visits for
TED talks is astronomically higher than, for instance, Cambridge University's
archive of Film Interviews
With Leading Thinkers, where Aaron Klug has had around 300 hits in just
over one year, and Fred Sanger a mere 148. The reason is easy to guess: many of
these Cambridge interviews last two hours or more. They constitute priceless archive
material, and a wealth of insights into the influences that shape great academic minds, but they aren't suited to the casual viewer.

For most academics, though, shorter pieces pose a dilemma:
they don't allow you to present the evidence for what you are saying. I felt
this keenly when viewing a TED
talk by autism expert Ami Klin. At 22 minutes, this was rather longer than
the usual TED talk, but Klin is an engaging speaker, and he held my attention
for the whole time. As I listened, though, I became increasingly uneasy. He was
making some pretty dramatic claims. Specifically, as the accompanying blurb
stated: "Ami Klin describes a new early detection method that uses eye-tracking
technologies to gauge babies' social engagement skills and reliably measure
their risk of developing autism". I was very surprised at the claims made
for eye-tracking, and the data shown in the presentation were unconvincing. More
generally, Klin talked about universal screening for 6-month-olds, but I was not sure that he understood the
requirements for an effective screening test. After the end of the talk I
checked out Klin's publications on Web of Science and couldn't find any
published papers that gave a fuller picture to back up this claim. I asked my
colleagues who work in autism and none of them was aware of such evidence. I
emailed Klin last week to ask if he can point me to relevant sources but so far
I've not had a reply. (If I do, I'll add the information). At the time of writing, his talk has had over 132,000
views.

So we have a dilemma here. Nearly everyone agrees that
scientists should engage with audiences beyond their traditional narrow
academic confines. But the usual academic lecture, saturated with PowerPoint
explaining and justifying every statement, is ill-suited to such an audience.
However, if we reduce our communications to the bottom line, then the audience
has to take a lot on trust. It may be impossible to judge whether the
speaker is expressing an accepted mainstream view. If, as in the Klin case, the
speaker is both famous and charismatic, then it's unlikely that a general
audience will realise that many experts in his field would want to see a lot more hard evidence before accepting what he was saying.

I've been brooding about this issue because I've recently
joined up with some colleagues in a web-based campaign to raise awareness of
language impairments in children. My initial idea was that we'd post lectures
by experts, attempting to explain what we know about the nature, causes, and
impacts of language impairments. Fortunately, we were dissuaded from this idea
by our friends in TeamSpirit, a public relations company who have come on board
to help us get launched. With their assistance, we've posted several videos and
worked out a clearer idea of what our
YouTube channel should do. We will have professionally produced films that feature
the experiences of young people with language impairments and their families,
as well as the professionals working with them. But we also wanted to ensure
that the material we put out was evidence-based, and to include some pieces on
issues where there were relevant research findings. We were advised that any
piece by a talking academic head should be no more than 3 minutes long. I could
see the wisdom of that, given my own reactions to longer video pieces. But I
was uncomfortable. In 3 minutes, it's impossible to do more than give a bottom
line. I didn't want people to have to take what I said on trust: I wanted them
to have access to the evidence behind it. Well, we're now experimenting with an
approach that I think may work to keep everyone happy. Our academic-style talks
will stick to the 3 minute limit, but will be associated with a link to a PowerPoint
presentation which will give a fuller account. This is still shorter than the
usual academic talk - we aim for around 15-20 slides, all of which should be
self-explanatory without needing an oral narrative. And, crucially, the PowerPoint
will include references to peer-reviewed research to support what is said, and
will include a link to a reference list, including where possible a review
article. I anticipate that most people who visit our YouTube site will only get
as far as the 3 minute video. That's absolutely fine - after all, only a small
proportion of potential visitors will be evidence geeks. But, importantly, the
evidence will be there for those who want it. The PowerPoint will give the bare
bones, and the references will allow people to track back to the original
sources.

We live in exciting times, where it has become remarkably
easy to harness the power of the internet to disseminate research. The
challenge is to do so in a way that is effective while preserving academic
rigour.

Schoolgirls' health put at risk by Catholic view on vaccination

Today's Time Newsfeed carries a remarkable story: parents of children attending Catholic schools in Calgary were sent a special letter to accompany details of a vaccination programme against human papillomavirus (HPV), which protects against cervical cancer. In it, local bishops wrote: "Although school-based immunization delivery systems generally result in high numbers of students completing immunization, a school-based approach to vaccination sends a message that early sexual intercourse is allowed.”
I find this amazing for several reasons:

• There's a complete failure to understand what affects teenagers' behaviour. Do the bishops seriously think that teenaged girls who are thinking of having sex say to themselves "Oh, wait a minute. I might get HPV. Let's not do it." Potential consequences of sex include a host of sexually transmitted diseases, as well as pregnancy. If these don't put girls off, then why should a risk of HPV? 


• HPV is a sexually transmitted disease. You can get it if you are a virgin who marries someone with HPV. You can get it if you are raped (something which has been known to occur in Catholic schools). 


• The recommendation seems theologically dubious. I'm an atheist, but my understanding of Catholicism is that whether or not something is a sin is largely to do with motivation rather than action. So if you are tempted to sex but desist because it would upset God, then that's good. If you are tempted to sex but desist only because of a fear of disease, that's still a sin. The church should be teaching girls to love God so much that they won't do things that offend him, not to conform to standards of sexual behaviour out of fear. No doubt religious readers will put me right if I've misunderstood this distinction. 


• These girls are attending a Catholic school where I assume morality is
  drummed into them day and night. The bishops assume that their grasp of
  that morality is so weak that having a HPV vaccination will be
  sufficient to overturn everything they have been told about sexual
  ethics. Doesn't say much for the religious teaching in the schools, or for the intelligence of the pupils. 


• One thing Jesus really understood is that humans aren't perfect and frequently fall short of the moral standards they try to adhere to. There's a huge emphasis on forgiveness of sin in his teachings. The Bishops are in effect saying that God won't forgive you if you stray from the straight and narrow: he'll commit you to a life with an unpleasant disease, and increase your risk of dying from cancer. That's not the Christian God I was taught about.
  
  
  
  

Causal models of developmental disorders: the perils of correlational data

Experimental psychology depends heavily on statistics, but
psychologists don’t always agree about the best ways of analyzing data. Take
the following problem:

I have two groups each of 30 children, dyslexics and
controls. I give them a test of auditory discrimination and find a significant
difference between the groups, with the dyslexic mean being lower. I want to
see whether reading ability is related to the auditory task. I compute the
correlation between the auditory measure and reading, and find it is .42, which
in a sample of 64 cases is significant at the .001 level.

I write up the results, concluding that poor auditory skill
is a risk factor for poor reading. But reviewers are critical.
So what’s wrong with this?

I’ll deal quickly with two obvious points. First, there is
the well-worn phrase that correlation does not equal causation. The correlation
could reflect a causal link from auditory deficit to poor reading,
but we need also to consider other causal routes, as I’ll illustrate further
below. This is an issue about interpretation rather than data analysis.

A second point concerns the need to look at the data rather
than just computing the correlation statistic. Correlations can be sensitive to
distributional properties of the data and can be heavily influenced by
outliers. There are statistical ways of checking for such effects, but a good
first step is just plotting a scatterplot to see whether the data look orderly.
A tip for students: if your supervisor
asks to see your project data, don’t just turn up with numerical output from
the analysis: be ready to show some plots.

Figure 1: Fictitious data showing spurious correlation between height and reading ability

A less familiar point concerns the pooling of data across
the dyslexic and control groups. Some people have strong views about this, yet,
as far as I’m aware, it hasn’t been discussed much in the context of
developmental disorders. I therefore felt it would be good to give it an airing
on my blog and see what others think.

Let’s start with a fictitious example that illustrates the
dangers of pooling data from two groups. Figure 1 is a scatterplot showing the
correlation between height and reading ability in groups of 6-year-olds and
10-year-olds. If I pool across groups, I’m likely to see a strong correlation
between height and reading ability, whereas within any one age group the
correlation is negligible. This is a clear case of spurious correlation, as
illustrated in Figure 2. Here the case against pooling is unambiguous, and it's
clear that if you look at the correlation within either age band, there is no
relationship between reading ability and height.

Figure 2: Model showing how a spurious correlation between height and reading arises because both are affected by age

Examples such as this have led some people to argue that you
shouldn’t pool data in studies such as the dyslexic vs. control example. Or, to
be more precise, the recommendation is usually that you should check the
correlations within each group, and
avoid pooling if they don’t look consistent with the pooled correlation. I’ve
always been a bit uneasy about this logic and have been giving some thought as
to why.

First, there is the simple issue of power. If you halve your
sample size, then you increase the standard error of estimate for a correlation
coefficient, making it more likely that it will be nonsignificant. Figure 3
shows the 95% confidence intervals around a correlation of .5 depending on
sample size, and you can readily see that these are larger for small than big
samples. There's a nice website by Stan
Brown that gives relevant formulae in Excel.

Figure 3: 95% confidence interval around estimated correlation of .5, with different sample sizes

A less obvious point is that the data in Figure 1 look
analogous to the dyslexic vs. control example, but there is an important
difference. We know where we are with age: it is unambiguous to define and measure.
But dyslexia is more tricky. Suppose we substitute dyslexia for age, and
auditory processing for height, in the model of spurious correlation in Figure
2. We have a problem: there is no independent diagnostic test for dyslexia. It
is actually defined in terms of one of our correlated variables, reading
ability. Thus, the criterion used to allocate children to groups is not
independent of the measures that are entered into the correlation. This creates
distortions in within-group correlations, as follows.

If we define our groups in terms of their scores on one
variable, we effectively restrict the range of values obtained by each group,
and this lowers the correlation.  Furthermore, the restriction will be less for
the controls than for the dyslexic group - who are typically selected as
scoring below a low cutoff, such as one SD below the mean. Figure 4 shows simulated
data for two groups selected from a population where the true correlation
between variables A and B is .5. Thirty individuals (dyslexics) are selected as
scoring more than 1 SD below average on variable A, and another 30 (controls)
are selected as scoring above this level. 

Figure 4: Correlations obtained in samples of dyslexic (red) and controls (blue) for 20 runs of simulation with N = 30 per group.

The Figure shows correlations from twenty
runs of this simulation. For both groups, the average correlation is less than
the true value of .5, because of the restricted range of scores on variable A.
However, because the range is more restricted for the dyslexic group, their
average correlation is lower than that of the controls. A correlation of .42 corresponds to the .05 significance level for a sample of
this size, and we can see that the controls are more likely to exceed this
value than the dyslexic group. All these results are just artefacts of the way
in which the groups were selected: both groups come from the same population
where r = .5.

What can we conclude from all this? Well, the bottom line is
that if we find non-significant within-group
correlations this does not necessarily invalidate a causal model. The
simulation shows that we may find that within-group correlations look quite
different in dyslexic and control groups, even if they come from a common
distribution.

So where does this leave us?! It would seem that in general,
within-group data are unlikely to help us distinguish between causal and
non-causal models: they may be compatible with both. So how should we proceed?

There’s no simple solution, but here are some suggestions:

1. If considering correlational data, always report the 95%
confidence interval. Usually people (including me!) just report the correlation coefficient,
degrees of freedom and p-value. It’s so uncommon to add confidence intervals
that I suspect most psychologists don’t know how to compute it. Do not assume
that because one correlation is significant and another is not that they are
meaningfully different. This
website can be used to test for the significance of the difference between
correlations. I would, however, advise against interpreting such a comparison
if your data are affected by the kinds of restriction of range discussed above.

2. Study the relationship between key variables in a large unselected
sample covering a wide range of scores. This is a more tractable solution, but
is seldom done. Typically, people recruit an equivalent number of cases and
controls, with a sample size that is inadequate for getting a precise estimate
of a correlation in either group. If your underlying model predicts a linear
relationship between, say, auditory processing and phonological awareness, then
with a sample of 200 cases, a fairly precise estimate can be obtained. With this approach, one
can also identify whether the relationship is linear.

3. More generally, it’s important to be explicit about what
models you are testing. For instance, I’ve identified four underlying models of
the relationship between auditory deficit and language impairment, as shown in Figure
5. In general, correlational data on these two skills won’t distinguish between
these models, but specifying the alternatives may help you think of other data
that could be informative. 

Figure 5: Models of causal relationships underlying observed correlation between auditory deficit and language impairment

For instance:

• We
  found that, when studying heritable conditions, it is useful to include
  data on parents or siblings. Models differ in predictions about how
  measures of genetic risk - for instance, family history, or presence of
  specific genetic variants - relate to A (auditory deficit) and B (language impairment) in the child. This approach is
  illustrated in this
  paper. Interestingly, we found that the causal model that is often implicitly assumed, which we termed the Endophenotype model, did not fit the data, but nor did the spurious correlation model, which corresponds here to the Pleiotropy model.


• There
  may be other groups that can be informative: for instance, if you think
  auditory deficits are key in causing language problems, it may be worth
  including children with hearing loss in a study - see this
  paper for an example of this approach using converging evidence.


• Longitudinal
  data can help distinguish whether A causes B or B causes A.


• Training
  studies are particularly powerful, in allowing one to manipulate A and see
  if it changes B.

So what’s the bottom line? In general, correlational data
from small samples of clinical and control groups are inadequate for testing
causal models. They can lead to type I errors, where pooling data leads to a
spurious association between variables, but also to type II errors, where a
genuine association is discounted because it isn’t evident within subject
groups. For the field to move forward, we need to go beyond correlational data.



P.S. 9th July 2012

I've written a little tutorial on simulating data using R to illustrate some of these points. No prior knowledge of R required. see: http://tinyurl.com/d2868cg

Bishop DV, Hardiman MJ, & Barry JG (2012). Auditory deficit as a consequence rather than endophenotype of specific language impairment: electrophysiological evidence. PloS one, 7 (5) PMID: 22662112



If you liked this post, you may also be interested in my other posts on statistical topics:

Getting genetic effect sizes in perspective
The joys of inventing data
A short nerdy post about the use of percentiles
The difference between p < .05 and a screening test

The ‘autism epidemic’ and diagnostic substitution

Based on: King &
Bearman (2011) American Sociological Review, 76(2), 320-346; 

Data from birth
and diagnostic records for all children born in California 1992-2000

Everyone agrees there has been a remarkable increase in
autism diagnosis across the world. There is, however, considerable debate about
the reasons for this. Three very different kinds of explanation exist.

• Explanation #1 maintains that something in our modern environment has come
  along to increase the risk of autism. There are numerous candidates, as indicated in this blogpost by Emily Willingham. 


• Explanation #2 sees the risks as largely biological or genetic, with changing
  patterns of reproduction altering prevalence rates, either because of
  assortative mating (not much evidence, in my view) or because of an increase in
  older parents (more plausible). 


• Explanation #3 is very different: it says the
  increase is not a real increase - it’s just a change in what we count as
  autism. This has been termed ‘diagnostic substitution’ - the basic idea is that
  children who would previously have received another diagnosis or no diagnosis are
  now being identified with autism spectrum disorder (ASD). This could be in part
  because of new conceptualisations of autism, but may also be fuelled by
  strategic considerations: resources for children with ASD tend to be much
  better than those for children with other related conditions, such as language
  impairment or intellectual handicaps, so this diagnosis may be preferred.

In 2008, my research group published a study that
documented one kind of diagnostic substitution. We contacted people who had
taken part in our studies of children with specific language impairment years
ago. We carried out a standard diagnostic observation procedure for autism with
the young adults themselves and, where possible, interviewed their parents
about their early history. We found a number of individuals who had been
regarded as cases of specific language impairment ten or twenty years ago but who
would nowadays be diagnosed with ASD. Although it’s possible that some people
develop autistic symptomatology as they get older, in our cases the autistic
symptoms appeared to have been present from early childhood - as indicated by
the parental interviews. Around half of the sample had been identified as having
‘semantic-pragmatic disorder’ in childhood, but autism had been excluded because
at that time, prior to publication of DSM-IV diagnostic guidelines, it was
regarded as a very rare condition in which there were severe social and
behavioural impairments. How many children would have qualified for ASD
diagnoses had they been seen today? Well, it depends. I suspect few people
appreciate just how flexible the diagnostic criteria are for autism, even when lengthy standardized diagnostic instruments are used. Although we used the gold
standard diagnostic procedures (ADOS-G and ADI-R) we found they seldom gave the
same answer. If we diagnosed ASD only when both diagnostic instruments agreed,
21% of cases met criteria. If we included anyone who met criteria for autism or
PDDNOS on either ADI-R or ADOS, the rate shot up to 66%.

Last year, a fascinating study by Brugha and colleagues attacked the same question from a different angle. They did an epidemiological
survey of a representative sample of adults from the English population, using
the ADOS-G, and found that the rates of ASD were similar to those recently
reported in children. Within the adult population, rates of ASD did not change
with age. Thus, provided we stick to the same diagnostic criteria, then the
prevalence of autism is the same for those born several decades ago, as it is
for the current generation of children. Importantly, none of these adults with
ASD had received a formal diagnosis.

Recently, we conducted a study with another group: children with an additional sex chromosome (i.e. trisomy). We had not intended to study
diagnostic substitution: the goal was rather to understand more about the
language difficulties that had previously been described in children with sex
chromosome trisomies. The effect of an extra sex chromosome is relatively mild:
most of these children attend mainstream schools and they do not have any
obvious physical abnormalities. Indeed, they can be hard to study because many
individuals with trisomies will be unaware of their condition. We gathered
information by parental report, and did not do any direct evaluation of the
child, but we did ask about whether the child had had any kind of diagnosis by
a medical or psychological expert. We confirmed that there was a strong
association with language problems in all three kinds of trisomy (girls with
XXX, and boys with XYY or XXY), many of whom had had speech-language therapy.
But we also found that 2/19 (11%) of boys with XXY and 11/58 (19%) of those
with XYY had received an ASD diagnosis.

It is important to emphasise that most children with a sex
chromosome trisomy did not have an ASD diagnosis, and many were not giving any
cause for concern. Nevertheless, although they are only a minority of cases,
the proportion with ASD is much higher than in the general population. We were
really surprised at this because before publishing our study we had done a
systematic review of the literature on children with sex chromosome trisomies,
focusing on studies that avoided ascertainment bias. In these studies, not a
single case of autism had been mentioned when discussing outcomes. So was our
study a fluke? We are confident this is not the case, because this year two
further studies from the USA
have been reported (Ross et al and Lee et al, in press), both of which got results very similar to ours, though
using different methods.

This research provides further evidence that diagnostic
substitution has occurred, suggesting that children who in the past would have
been diagnosed with language impairment are now being diagnosed with ASD. The
only other way to explain the increased diagnosis rate in children with a known
chromosomal abnormality would be if the trisomy acted as a risk factor, making
children more sensitive to environmental factors that could cause autism. That’s
a possibility, but it seems more likely that cases of ASD were missed in the
past because more stringent diagnostic criteria were used, just as was found in our follow-up of children with SLI and in the epidemiological study
of adults by Brugha and colleagues.

It is becoming clear that changing diagnostic criteria,
increased awareness of ASD, and strategic use of diagnosis to gain access to
services, have had a massive effect on the numbers of children with ASD. When I
started studies in this area, I thought diagnostic substitution had happened
but I did not think it would be sufficient to explain the increase in numbers of ASD
diagnoses. But now, on the basis of studies reviewed here, I think it could be
the full story.



PS: a slightly extended version of this blogpost was featured on PLOS Blogs on 8th June 2012.



References

Bishop, D., Jacobs, P., Lachlan, K., Wellesley, D., Barnicoat, A., Boyd, P., Fryer, A., Middlemiss, P., Smithson, S., Metcalfe, K., Shears, D., Leggett, V., Nation, K., & Scerif, G. (2010). Autism, language and communication in children with sex chromosome trisomies Archives of Disease in Childhood, 96 (10), 954-959 DOI: 10.1136/adc.2009.179747

 


Bishop, D., Whitehouse, A., Watt, H., & Line, E. (2008). Autism and diagnostic substitution: evidence from a study of adults with a history of developmental language disorder Developmental Medicine & Child Neurology, 50 (5), 341-345 DOI: 10.1111/j.1469-8749.2008.02057.x
 



Brugha, T. (2011). Epidemiology of Autism Spectrum Disorders in Adults in the Community in England Archives of General Psychiatry, 68 (5) DOI: 10.1001/archgenpsychiatry.2011.38



Lee, N. R., Wallace, G. L., Adeyemi, E. I., Lopez, K. C., Blumenthal, J. D., Clasen, L. S., & Giedd, J. N. (2012, in press). Dosage effects of X and Y chromosomes on language and social functioning in children with supernumerary sex chromosome aneuploidies: Implications for idiopathic language impairment and autism spectrum disorders. Journal of Child Psychology and Psychiatry. 



Ross, J. L.,et al (2012). Behavioral and social phenotypes in boys with 47, XYY syndrome or 47, XXY Klinefelter syndrome.  Pediatrics, 129(4), 769-778. doi: 10.1542/peds.2011-0719