Have we become slower and dumber?

Guest post by Patrick Rabbitt

http://www.flickr.com/photos/sciencemuseum/3321607591/

This week, a paper by Woodley et al (2013) was widely quoted in the media (e.g. Daily Mail, Telegraph). The authors dramatically announced that the average intelligence of populations of Western industrialised societies has fallen since the Victorian era. This is provocative because previous analyses of large archived datasets of intelligence tests scores by Flynn and others show the opposite. However, Woodley et al did not examine average intelligence test scores obtained from different generations. They compared 16 sets of data from Simple Reaction - Time (SRT) experiments made on groups of people at various times between 1884 and 2002. In all of  these experiments volunteers responded to a single light signal by pressing a single response key.  Data for women are incomplete but averages of  SRTs for men increase significantly with year of testing.  Because Woodley et al regard SRTs as good inverse proxy measures for intelligence test scores, which are in some senses “purer” measures of intelligence than pencil and paper tests, they concluded that more recent samples are less intelligent than earlier ones



Throughout their paper the authors argue that higher intelligence of persons alive during the Victorian era can explain why their creativity and achievements were markedly greater than for later, duller generations. We can leave aside an important question whether there is any sound evidence that creativity and intellectual achievements have declined since a Great Victorian Flowering because only two of  the 16 datasets they compared were collected before Victoria’s death in 1901. The remaining 14 datasets date between 1941 and 2004 and, of these, only four were collected before 1970. So most of the studies analysed were made within my personal working lifespan. This provokes both nostalgia and distrust. Between 1959 and 2004 I collected  reaction times (RTs) from many large samples of people but it would make no sense for me to compare absolute values of group mean RTs that I obtained before and after 1975. This was because, until  1975, like nearly all of my colleagues, the only apparatus I had were Dekatron counters, the Birren Psychomet or SPARTA apparatus, none of which measured intervals shorter than 100 msec. Consequently, when my apparatus gave a reading of 200 msec. the actual Reaction Time might be anywhere between 200 and 299 msec. Like most of my colleagues I always computed and published mean RTs to three decimal places, but this was pretentious because all the  RTs  I had collected had been, in effect, rounded down by my equipment. After 1975, easier access to computers and better programs gradually began to allow true millisecond resolution. More investigators took advantage of new equipment and our reports of millisecond averages became less misleading. I am unsurprised that mean RTs computed from post-1975 data were consistently, and significantly longer than those for pre-1975 data.



Changes in recording accuracy are a sufficient reason to withold excitement at Woodley et al’s comparison. It is worth noticing that different  methodological issues also make it tricky to compare absolute values for means of RTs that were collected at different times and so with different kinds of equipment. For example RTs are affected by differences in signal visibility and rise-times to maximum brightness between tungsten lamps, computer monitor displays, neon bulbs and LCDs. The stiffness and “throw” of response buttons will also have varied between the set-ups that investigators used. When comparing absolute values of SRTs, another important factor is whether or not each signal to respond is preceded by a warning signal, whether the periods between warning signals and response signals are constant or variable and just how long they are (intervals between, approximately, 200 and 800 ms allow faster RTs than shorter or longer ones) Knowing  these methodological  quirks makes us realise that, in marked contrast to intelligence tests, methodologies for measuring RT have been thoroughly explored but never standardised.



So I do not yet believe that Wooley et al’s analyses show that psychologists of my generation were probably (once!) smarter than our young colleagues (now) are. This seems unlikely, but perhaps if I read further publications by these industrious investigators  I may become convinced that this is really the case. 





References

Flynn, J. R. (1987). Massive IQ gains in 14 nations - what IQ tests really measure. Psychological Bulletin, 101(2), 171-191. doi: 10.1037/0033-2909.101.2.171
Michael A. Woodley, Jan te Nijenhuis, & Raegan Murphy (2013). Were the Victorians cleverer than us? The decline in general intelligence
estimated from a meta-analysis of the slowing of simple reaction time Intelligence : http://dx.doi.org/10.1016/j.intell.2013.04.006



POST SCRIPT, 24th May 2013



Dr Woodley has published a response to my critique on James Thompson's blog. He asks me to answer. I am glad to do so. Sluggishness has been due only to the pleasure of reading the many articles to which Woodley drew my attention. Dorothy’s remorseless archaeology of this trove, summarised in the table below, has provoked much domestic merriment during the past few days. We are grateful to Dr Woodley for this diversion. Here are my thoughts on his comments on my post.

Woodley et al used data from a meta-analysis by Silverman (2010). I am grateful to Prof Silverman for very rapid access to his paper in which he compared average times to make a single response to a light signal from large samples in Francis Galton's anthropometric laboratories and from several later, smaller samples dating from 1941 to 2006. To these Woodley et al added a dataset from Helen Bradford Thompson's 1903 monograph "The mental traits of sex".

As Silverman (2010) trenchantly points out there is a limit to possible comparisons from these datasets,: “In principle, it would be possible to uncover the rate at which RT increased (since the Galton studies) by controlling for potentially confounding variables in a multiple regression analysis. However, this requires that each of these variables be represented by multiple data points, but this requirement cannot be met by the present dataset. Accurately describing change over time also requires that both ends of the temporal dimension be well represented in the dataset and that the dataset be free of outliers (Cohen, Cohen, West, & Aiken, 2003); neither of these requirements can be met …… Thus, it is important to reiterate that the purpose … is not to show that RT has changed according to a specific function over time but rather to show that modern studies have obtained RTs that are far longer than those obtained by Galton."

Neither Silverman nor Woodley et al seem much concerned that results of comparisons might depend on differences between studies in apparatus and methods, which are shown here, together with temporal resolution where reported. 

Since Galton's dataset is the key baseline for the conclusion that population mean RT is increasing, it is worth considering details of his equipment described here and in a wonderful archival paper “Galton’s Data a Century Later” by Johnson et al (1985): “……during its descent the pendulum gives a sight-signal by brushing against a very light and small mirror which reflects a light off or onto a screen, or, on the other hand, it gives a sound-signal by a light weight being thrown off the pendulum by impact with a hollow box. The position of the pendulum at either of these occurrences is known. The position of the pendulum when the response is made is obtained by means of a thread stretched parallel to the axis of the pendulum by two elastic bands one above and one below, the thread being in a plane through the axes of the pendulum, perpendicular to the plane of the pendulum's motion. This thread moves freely between two parallel bars in a horizontal plane, and the pressing of a key causes the horizontal bars to clamp the thread. Thus the clamped thread gives the position of the pendulum on striking the key. The elastic bands provide for the pendulum not being suddenly checked on the clamping. The horizontal bars are just below a horizontal scale, 800 mm. below the point of suspension of the pendulum. Galton provided a table for reading off the distance along the scale from the vertical position of the pendulum in terms of the time taken from the vertical position to the position in which the thread is clamped." (p. 347).

Contemporary journal referees would press authors for reassurance that the apparatus produced consistent values over trials and had no bias to over or underestimate. Obviously this would have been very difficult for Galton to achieve.

In my earlier post I noted that over the mid-to late 20th century it became obvious that to report reaction times (RT) to three decimal places is misleading if equipment only allows centi-second recording. In the latter case a reading of 200 ms will remain until a further 100 ms have elapsed, effectively "rounding down" the RT. Woodley argues that we cannot assume that rounding down occurred. I do not follow his reasoning on this point. He also offers a statistical analysis to confirm that if the temporal resolution of the measure is the only difference between studies, this would not systematically underestimate RT. Disagreement on whether rounding occurred may only be resolved with empirical data comparing recorded and true RTs between equipments.

A general concern with comparisons of RTs between studies is that they are significantly affected by the methodology and apparatus used to collect them. This is not only due to differences in resolution but can lead to systematic bias in timing of trials. For a comprehensive account of how minor differences between different 21st century computers and commercial software can flaw comparisons between studies see Plant and Quinlan (2013), who write: "All that we can state with absolute certainty is that all studies are likely to suffer from varying degrees of presentation, synchronization, and response time errors if timing error was not specifically controlled for." I earlier suggested that apparently trivial procedural details can markedly affect RTs. Among these are whether or not participants are given warning signals, whether the intervals between warning signals and response signals are constant or vary across trials and how long these intervals are, the brightness of signal lamps and the mechanical properties of response keys. A further point also turns out to be relevant to assessment of Woodley et al's argument: average values will also depend on the number of trials recorded, and averaged, for each person, and whether outliers are excluded. Note, for instance, that the equipment used in the studies by Deary and Der, though appropriate for the comparisons that they made and reported, did not record RTs for individual trials but an averaged RT for an entire session. This makes it impossible to exclude outliers, as is normal good practice. The point is that comparisons that are satisfactory within the context of a single well-run experiment may be seriously misleading if made between equally scrupulous experiments using different apparatus and procedures. Johnson et al (1985) and Silverman (2010) stress that Galton’s data were wonderfully internally consistent. This reassures us that equipment and methods were well standardised within his own study. It cannot give any assurance that his data can be sensibly comparable with those obtained with other very diverse equipments and methodologies.

Another excellent feature of the Galton dataset is that re-testing of part of his large initial sample allowed estimates of reliability of his measures. With his large sample sizes even low values of test/re-test correlations were better than chance. Nevertheless it is interesting that the test-retest correlation for visual RT, at .17, on which Silverman’s and Woodley’s conclusions depends, was lower than the next lowest (high frequency auditory acuity,.28), or Snellen eye-chart (.58) and visual acuity (.76 to.79) (Johnson et al, 1985, Table 2).

We do not know whether warning signals were used in Galton's RT studies, or, if so, how long the preparatory intervals between warning and response signals might have been. Silverman (2010) had earlier acknowledged that preparatory interval duration might be an issue but felt that he could ignore it because a report by Teichner of Wundt’s discovery that fore-period duration effects could not be independently substantiated and also because he accepted Seashore et al ‘s (1941) reassurance that there are no effects on RT of fore-period duration.

Ever since a convincing study by Klemmer (1957) it has been recognised that the durations of preparatory intervals do significantly affect reaction times, that the effects of fore-period variation are large and that results cannot be usefully compared unless these factors are taken into consideration. Indeed during the 1960s fore-period effects were the staple topic of a veritable academic industry (see review by Niemi and Naatanen, 1981, and far too many other papers by Bertelson, Nickerson, Sanders, Rabbitt etc. etc). In this context Seashore et al’s (1941) failure to find for-period effects does not increase our confidence in their results as one of the data points on which Woodley et al’s analysis is based.

Silverman’s lack of interest in fore-period duration was also heightened by Johnson et al’s (1985) comment that, as far as they were able to discover, each of Galton’s volunteers was only given one trial. Silverman implies that if each of Galton’s volunteers only recorded a single RT, variations in preparatory intervals are hardly an issue. It is also arguable that this relaxed procedure might have lengthened rather than shortened RTs. Well… Yes and No. First, it would be nice to know just how volunteers were alerted that their single trial was imminent? By a nod or a wink? A friendly pat on the shoulder? A verbal “Ready”? Second, an important point of using warning signals, and of recording many rather than just one trial is that the first thing that all of us who run simple RT Experiments discover is that volunteers are very prone to “jump the gun” and begin to respond before any signal appears, so recording absurdly fast “RTs” that can be as low as 10 to 60 ms. 20th and 21st century investigators are usually (!) careful to identify and exclude such observations. Many also edit out what they regard as implausibly slow responses. We do not know whether or how either kind of editing occurred in the Galton laboratories. Many participants would have jumped the gun and if this was their sole recorded reaction the effects on group means would have been considerable. If Galton’s staff did edit RTs, both acceptance of impulsive responses or dismissal of very slow responses would reduce means and favour the idea of “Speedy Victorians”.

I would like to stress that my concerns are methodological rather than dogmatic. Investigators of reaction times try to test models for information processing by making small changes in single variables in tasks run on the same apparatus and with exactly the same procedures. This makes us wary of conclusions from comparisons between datasets collected with wildly different equipments, procedures and groups of people. My concerns were shared by some of those whose data are used by Silverman and Woodley et al. For example, the operational guide for the Datico Terry 84 device used by Anger et al states that "A single device has been chosen because it is very difficult to compare reaction time data from different test devices".

Because I have spent most of my working life using RTs to compare the mental abilities of people of different ages I am very much in favour of using RT measurements as a research tool for individual differences. (For my personal interpretation of the relationships between people’s calendar ages and gross brain status and their performance on measures of mental speed, of fluid intelligence, of executive function, and of memory see e.g. Rabbitt et al, 2007). I also strongly believe that mining archived data is a very valuable scientific endeavour and becomes more valuable as the volume of available data exponentially increases. For example, Flynn’s dogged analyses of archived intelligence test scores show that data mining has raised provocative and surprising questions. I also believe, with Silverman, that large population studies provide good epidemiological evidence of the effects of changes in incidence of malnutrition or of misuse of pesticides or antibiotics. I am more amused than concerned when, in line with Galton’s strange eugenic obsessions, they are also discussed as potential illustrations of growing degeneracy of our species due to increased survival odds for the biologically unfit. As I noted in my original post, my only concern is that it is a time-wasting mistake to uncritically treat measurements of Reaction Times as being, in some sense, “purer”, more direct and more trustworthy indices of individual differences than other measures such as intelligence tests. Of course RTs can be sensitive and reliable measures of individual differences but, as things stand, equipments and procedures are not standardised and, because RTs are liable to many methodological quirks, we obtain widely different mean values from different population samples even from apparently very similar tasks.

The academic backlog

Photo from http://www.pa-legion.com

Here’s an interesting
question to ask any scientist: If you were to receive no more research funding,
and just focus on writing up the data you have, how long would it take? The
answer tends to go up with seniority, but a typical answer is 3 to 5 years.

I don’t have any hard
data on this – just my own experience and that of colleagues – and I suspect it
varies from discipline to discipline. But my impression is that people generally
agree that the academic backlog is a real phenomenon, but they disagree on
whether it matters.

One view is that
completed but unpublished research is not important, because there’s a kind of
“survival of the fittest” of results. You focus on the most interesting and
novel findings, and forget about the rest. It’s true that we’ve all done failed studies with inconclusive results,
and it would be foolish trying to turn such sow’s ears into silk purses.  But I suspect there’s a large swathe of
research that doesn’t fall into that category, but still never gets written up.  Is that right, given the time and money that
have been expended in gathering data? Indeed, in clinical fields, it’s not only
researchers putting effort into the research – there are also human participants
who typically volunteer for studies on the assumption that the research will be
published.

I’m not talking about
research that fails to get published because it’s rejected by journal editors,
but rather about studies that don’t get to the point of being written up for
publication. Interest in this topic has been stimulated by Ben Goldacre’s book
Bad Pharma, which has highlighted the numerous clinical trials that go
unreported – often because they have negative results. In that case the concern
is that findings are suppressed because they conflict with the financial
interests of those doing the research, and the Alltrials campaign is doing a sterling job to tackle that issue. But beyond the field of clinical trials, there’s still a
backlog, even for those of us working in areas where financial interests are
not an issue.



It’s worth pausing to
consider why this is so. I think it’s all to do with the incentive structure of
academia. If you want to make your way in the scientific world, there are two
important things you have to do: get grant funding and publish papers. This
creates an optimisation problem, because both of these activities take time,
and time is in short supply for the average academic.  It’s impossible to say how long it takes to
write a paper, because it will depend on the complexity of the data, and will
vary from one subject area to the next, but it’s not something that should be
rushed. A good scientist checks everything thoroughly, thinks hard about
alternative interpretations of results, and relates findings to the existing research
literature. But if you take too much time, you’re at risk of being seen as
unproductive, especially if you aren’t bringing in grant income. So you have to
apply for grants, and having done so, you have then to do the research that you
said you’d do. You may also be under pressure to apply for grants to keep your
research group going, or to fund your own salary.



When
I started in research, a junior person would be happy to have one grant, but that
was before the REF. Nowadays  heads of
department  will encourage their staff to
apply for numerous grants, and it’s commonplace for senior investigators have
several active grants, with estimates of around 1-2 hours per week spent on
each one. Of course, time isn’t neatly divided up, and it’s more likely that
the investigator will get the project up and running and then delegate it to
junior staff, then putting in additional hours at the end of the project when
it’s time to analyse and write up the data. The bulk of the day-to-day work will
be done by postdocs or graduate students, and it can be a good training
opportunity for them. All the same, it’s often the case that the amount of time
specified by senior investigators is absurdly unrealistic. Yet this approach is encouraged: I doubt anyone ever questions a senior
investigator’s time commitment when evaluating a grant, few funding bodies
check whether you’ve done what you said you’d do, and even if they do, I’ve
never heard of a funder demanding that a previous project be written up before
they’ll consider a new funding application.

I don’t think the
research community is particularly happy about this: many people have a sense
of guilt at the backlog, but they feel they have no option. So the current
system creates stress as well as inefficiency and waste. I’m not sure what the
solution is, but I think this is something that research funders should start
thinking about. We need to change the incentives to allow people time to think.
I don’t believe anyone goes into science because they want to become rich and
famous: we go into it because we are excited by ideas and want to discover new
things. But just as bankers seem to get into a spiral of greed whereby they
want higher and higher bonuses, it’s easy to get swept up in the need to prove
yourself by getting more and more grants, and to lose sight of the whole
purpose of the exercise – which should be to do good, thoughtful science.  We won’t get the right people staying in the
field if we value people solely in terms of research income, rather than in
terms of whether they use that income efficiently and effectively.

Forget the Tower of Hanoi: A new ecologically valid test of executive function

Here’s a problem I encountered the other day.



I’d left my keys at home (a bad start). But I’ve learned to arrange life to allow for cognitive failures, and I knew there was a spare key in the keybox in my PA’s office.



My PA was on holiday, and her door was locked, but I was not foiled by this because there’s a spare key to her room in another keybox (#2) outside her door.



Both keyboxes have numeric codes, but codes for the keyboxes have been set to be memorable, even by someone with a barely functioning hippocampus.



So I managed to get into my PA’s office, extract key for my office, lock up her office, enter my office, and do a full day’s work.



It was going in the other direction, restoring my office key to the keybox, that proved challenging.



Before you read on, you might want to give your frontal lobes a little work-out: what should the correct sequence of events be?



Well, here’s what I did:



1.    Go down corridor to keybox to retrieve key to PA’s office. Unable to read the number codes without spectacles.

2.    Return to my office to get spectacles. Open keybox. Release key to PA’s office. Enter office.

3.    Open keybox #1.

4.    Put key in keybox and lock keybox.

5.    Find myself outside door of PA office attempting to lock the door with wrong key .

6.    Re-open keybox #1 and extract key to PA office.

7.    Put my office key in keybox #1 and lock keybox.

8.    Lock door to PA office.

9.    Return key to PA office to keybox #2.

10.    Return to my office to pack up computer and stuff, only to realise….

          I had failed to lock my office door.

DOH!

But to look on the bright side: I realise I’ve discovered a real-life version of the Tower of Hanoi.

A short rant about numbered journal references

Well, I had not planned a blogpost this morning, but I am goaded to do so by growing frustration as I attempted to read a review that was published in Trends in Neurosciences. The authors were proposing a novel theory in a controversial area, and came out with definitive statements, such as “Imaging studies have revealed the involvement of brain regions including the basal ganglia and cerebellum”. Quite appropriately, they supported such assertions with references. But the journal format meant that these references were identified only by numbers, so I had to flip to the reference list at the back of the paper to find out what was being referred to. The paper had over 70 references, and in the course of reading it, I reckon I did this about 20 times. I would have done it more, but it was distracting and I had to put down my mug of tea each time while I hunted for the relevant page. If I forgot the number on the way (a not unusual occurrence at my age), I had to go through this loop again.



Needless to say, TINS is not alone in adopting this referencing format: it is especially common in journals that have strict length limits on papers. I recently had a paper accepted for such a journal, and I found that any mention of the authors of a referenced work was frowned upon, and I was required to rewrite so that the only identifier of a reference was the number.This often meant turning a short active sentence (Jones [54] showed that.X..) into a longer passive one ( X was found by another study ... [54]).



Why does this matter? Well, if the reader knows the topic pretty well, the referencing will indicate whether the author knows the relevant literature and whether the evidence that is cited is balanced or involves cherrypicking. References also give clues as to where to allocate one's attention. If I see a set of statements associated with a list of familiar names,  I can read more casually because I already know the subject matter; if there is reference to novel material, I’ll focus in more depth. Maybe this is just an obsessive tendency in me, but I want to know what is being referred to, and I find that numbered referencing impairs the readability of an article. Journals may save a bit of space by using numbers rather than names, but as more and more articles move to on-line only, this is not a major consideration, and what is gained in space is at the expense of the reader experience. Hrrmph!



P.S. 13.52 5/4/13

Thanks to Neil Martin (@ThatNeilMartin) for drawing my attention to this v. relevant article. Might explain why psychology journals, whose editors understand how cognition works, tend to prefer the non-numbered system.

Clauss, M., Müller, D., & Codron, D. (2013). Source References and the Scientist's Mind-Map: Harvard vs. Vancouver Style Journal of Scholarly Publishing, 44 (3), 274-282 DOI: 10.3138/jsp.44.3.005

Copy of the article available from the 1st author: mclauss@vetclinics.uzh.ch 

Blogging as post-publication peer review: reasonable or unfair?

In
a previous blogpost, I criticised a recent paper claiming that playing action
video games improved reading in dyslexics. In a series of comments below the
blogpost, two of the authors, Andrea Facoetti and Simone Gori, have responded
to my criticisms. I thank them for taking the trouble to spell out their views
and giving readers the opportunity to see another point of view. I am, however,
not persuaded by their arguments, which make two main points. First, that their
study was not methodologically weak and so Current Biology was right to publish
it, and second, that it is unfair, and indeed unethical, to criticise a
scientific paper in a blog, rather than through the regular scientific
channels.

Regarding the study
methodology, as noted above, the principal problem with the study by
Franceschini et al was that it was underpowered, with just 10 participants per
group.  The authors reply with an
argument ad populum, i.e. many other studies have used equally small samples.
This is undoubtedly true, but it doesn’t make it right. They dismiss the paper
I cited by Christley (2010) on the grounds that it was published in a low
impact journal. But the serious drawbacks of underpowered studies have been
known about for years, and written about in high- as well as low-impact
journals (see references below).

The response by Facoetti
and Gori illustrates the problem I had highlighted. In effect, they are saying
that we should believe their result because it appeared in a high-impact
journal, and now that it is published, the onus must be on other people to
demonstrate that it is wrong. I can appreciate that it must be deeply
irritating for them to have me expressing doubt about the replicability of
their result, given that their paper passed peer review in a major journal and
the results reach conventional levels of statistical significance. But in the
field of clinical trials, the non-replicability of large initial effects from
small trials has been demonstrated on numerous occasions, using empirical data
- see in particular the work of Ioannidis, referenced below. The reasons for
this ‘winner’s curse’ have been much discussed, but its reality is not in
doubt. This is why I maintain that the paper would not have been published if
it had been reviewed by scientists who had expertise in clinical trials
methodology. They would have demanded more evidence than this.

The response by the
authors highlights another issue: now that the paper has been published, the
expectation is that anyone who has doubts, such as me, should be responsible
for checking the veracity of the findings. As we say in Britain, I should put
up or shut up. Indeed, I could try to get a research grant to do a further
study. However, I would probably not be allowed by my local ethics committee to
do one on such a small sample and it might take a year or so to do, and would
distract me from my other research. Given that I have reservations about the
likelihood of a positive result, this is not an attractive option. My view is
that journal editors should have recognised this as a pilot study and asked the
authors to do a more extensive replication, rather than dashing into print on
the basis of such slender evidence. In publishing this study, Current Biology
has created a situation where other scientists must now spend time and
resources to establish whether the results hold up.

To establish just how
damaging this can be, consider the case of the FastForword intervention,
developed on the basis of a small trial initially reported in Science in 1996.
After the Science paper, the authors went directly into commercialization of
the intervention, and reported only uncontrolled trials. It took until 2010 for
there to be enough reasonably-sized independent randomized controlled trials to
evaluate the intervention properly in a meta-analysis, at which point it was
concluded that it had no beneficial effect. By this time, tens of thousands of
children had been through the intervention, and hundreds of thousands of
research dollars had been spent on studies evaluating FastForword.

I appreciate that those
reporting exciting findings from small trials are motivated by the best of
intentions – to tell the world about something that seems to help children. But
the reality is that, if the initial trial is not adequately powered, it can be
detrimental both to science and to the children it is designed to help, by
giving such an imprecise and uncertain estimate of the effectiveness of
treatment.

Finally, a comment on
whether it is fair to comment on a research article in a blog, rather than
going through the usual procedure of submitting an article to a journal and
having it peer-reviewed prior to publication. The authors’ reactions to my
blogpost are reminiscent of Felicia Wolfe-Simon’s response to blog-based
criticisms of a paper she published in Science: "The items you are
presenting do not represent the proper way to engage in a scientific
discourse”. Unlike Wolfe-Simon, who simply refused to engage with bloggers,
Facoetti and Gori show willingness to discuss matters further, and present
their side of the story, but they nevertheless it is clear they do not regard a
blog as an appropriate place to debate scientific studies. 

I could not disagree
more. As was readily demonstrated in the Wolfe-Simon case, what has come to be
known as ‘post-publication peer review’ via the blogosphere can allow for new
research to be rapidly discussed and debated in a way that would be quite
impossible via traditional journal publishing. In addition, it brings the
debate to the attention of a much wider readership. Facoetti and Gori feel I
have picked on them unfairly: in fact, I found out about their paper because I
was asked for my opinion by practitioners who worked with dyslexic children.
They felt the results from the Current Biology study sounded too good to be
true, but they could not access the paper from behind its paywall, and in any
case they felt unable to evaluate it properly. I don’t enjoy criticising
colleagues, but I feel that it is entirely proper for me to put my opinion out
in the public domain, so that this broader readership can hear a different
perspective from those put out in the press releases. And the value of blogging
is that it does allow for immediate reaction, both positive and negative. I
don’t censor comments, provided they are polite and on-topic, so my readers
have the opportunity to read the reaction of Facoetti and Gori. 

I should emphasise that I
do not have any personal axe to grind with the study's authors, who I do not
know personally. I’d be happy to revise my opinion if convincing arguments are
put forward, but I think it is important that this discussion takes place in
the public domain, because the issues it raises go well beyond this specific
study.

References

Button, K. S., Ioannidis,
J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., &
Munafo, M. R. (2013). Power failure: why small sample size undermines the
reliability of neuroscience. Nature Reviews Neuroscience, advance online publication.
doi: 10.1038/nrn3475

Ioannidis, J. P. A. (2005).
Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:
10.1371/journal.pmed.0020124

Ioannidis, J. P. (2008).
Why most discovered true associations are inflated. Epidemiology 19(5),
640-648.

Ioannidis JP, Pereira TV,
& Horwitz RI (2013). Emergence of large treatment effects from small
trials--reply. JAMA : the journal of the American Medical Association, 309 (8),
768-9 PMID: 23443435

Ten things than can sink a grant proposal: Advice for a young psychologist

 © cartoonstock.com

So you’ve slaved away for weeks giving up any semblance of social or family life in order to put your best ideas on paper. The grant proposal disappears into the void for months during which your mental state oscillates between optimistic fantasies of the scientific glory that will result when your research is funded, and despair and anxiety at the prospect of rejection. And then it comes: the email of doom: “We regret that your application was not successful.” Sometimes just a bald statement, and sometimes embellished with reviewer comments and ratings that induce either rage or depression, depending on your personality type.



There are three things worth noting at this point. First, rejection is the norm: success rates vary depending on the funding scheme, but it’s common to see funding rates around 20% or less. Second, resilience in the face of rejection is a hallmark of the successful scientist, at least as important as intelligence and motivation. Third, there is a huge amount of luck in the grants process: just as with the journal peer review process, reviewers and grant panel members frequently have disparate opinions, and rejection does not mean the work is no good. However, although chance is a big factor, it's not the only thing.



This week I participated in a workshop on “How to get a grant” run by my colleague Masud Husain. We are both seasoned grant reviewers and have served on grants panels. Masud prepared some slides where he noted things that can lead to grant rejection, and I dug out an old powerpoint from a similar talk I’d given in 2005. There was remarkable convergence between the points that we highlighted, based on our experiences of seeing promising work rejected by grants panels. So it seemed worth sharing our insights with the wider world. These comments are tailored to postdocs in psychology/neuroscience in the UK, though some will have broader applicability.

1. Lack of clarity

The usual model for grant evaluation is that the proposal goes to referees with expertise in the area, and is then considered by a panel of people who cover the whole range of areas that is encompassed by the funding scheme. The panel will, of course, rely heavily on expert views, but your case can only be helped if the other panel members can understand what you want to do and why it is important. Even if they can't follow all the technical details, they should be able to follow the lay abstract and introduction.



It's crucial, therefore, that you give the draft proposal to someone who is not an expert in your research topic - preferably not a close friend, but someone more likely to be critical. Ask them to be brutally honest about the bits that they don't understand. When they give you this feedback, don't argue with them or attempt verbal explanations; just rewrite until they do understand it.

2.Badly written proposal

In an ideal world, funders should focus on the content of your proposal rather than the presentation, right? Things like spelling, formatting, and so on are trivial details that only inferior brains worry about, right?



Nope. Wrong on both counts. The people reading your grant are busy. They may have a stack of proposals to evaluate. I have, for instance, been involved in evaluating for a postdoctoral fellowship scheme where my task was to select the top five from a heap of forty odd proposals. The majority of proposals are very good, and so this is a task that is both difficult and important. You  can end up feeling like one of Pavlov's dogs forced to make ever-finer discriminations, and this can put you in a grumpy and unforgiving mood. You take a dim view of proposals where there are typos, spelling errors and missing references. I've seen grant proposals where the applicant failed to turn off 'track changes', or where 'insert reference here' is in the text. In this highly competitive context, there's a high chance that these will go on the 'reject' heap. Even if there are no errors in the text, a densely packed page of verbiage is harder for the reviewer to absorb than a well laid-out document with spacing and headings. You will usually feel that the word limit is too short, and it is tempting to pack in as many words as possible, but this is a mistake. Better ditch material than confront your reviewer with an intimidating wall of words. Judicious use of figures can make a huge difference to the readability of your text, and readability is key. I personally dislike it when numbers are used to indicate references, especially if the reference list then omits titles of referenced papers: people commonly do this to save space, but I like to be able to readily work out what references are being referred to.



Anyone can improve the presentation of a grant. Use of a spell-checker is obvious, but if possible, you should also look at examples of successful applications to see what works in terms of layout etc. You can also Google "good document layout" to find websites full of advice.

3. Boring or pointless proposal

This is a difficult one, because what one person finds riveting, another finds tedious. But if you find your proposal boring, then there's close to zero chance anyone will want to fund it. You should never submit a grant proposal unless you are genuinely excited by the work that you are proposing. You need to ask yourself "Is this what I most want to spend my life doing over the next 2-3 years?" If the answer is no, then rethink the proposal. If yes, then it's crucial to convey your enthusiasm.

4. Lack of hypotheses

This is a common reason for rejection of grant proposals. The phrase 'fishing expedition' is often used to dismiss research that involves looking at a large number of variables in an unfocussed way. As an aside, I remember an exasperated colleague saying that a fishing expedition was an entirely sensible approach if the aim was to catch fish! But funding bodies want to see clear, theoretically-driven predictions with an indication of how the research will test these. A hypothesis should have sufficient generality to be interesting, and usually will be tested by a variety of methods.  For instance, suppose I think that dyslexia may be caused by a particular kind of sensory deficit, and I plan to test children on a range of visual and auditory tasks. I could say that my hypothesis is that there will be differences between dyslexics and controls on the test battery, but this is too vague. It would be better to describe a particular hypothesis of, say visual deficit, and make predictions about the specific tasks that should show deficits. Better still one would set out a general hypothesis about links between the putative deficit and dyslexia, and specify a set of experiments that tested the predictions using a range of methods.



Also, ask yourself, is your hypothesis is falsifiable, and will it yield interesting findings even if it is rejected. If the answer is no, rethink.

5. Overambitious proposal

This is another common reason for rejection of proposals, particularly by junior applicants. In psychology, people commonly overestimate how many participants can be recruited (especially in clinical and longitudinal studies) and how much testing experimenters can do. Of course, you do sometimes see cases where the proposal does not contain enough. But that is much less common that the opposite.



If you are working with human participants, you need to demonstrate that you have thought about two things:

a) Participant recruitment

• Where will you recruit from?


• Have you liaised with referral sources?


• How many suitable people exist?


• What proportion will agree to take part?


• Overall, how many participants will you be able to include in a given period (e.g. 3 months/ 1 year)?


• Have you taken into account the time it will take to get ethics approval?


• Have you costed proposal to take into account reimbursements to partipants and travel?

b) Is your estimate of research personnel realistic?

• How long does it take to test one participant?


• Have you taken into account  the fact that researchers need to spend time on :
  
  - Scheduling appointments
  
  - Travelling
  
  - Scoring up/entering/analysing data
  
  - Doing other academic things (e.g. reading relevant literature, attending seminars)

If you are working with fancy equipment, then you need to consider things like whether you or your research staff will need training to use it, as well as availability.



For more on this, see my previous blogpost about an excellent article by Hodgson and Rollnick (1989): "More fun, less stress: How to survive in research", which details the mismatch between people's expectations of how long research takes and the reality.

6. Overoptimistic proposal

An overoptimistic proposal assumes that results will turn out in line with prediction and has no fall-back position if they don't. A proposal should tell us something useful even if the exciting predictions don't work out. You should avoid multi-stage experiments where the whole enterprise would be rendered worthless if the first experiment failed.

7. Proposal depends on untried or complex methods

You're unlikely to be funded if you propose a set of studies using a method in which you have limited experience, unless you can show that you have promising pilot data. If you do want to move in a new direction, try to link up with someone who has some expertise in it, and consider having them as a collaborator. Although funders don't want to take risk with applicants who have no experience in a new method, they do like proposals to include a training component, and for researchers to gain experience in different labs, even if just for a few months.

8. Overcosted (or undercosted) proposal

This one is easy: Ask for everything that you do need, but don't ask for things you don't need. This is not the time to smuggle in funding for that long-desired piece of equipment unless it is key to the proposal.



The committee will also be unimpressed if you ask for things the host institution should provide. But don't omit crucial equipment because of concerns about expense: just be realistic about what you need and explicitly justify everything.

9. Proposal is too risky

This is much harder one to call. Most funding bodies say they don’t want to fund predictable studies, but they are averse to research where there is high risk of nothing of interest emerging. A US study of NIH funding patterns came to the depressing conclusion that researchers who did high-impact but unconventional research often missed out on funding (Nicholson & Ioannidis, 2012). Funders often state that they like multidisciplinary research, but that runs the risk that, unless methodologically impeccable in all the areas that are covered, it will get turned down.



If you want to include a high-risk element to the proposal, take advice from a senior person whose views you trust - their reaction should give you an indication of whether to go ahead, and if so which aspects will need most justification. And if you want to include a component from a field you are not an expert in, it is vital to take advice from someone senior who does know that area.



It is usually sensible to be up-front about the risky element, and to explain why the risk is worth taking. If you are planning a high-risk project, always have a safety net - i.e. include some more conventional studies in the proposal to ensure that the whole project won't be sunk if the risky bit doesn't pan out.

10. Statistics underspecified or flawed

You need to describe the statistical analysis that you plan, even if it seems obvious to you - if only to demonstrate to the panel that you know what you are doing and have the competence to do it. If you are planning to use complex statistics, get advice from a statistician, and make it clear in the proposal that you have done so. If you don't have adequate statistical skill, consider having a statistician as consultant or collaborator on the grant. And do not neglect power analysis: underpowered studies are a common reason for grants to be rejected in biomedical areas.



Most grants panels are multidisciplinary, and there can be huge
cultural differences in statistical practices between disciplines. I've
seen cases where a geneticist has criticised a psychology project for
lack of statistical power (something geneticists are very hot on), or
where a medic criticises an experimental intervention study for not
using a randomised controlled design. Don't just propose the analysis that you usually do: find out what is best practice to ensure you won't be shot down for a non-optimal research design or analytic approach.

***********************

Finally, remember that the proposed research is one of three elements that will be assessed: the others are the candidate and the institution. There's no point in applying for a postdoctoral fellowship if you have a weak CV: you do need to have publications, preferably first-authored papers. There's a widespread view that you don't stand a chance of funding unless you have papers in high impact journals, but that's not necessarily true, especially in psychology. I'm more impressed by one or two solid first-authored papers than by a long string of publications where you are just one author among many, and (in line with Wellcome Trust policy) I don't give a hoot about journal impact factors. Most funding agencies will give you a steer on whether your CV is competitive if you ask for advice on this.



As far as the institution goes, it helps to come from a top research institution, but the key thing is to have strong institutional support, with access to the resources you need and to supportive colleagues. You will need a cover letter from your institution, and the person writing it should convey enthusiasm for your proposal and be explicit in making a commitment to providing space and other resources.



Good luck!



Reference

Nicholson JM, & Ioannidis JP (2012). Research grants: Conform and be funded. Nature, 492 (7427), 34-6 PMID: 23222591

High-impact journals: where newsworthiness trumps methodology

Here’s a paradox: Most scientists would give their eye teeth to get a paper in a high impact journal, such as Nature, Science, or Proceedings of the National Academy of Sciences. Yet these journals have had a bad press lately, with claims that the papers they publish are more likely to be retracted than papers in journals with more moderate impact factors. It’s been suggested that this is because the high impact journals treat newsworthiness as an important criterion for accepting a paper. Newsworthiness is high when a finding is both of general interest and surprising, but surprising findings have a nasty habit of being wrong.



A new slant on this topic was provided recently by a paper by Tressoldi et al (2013), who compared the statistical standards of papers in high impact journals with those of three respectable but lower-impact journals. It’s often assumed that high impact journals have a very high rejection rate because they adopt particularly rigorous standards, but this appears not to be the case. Tressoldi et al focused specifically on whether papers reported effect sizes, confidence intervals, power analysis or model-fitting. Medical journals fared much better than the others, but Science and Nature did poorly on these criteria. Certainly my own experience squares with the conclusions of Tressoldi et al (2013), as I described in the course of discussion about an earlier blogpost.



Last week a paper appeared in Current Biology (impact factor = 9.65) with the confident title: “Action video games make dyslexic children read better.” It's a classic example of a paper that is on the one hand highly newsworthy, but on the other, methodologically weak. I’m not usually a betting person, but I’d be prepared to put money on the main effect failing to replicate if the study were repeated with improved methodology. In saying this, I’m not suggesting that the authors are in any way dishonest. I have no doubt that they got the results they reported and that they genuinely believe they have discovered an important intervention for dyslexia. Furthermore, I’d be absolutely delighted to be proved wrong: There could be no better news for children with dyslexia than to find that they can overcome their difficulties by playing enjoyable computer games rather than slogging away with books. But there are good reasons to believe this is unlikely to be the case.



An interesting way to evaluate any study is to read just the Introduction and Methods, without looking at Results and Discussion. This allows you to judge whether the authors have identified an interesting question and adopted an appropriate methodology to evaluate it, without being swayed by the sexiness of the results. For the Current Biology paper, it’s not so easy to do this, because the Methods section has to be downloaded separately as Supplementary Material. (This in itself speaks volumes about the attitude of Current Biology editors to the papers they publish: Methods are seen as much less important than Results). On the basis of just Introduction and Methods, we can ask whether the paper would be publishable in a reputable journal regardless of the outcome of the study.



On the basis of that criterion, I would argue that the Current Biology paper is problematic, purely on the basis of sample size. There were 10 Italian children aged 7 to 13 years in each of two groups: one group played ‘action’ computer games and the other was a control group playing non-action games (all games from Wii's Rayman Raving Rabbids - see here for examples). Children were trained for 9 sessions of 80 minutes per day over two weeks. Unfortunately, the study was seriously underpowered. In plain language, with a sample this small, even if there is a big effect of intervention, it would be hard to detect it. Most interventions for dyslexia have small-to-moderate effects, i.e. they improve performance in the treated group by .2 to .5 standard deviations. With 10 children per group, the power is less than .2, i.e. there’s a less than one in five chance of detecting a true effect of this magnitude. In clinical trials, it is generally recommended that the sample size be set to achieve power of around .8. This is only possible with a total sample of 20 children if the true effect of intervention is enormous – i.e. around 1.2 SD, meaning there would be little overlap between the two groups’ reading scores after intervention. Before doing this study there would have been no reason to anticipate such a massive effect of this intervention, and so use of only 10 participants per group was inadequate. Indeed, in the context of clinical trials, such a study would be rejected by many ethics committees (IRBs) because it would be deemed unethical to recruit participants for a study which had such a small chance of detecting a true effect.



But, I hear you saying, this study did find a significant effect of intervention, despite being underpowered. So isn’t that all the more convincing? Sadly, the answer is no. As Christley (2010) has demonstrated, positive findings in underpowered studies are particularly likely to be false positives when they are surprising – i.e., when we have no good reason to suppose that there will be a true effect of intervention. This seems particularly pertinent in the case of the Current Biology study – if playing active computer games really does massively enhance children’s reading, we might have expected to see a dramatic improvement in reading levels in the general population in the years since such games became widely available.



The small sample size is not the only problem with the Current Biology study. There are other ways in which it departs from the usual methodological requirements of a clinical trial: it is not clear how the assignment of children to treatments was made or whether assessment was blind to treatment status, no data were provided on drop-outs, on some measures there were substantial differences in the variances of the two groups, no adjustment appears to have been made for the non-normality of some outcome measures, and a follow-up analysis was confined to six children in the intervention group. Finally, neither group showed significant improvement in reading accuracy, where scores remained 2 to 3 SD below the population mean (Tables S1 and S3): the group differences were seen only for measures of reading speed.



Will any damage be done? Probably not much – some false hopes may be raised, but the stakes are not nearly as high as they are for medical trials, where serious harm or even death can result from wrong results. There is concern, however, that quite apart from the implications for families of children with reading problems, there is another issue here, about the publication policies of high-impact journals. These journals wield immense power. It is not overstating the case to say that a person’s career may depend on having a publication in a journal like Current Biology (see this account – published, as it happens, in Current Biology!). But, as the dyslexia example illustrates, a home in a high-impact journal is no guarantee of methodological quality. Perhaps this should not surprise us: I looked at the published criteria for papers on the websites of Nature, Science, PNAS and Current Biology. None of them mentioned the need for strong methodology or replicability; all of them emphasised “importance” of the findings.



Methods are not a boring detail to be consigned to a supplement: they are crucial in evaluating research. My fear is that the primary goal of some journals is media coverage, and consequently science is being reduced to journalism, and is suffering as a consequence.



References



Brembs, B., & Munafò, M. R. (2013). Deep impact: Unintended consequences of journal rank. arXiv:1301.3748.



Christley, R. M. (2010). Power and error: increased risk of false positive results in underpowered studies. The Open Epidemiology Journal, 3, 16-19.



Halpern, S. D.,  Karlawish, J. T, & Berlin, J. A. (2002). The continuing unethical conduct of underpowered clinical trials. Journal of the American Medical Association, 288(3), 358-362. doi: 10.1001/jama.288.3.358



Lawrence, P. A. (2007). The mismeasurement of science. Current Biology, 17(15), R583-R585. doi: 10.1016/j.cub.2007.06.014



Tressoldi, P., Giofré, D., Sella, F., & Cumming, G. (2013). High Impact = High Statistical Standards? Not Necessarily So. PLoS ONE, 8 (2) DOI: 10.1371/journal.pone.0056180