Data from the phonics screen: a worryingly abnormal distribution

The new phonics screening test for children
has been highly controversial.  I’ve been
surprised at the amount of hostility engendered by the idea of testing
children’s knowledge of how letters and sounds go together. There’s plenty of
evidence that this is a foundational skill for reading, and poor ability to do
phonics is a good predictor of later reading problems. So while I can see there
are aspects of the implementation of the phonics screen that could be
improved,  I don’t buy arguments that it
will ‘confuse’ children, or prevent them reading for meaning.

I discovered today that some early data on
the phonics screen had recently been published by the Department for Education,
and my inner nerd was immediately stimulated to visit the website and
download the tables.  What I found was
both surprising and disturbing.

Most of the results are presented in terms
of proportions of children ‘passing’ the screen, i.e. scoring 32 or more. There
are tables showing how this proportion varies with gender, ethnic background,
language background, and provision of free school meals. But I was more
interested in raw scores: after all, a cutoff of 32 is pretty arbitrary. I
wanted to see the range and distribution of scores.  I found just one table showing the relevant
data, subdivided by gender, and I have plotted the results here.

Data from Table 4, Additional Tables 2, SFR21/2012

Department for Education (weblink above)

Those of you who are also statistics nerds
will immediately see something very odd, but other readers may need a bit more
explanation.  When you have a test like
the phonics test, where each item is scored right or wrong, and the number of
correct items is totalled up, you’d normally expect to get a continuous
distribution of scores. That is to say, the numbers of children obtaining a
given score should increase gradually up to some point corresponding to the
most typical score (the mode), and then gradually decline again. If the test is
pretty easy, you may get a ceiling effect, i.e. the mode may be at or close to
the maximum score, so you will see a peak at the right hand side of the plot,
with a long straggly tail of lower scores. 
There may also be a ‘bump’ at the left hand edge of the distribution,
corresponding to those children who can’t read at all – a so-called ‘floor’
effect.  That's evident in the scores for boys. But there's also something else. There’s a sudden upswing in the distribution, just at
the ‘pass’ mark. Okay, you might think, that’s because the clever people at the
DfE have devised the phonics test that way, so that 31 of the items are really
easy, and most children can read them, but then they suddenly get much
harder.  Well, that seems unlikely, and
it would be a rather odd way to develop a test, but it’s not impossible. The
really unbelievable bit is the distribution of scores just above and below the
cutoff. What you can see is that for both boys and girls, fewer children score
31 than 30, in contrast to the general upward trend that was seen for lower
scores. Then there’s a sudden leap , so that about five times as many children
score 32 than 31. But then there’s another dip: fewer children score 33 than
32. Overall, there’s a kind of ‘scalloped’ pattern to the distribution of
scores above 32, which is exactly the kind of distribution you’d expect if a
score of 32 was giving a kind of ‘floor effect’.  But, of course, 32 is not the test floor.

This is so striking, and so abnormal, that
I fear it provides clear-cut evidence that the data have been manipulated, so
that children whose scores would put them just one or two points below the
magic cutoff of 32 have been given the benefit of the doubt, and had their
scores nudged up above cutoff.

This is most unlikely to indicate a problem
inherent in the test itself. It looks like human bias that arises when people
know there is a cutoff and, for whatever reason, are reluctant to have children
score below that cutoff.  As one who is basically in favour of phonics testing, I’m sorry to put another cat among the
educational pigeons, but on the basis of this evidence, I do query whether
these data can be trusted.

Communicating science in the age of the internet

© www.CartoonStock.com

Here's an interesting test for those on Twitter. You see a
tweet giving a link to an interesting topic. You click on the link and see it's
a YouTube piece. Do you (a) feel pleased that it's something you can watch or
(b) immediately lose interest. The answer is likely to depend on content, but
also on how long it is. Typically, if I see a video is longer than 3 minutes,
I'll give up unless it looks super-interesting.

Test #2 is for those of you who are scientists. You have to
give a presentation about a recent piece of work to a non-specialist audience.
How long do you think you will need? (a) one hour; (b) 20 minutes; (c) 10
minutes; (d) 3 minutes.

If you're anything like me, there's a disconnect between
your reactions to these different scenarios. The time you feel you need to
communicate to an audience is much greater than the time you are willing to
spend watching others. Obviously, it's not a totally fair comparison: I'm
willing to spend up to an hour listening to a good lecture (but
no more!); though to tell the truth, it's an unusual lecturer who can keep
me interested for the whole duration.

Those who use the internet to communicate science have
learned that the traditional modes of academic communication are hopelessly
ill-suited for drawing in a wider audience. TED
talks have been a remarkably successful phenomenon, and are a million miles
from the normal academic lecture: the ones I've seen are typically no longer than
15 minutes and make minimal use of visual aids. The number of site visits for
TED talks is astronomically higher than, for instance, Cambridge University's
archive of Film Interviews
With Leading Thinkers, where Aaron Klug has had around 300 hits in just
over one year, and Fred Sanger a mere 148. The reason is easy to guess: many of
these Cambridge interviews last two hours or more. They constitute priceless archive
material, and a wealth of insights into the influences that shape great academic minds, but they aren't suited to the casual viewer.

For most academics, though, shorter pieces pose a dilemma:
they don't allow you to present the evidence for what you are saying. I felt
this keenly when viewing a TED
talk by autism expert Ami Klin. At 22 minutes, this was rather longer than
the usual TED talk, but Klin is an engaging speaker, and he held my attention
for the whole time. As I listened, though, I became increasingly uneasy. He was
making some pretty dramatic claims. Specifically, as the accompanying blurb
stated: "Ami Klin describes a new early detection method that uses eye-tracking
technologies to gauge babies' social engagement skills and reliably measure
their risk of developing autism". I was very surprised at the claims made
for eye-tracking, and the data shown in the presentation were unconvincing. More
generally, Klin talked about universal screening for 6-month-olds, but I was not sure that he understood the
requirements for an effective screening test. After the end of the talk I
checked out Klin's publications on Web of Science and couldn't find any
published papers that gave a fuller picture to back up this claim. I asked my
colleagues who work in autism and none of them was aware of such evidence. I
emailed Klin last week to ask if he can point me to relevant sources but so far
I've not had a reply. (If I do, I'll add the information). At the time of writing, his talk has had over 132,000
views.

So we have a dilemma here. Nearly everyone agrees that
scientists should engage with audiences beyond their traditional narrow
academic confines. But the usual academic lecture, saturated with PowerPoint
explaining and justifying every statement, is ill-suited to such an audience.
However, if we reduce our communications to the bottom line, then the audience
has to take a lot on trust. It may be impossible to judge whether the
speaker is expressing an accepted mainstream view. If, as in the Klin case, the
speaker is both famous and charismatic, then it's unlikely that a general
audience will realise that many experts in his field would want to see a lot more hard evidence before accepting what he was saying.

I've been brooding about this issue because I've recently
joined up with some colleagues in a web-based campaign to raise awareness of
language impairments in children. My initial idea was that we'd post lectures
by experts, attempting to explain what we know about the nature, causes, and
impacts of language impairments. Fortunately, we were dissuaded from this idea
by our friends in TeamSpirit, a public relations company who have come on board
to help us get launched. With their assistance, we've posted several videos and
worked out a clearer idea of what our
YouTube channel should do. We will have professionally produced films that feature
the experiences of young people with language impairments and their families,
as well as the professionals working with them. But we also wanted to ensure
that the material we put out was evidence-based, and to include some pieces on
issues where there were relevant research findings. We were advised that any
piece by a talking academic head should be no more than 3 minutes long. I could
see the wisdom of that, given my own reactions to longer video pieces. But I
was uncomfortable. In 3 minutes, it's impossible to do more than give a bottom
line. I didn't want people to have to take what I said on trust: I wanted them
to have access to the evidence behind it. Well, we're now experimenting with an
approach that I think may work to keep everyone happy. Our academic-style talks
will stick to the 3 minute limit, but will be associated with a link to a PowerPoint
presentation which will give a fuller account. This is still shorter than the
usual academic talk - we aim for around 15-20 slides, all of which should be
self-explanatory without needing an oral narrative. And, crucially, the PowerPoint
will include references to peer-reviewed research to support what is said, and
will include a link to a reference list, including where possible a review
article. I anticipate that most people who visit our YouTube site will only get
as far as the 3 minute video. That's absolutely fine - after all, only a small
proportion of potential visitors will be evidence geeks. But, importantly, the
evidence will be there for those who want it. The PowerPoint will give the bare
bones, and the references will allow people to track back to the original
sources.

We live in exciting times, where it has become remarkably
easy to harness the power of the internet to disseminate research. The
challenge is to do so in a way that is effective while preserving academic
rigour.