Profilers get the whole right, not the detail

In 2005 the behavioural advisory unit of the British National Policing Improvement Agency wrote 47 reports for ongoing investigations. Louise Almond, Laurence Alison and Louise Porter counted 805 individual claims in them and examined each one for the argument it carried with it. The result: 96 percent of the claims gave a reason. Only 34 percent had formal support — a study, a statistic, a body of data. Seventy percent were verifiable at all, but only 43 percent falsifiable, meaning that after a conviction they could be objectively established as right or wrong.

That is the soberest reckoning that can be drawn about offender profiles, and it comes not from critics of the method but from its own workshop. More than half of what stands in an advisory report cannot afterwards be held up against whether it was true.

What a profile presupposes

An offender profile is an inference run backwards: from observed actions, stable traits of an unknown person are to be inferred. For that to work, two things must hold. First, the same person must behave similarly across situations. Second, people with similar characteristics must commit similar offences — the so-called homology assumption.

In 2002 Laurence Alison, Craig Bennell, Andreas Mokros and David Ormerod called the first requirement the personality paradox of profiling: the field borrows its basic assumption from a trait psychology that has been qualifying exactly that assumption for decades. Behaviour is far less constant across situations than everyday intuition supposes — situationism is the school that drew its lesson from this. A crime scene, moreover, carries the handwriting of offender, victim, opportunity, time pressure, chance and later disturbance all at once.

The second requirement has been tested directly. Mokros and Alison took 100 British men who had raped women unknown to them, ordered their offence behaviours by similarity — reconstructed from witness statements — and compared that with their sociodemographic characteristics and previous convictions from police records. If the homology assumption holds, the two orderings of similarity ought to correspond. They did not, in a single comparison. Men who proceeded in similar ways were no more alike in age, in employment situation, in background or in criminal history. The authors do not conclude from this that profiling is impossible, but they do conclude that this assumption in its simple form supplies no foundation.

What the meta-analysis separates

In 2007 Brent Snook and colleagues presented a narrative review and a two-part meta-analysis of the published profiling literature. The review concluded that the field rests largely on everyday plausibility rather than on evidence. The two calculations are the more interesting part.

In the first, self-styled profilers and experienced investigators were compared together against control groups. They did no better on the offender’s thought processes, not on physical characteristics, not on offence behaviours, and not on social habits and history. Only on the overall judgement of offender characteristics were they narrowly ahead.

In the second the group was drawn more narrowly: self-styled profilers only. Now the picture shifted. On offence behaviours they retained no meaningful advantage — but on the overall judgement, on thought processes, on physical characteristics and on social history and habits they outperformed the comparison groups. There is, then, something to be found, and it is not nothing.

The pattern the two calculations share is narrow and revealing: on the overall judgement of offender characteristics the experts are ahead; on offence behaviours they are ahead in neither analysis. What happened at the crime scene — the very detail on which the crime novel hangs — is precisely the point at which the advantage disappears. The authors themselves stress the methodological weaknesses of the underlying studies: small samples, unclear criteria of success, missing blinding. In a follow-up paper in 2008 the same authors went further and declared the method unsuitable as an investigative tool. Their abstract names both gaps at once:

Why profiles seem convincing all the same

Many statements combine high base rates with elastic language: the offender knows the area, appears unremarkable from the outside, has difficulties in close relationships and may act again. Within the relevant group of cases, much of this applies to many suspects.

This has been measured. Alison, Matthew Smith and Keith Morgan put the same profile before police officers and forensic professionals, but paired it for one half with a genuine offender and for the other half with an invented one. Both groups took it to be accurate; over 75 percent found it at least reasonably accurate, and the ratings did not differ between the genuine and the invented offender. As with the Barnum effect, what stands out after an arrest is above all what fits; points that do not fit are reinterpreted or forgotten.

Behind this lies a familiar error of judgement. To ask how well a suspect fits the profile is to judge by similarity rather than by frequency — the representativeness heuristic in its purest form. The more colourful and coherent the description, the more convincing it seems, even though every additional detail makes it less probable.

The field’s best-known classification did not survive the same test. In 2004 David Canter and colleagues examined the FBI’s distinction between “organised” and “disorganised” offenders across 39 crime-scene features from 100 murders committed by 100 American serial offenders. Two offender types did not emerge: organised features characterise most serial killings in general, while disorganised ones are rare and form no type of their own. This archive’s piece on the serial-killer myth sets the finding out in detail.

What takes its place

It does not follow that behavioural analysis is useless. Alison and colleagues describe how, in Britain, a narrower practice grew out of the personal portrait, known there as behavioural investigative advice: linking cases by shared patterns, assessing risk, constraining the search area geographically, advising on interviews. Such questions have answers that can be checked. How exactly the advice comes about in an individual case is, by the same authors' assessment, still little understood.

That this shift achieves something is shown by the report analysis this piece began with. Almond, Alison and Porter compared the 47 reports of 2005 with an older sample of profiles that Alison and colleagues had examined in 2003. The difference was large: the newer reports qualified their claims far more clearly and laid out the material in a more orderly way and closer to its evidence. Profiling work can therefore be done better — by breaking it of the habit of portraying people.

What remains

The title of this piece is the whole reckoning: profilers get the whole right, not the detail. Where a meta-analysis grants them an advantage, it concerns a rough overall judgement about a type of offender; for age, appearance and offence behaviours it disappears. The assumption that similar offences go back to similar people has been tested on a sample of 100 offenders and was not confirmed. And more than half of what stands in real advisory reports was not phrased in a way that would have let anyone refute it afterwards.

How to recognise it in everyday life: by the direction in which a sentence points. “The offender probably lives in the region” is not information if it holds for almost everyone who could come into question; it becomes interesting only with a comparison group and an increment — how much more likely is this feature in this offender than in other suspects? The best profiler therefore resembles a cautious analyst more than Sherlock Holmes. She separates trace, conjecture and test, she names alternatives, and she knows that at worst a profile does not merely err but narrows attention onto an innocent person. The crime scene does not speak. People build a narrative out of it — and must then check whether it is more than a good story.