Identifying the Needs of the Forensic Science Community by M.Chris Fabricant

 

On the northeast corner of the National Mall in Washington, DC, at 21201 Constitutional Avenue, next to the Albert Einstein Memorial, sits a marble edifice that house the most prestigious scientific institution in the country, the National Academy of Science, the ‘Temple of Science.’ Established by Abraham Lincoln during the Civil War to provide independent  scientific advice to the federal government, the NAS has been called to study everything from climate change to biological weapons. In the fall of 2006, the NAS was engaged by Congress to research and report on the state of forensic sciences  in the United States.

Working under the project title ‘Identifying the Needs of the Forensic Science Community,’ an all-star panel was assembled to conduct the investigation. Forensic experts from various disciplines, medical examiners, and crime lab directors were recruited for the project to represent a broad range of views, but the NAS did not define the ‘relevant scientific community the same way courts had for so long, even after Daubert.* The committee wasn’t limited to those with vested interests in particular forensic techniques: forensic guilds and law enforcement experts. Scientists from other fields, several academics, a federal prosecutor, and even a fiery New York City defense attorney were included in the committee. Former chief judge of the US Court of Appeals for the District of Columbia, Harry T. Edwards, a heavyweight legal scholar, co-chaired the committee with Dr. Constantine Gatsonis, a professor of biostatistics at Brown.

All of the most commonly used forensics in the American justice system were to be scrutinized. After nearly a century of near-unanimous acceptance by the judiciary, in the fall of 2006 the NAS began to compel forensic experts from all fields to do what  Daubert was supposed to have done in 1993: show us the science.

On April 23, 2007, inside the walnut-paneled Lecture Room of the Temple of Science, a silver-bearded dentist from San Antonio set up his PowerPoint slides. David Senn, DDS, S-ABFO (American Board of Forensic Odontology), was there to deliver the first lecture by a bite mark expert. The dentists had taken their invention a very long way, from leveraging their work identify John Does by their dental records to presenting bit mark ‘science’ in the same room President Kennedy and Obama had made speeches. But they were worried. There had been wrongful convictions. David  Wayne Pence had been executed. Their ideas had never been exposed to serious scientific scrutiny, and they were not scientists. For the most part, they were hobbies, unfamiliar with the scientific methods. Senn was one of the few exceptions.

Urbane and comfortable with scientific jargon, the Texan was the obvious choice to defend the ABFO’s  thesis.  At sixty-five, Senn was at the apex of a twenty-four-year forensic career, the manifestation of the ABFO founding father’s wildest dreams. He was the chief forensic odontologist for the Bexar County medical examiner‘s office, a Fellow of the American Academy of Forensic Sciences, editor of several thick  textbooks on forensic dentistry, and a full-time clinical professor at the University of Texas. Senn also directed the nation’s only ‘fellowship’ program for forensic odontology and presided over the annual Southwestern Symposium on Forensic Dentistry at UT. His CV weighed in at twenty-two pages.

Among the panelists for Senn’as presentation was an esteemed scientist named Dr. Karn Kafadar. Chair of the statistics department at the University of Virginia, Kafadar had also worked for NASA and the Department of Defense, and, unlike most of her NAS committee colleagues, this was not the first time she had been asked to examine the science underlying shaky forensic techniques that courts had already embraced. She as a member of the NAS panel that had debunked comparative bullet lead analysis  (CBLA)  in a report published in 2004.

The FBI claimed for decades that CBLA evidence was capable of matching a bullet found at a crime scene to a box of bullets (or the ‘molten match’) from which it originated. Typically used when traditional ballistic ‘matches’ were not available, CBLA matches were said to be based on the ‘chemical signature’ of a bullet. More specifically, plasma-atomic emission spectrometry was used to measure concentrations of seven elements in the lead alloy of a bullet to develop its ‘chemical signature’ which was used to ‘match’ the bullet to a box of bullets with identical ‘signatures’. It was something like matching a fried egg found at a crime scene to a carton of eggs in a suspects fridge. The whizbang technology was very impressive to juries, but it was essentially meaningless, since tens of thousands of bullets share the same ‘signatures.’

The debunking of CBLA evidence demonstrated to Kafadar and other mainstream scientists that just because a technique ha been developed by the FBI laboratory and accepted in court does not mean the science is sound. Outside scrutiny is warranted. That (recent) history presented a challenge to the Texan’s defense of bite mark evidence. Acceptance by the courts was the foundation of the enterprise. Apart from case law, the dentist’s credibility was grounded primarily in ’training and experience’ and identification of human remains through limited sets of dental records. David Senn’s panel, however, understood that if the method itself was invalid, it doesn’t matter how much training and experience one has in the method. Nor would the panelists be unduly impressed with all the textbooks detailing these methods. They would recognize those tomes for what they were: untested ideas, garnished with case studies rather than data. And conflating dental record recognition with bite marks would not be possible inside the Temple of Science.

Senn would have to proceed with caution.

Gamely, he began by asserting that ‘competent,’ ‘skilled’ dentists who ‘understand the scientific method .  .  . can produce profiles from bite patterns.’ Forensic odontologists ‘understand .  .  . the dynamic mechanics of biting,’ which is ‘important information that can assist judges and juries understand situations involving bite marks,’ the dentist insisted. And guidelines promulgated by the American Board of Forensic Odontology assured reliable bite mark analysis.

That was the basis of the science. The rest? ‘Art.’

It was science and art. Predictably, the NAS panelists were more interested in science and inquired where research in the field was being conducted.

‘Forensic odontology research is conducted by individual odontologists and by odontologists at institutions and universities worldwide,’ Senn maintained. A laundry list of otherwise reputable academic institutions appeared in a series of slides, including SUNY Buffalo, where the first vigorous research to test any of Senn’s bald assertions was just underway. Two years later, that research would undermine Senn’s claims; at the time, however, nothing had been published. There was no basic research at all.

Kafadar and Jude Edwards had been patient. The question remained, there in a PowerPoint slide, stripped to its bare nakedness: What is the state of the science?

‘Forensic odontology, like other forensic identification modalities, relies on education, ability, and experience of the practitioners,’ answered the dentist. He was savvy enough to understand that a couple of concessions were necessary for his credibility. There were, he admitted, some ‘major problems with bite marks: teeth haven’t been proven unique; there is no proficiency testing of practitioners’ ability to match bite marks to teeth; skin isn’t a reliable medium to record bite marks; interpreting injuries is just an opinion.

The panel asked Senn to list some research questions, some things he’d like to know about his field. A few came to mind: Are teeth unique? Can skin retain a bite mark? How an wee tell a good bite mark – one of ‘evidentiary value’- from a bad one?

Important questions. Especially if the answer matters.

Despite the ‘major problems’ and gaps of knowledge he acknowledged, Senn omitted the most damning facts that did already exist: error rates and wrongful convictions. Only a few months before his presentation, former ABFO dentist Michael Bowers published a devastating article documenting many of the known wrongful convictions attributable to bite mark evidence, plus the results of a proficiency test the dentists took showing they ‘matched’ the wrong teeth to the bite mark more often than not – a 63 percent error rate. Weeks after Bower’s piece, and just a few months before Senn’s talk, Roy Brown walked out of prison after fifteen years of wrongful imprisonment for a murder conviction that rested nearly entirely on bite marks. . .

Max Houck, a former FBI Laboratory analyst, was a logical choice to testify before the NAS committee. Like the Texas odontologist before him. Houck passed for a progressive in his field. He was willing to accept critique from the scientific community but not prepared tp ‘throw the baby of with the bathwater’ – a favorite argument  among the proponents of science for poor people.** During Houck’s time in the FBI lab, his mentors trained him on the science of ‘matching’ human hairs under a microscope; and they trained him on the art – persuading jurors the ‘match’ was strong scientific evidence that the hair had come from the defendant.

Houck was a little different, though. In the FBI world of cargo pants and golf shits, he was a plaid-blazer-and-pocket-square kind of guy. He reined in some of the more outrageous testimonial claims he was taught at the FBI lab, never used the term ‘negroid’ as a ‘scientific’ classification of human hairs, he had a stubborn streak of empiricism. He was interested in how often FBI special; agents really got it right when they ‘matched’ two hairs together. When DNA made it possible to test the accuracy of such matches in actual case work, Houck did so, and it did not go well. Experts from the world’s most prestigious crime lab were wrong more than 10 percent of the time, yet routinely testified to ‘zero error rates.’ Worse, they trained hundreds of state practitioners – practitioners like Joyce Gilchrist, whose false testimony put Curtis McCarty on death row in Oklahoma – to regurgitate FBI folklore in court rooms coast to coast.

Although he had already left the FBI lab when he testified, Houck’s fidelity to data did not put him beyond a bit of prevarication. When questioned about his own study he posited that errors made ‘matching’ the wrong hairs together were not actually ‘errors’, they simply reflected a limitation of the science. ‘ Subsequent DNA tests don’t mean that the original tests were wrong,’ Houck asserted, ‘just that a more refined test has come along.’ To make this point, he drew an analogy to- what else? – art. An expert in fine art examines three oil paintings, according to Houck’s analogy. The expert concludes all three paintings are consistent with the nineteenth-century Dutch master Vincent van Gogh’s style. Subsequently, a chemical test is performed on the paintings establishing that two of the three ‘masterpieces’ were painted after van Gogh died.  The art expert was not wrong, Houck explained, the styles were consistent.

Judge Edwards, for one, was not persuaded. ‘I don’t think your analogy holds at all,’ he said, pointing out that they had not all gathered at the National Academy of Sciences for the last two years to discuss art history. Life and liberty  were at issue. Matches that don’t match are real errors when proffered by expert witnesses in criminal trials. Dr. Channing Robertson, a professor of chemical engineering at Stanford and NAS panelist, offered Houck a competing hypothetical, placing  him in the role of of junk science consumer, rather than a fine art appraiser. .

Seated behind Houck in the Lecture Room that day in 2007 waiting to be called to the dais was a veteran fire investigator from Big Pine Key, Florida, named John Lentini. A large man with the scholarly authority of a chemist and the swagger of a firefighter, Lentini wasn’t there to defend the guild master’s wisdom. He hoped to burn it down. Like Houck, he’s initially accepted the facile logic of his field, the ‘time-tested’ guild wisdom taught by legendary experts; all intuitively appealing, like most junk sciences. Bloodletting to ‘cleanse’ bodies of various maladies, for example, had enough curbside appeal for the practice to survive for two thousand years. And today there is a billion-dollar industry devoted to ‘cleansing’ our bodies of ‘toxins’ with various diets, gemstones, ideas with roughly the same amount of empirical support as blood letting.

For bite marks and hair microscopy, DNA evidence exposed the guild techniques as junk science. But in arson cases, where high temperatures degrade biological evidence, DNA evidence was not available. Instead, arson science was debunked by a series of high-profile arson-murder prosecutions – Lentini in the middle of most of them – that fell apart when prosecution theories were tested in controlled experiments, beginning with a near-fatal capital prosecution in Jacksonville, Florida, in the early ‘90s.

In characteristically unsparing terms. Lentini began his 2007 NAS hearing presentation with sobering numbers demonstrating the urgency of rooting out junk fire science. There are ‘500,000 structure fires’ every year in the United States; 10 percent of those are determined to be ‘incendiary or suspicious. Thhat’s 50,000 chances per year to make a serious error. Even if the error rate is only 5 percent, that’s 2,500 miscalls every year.’ A 5% error rate, in Lentini’s view was ‘wildly optimistic.’

A dog perched on a witness stand appeared on the next slide. It was Blaze, K-9s ‘expert testimony.’ After running through a myriad other myths debunked in the last fifteen years, yet still leading to wrongful convictions, Lentini summarized the response to the data from the forensic guilds in a single word: ‘Whining”. He clicked to the final slide. The familiar Gothic script of the Chicago Tribune appeared. A front-page story was projected onto the Lecture Room wall, a piece by Pulitzer Prize-winning investigative journalists Maurice Possley and Steven Mills. Lentini had contributed to the journalists’ research into the story. The headline read in bold capital letters: ‘Texas Man Executed on Disproved Forensics.’

One of the last scientists to testify during the NAS hearings was Dr. Itiel Dror, an expert in cognitive bias, a field of research that had been long ignored in forensics. But it is more than the lack of scientific research that allows junk science to convict the innocent – it’s also the basic nature of human cognition, the imperceptible currents of the mind that nudge decision-making, sometimes down the wrong path.

That humans make decisions –often irrational ones- based on ‘heuristics,’ or mental shortcuts like brand loyalty, rather thhan valid predictive information, is known as ‘cognitive bias.’ This area of neuroscience did not become recognized field of scientific study until the 1970s, largely through the work of two Israeli social scientists, Amos Tversky and Daniel Kahneman, who won the Nobel Prize for their work. Since the field’s establishment, research has shown that the mental shortcuts we use to make decisions –everything from drafting  professional athletes to investing in the stock market –are often misleading, based on snap intuition and irrelevant information.

Humans tend to see what they expect to see [and don’t see what they don’t expect to see]. Where there is subjectivity, information is interpreted to meet our expectations. Because all forensic techniques involve some amount of subjectivity – some much more than others – when experts are exposed to ‘facts’ pointing to a suspects guilt, they tend to conclude there is a ‘match’ between that suspect and the evidence. The six dentists who ‘matched’ Keith Harward to the bite marks on Teresa Perron’s thighs knew that he was a sailor about the right age, the right ethnicity and that he had once bitten his girlfriend. That (irrelevant )information was almost certainly far more influential than anything they actually observed in the bite marks. Two of the dentists had excluded Harward before he was identified as a suspect. It was only after he was arrested that they changed their minds. Nothing had changed about the bite marks (Harward was wrongfully imprisoned for 28 years).

In most fields of science, efforts have to be made to mitigate the influence of bias through blinded studies and anonymized per review. In forensics, the issue was never seriously discussed until 2004, when an Oregon man was falsely arrested in relation to the bombing of a commuter train in Madrid, Spain. . . Following that fiasco, Dror, the cognitive neuroscientist, led one of the most significant experiments ever conducted on cognitive bias in forensics. Six highly qualified experts were provided with eight sets of prints to analyze, but a significant fact was withheld from the participants: the prints came from their own prior casework. Each of the experts had previously analyzed the same prints and reported their conclusions. The only difference was that Dror’as experiment omitted the (irrelevant) contextual information included in the case files, hints nudging the examiners toward a particular conclusion, e.g. the suspect confessed. Two-thirds of the seasoned experts came to conclusions which were inconsistent with their original analysis. The fingerprints had not changed, only their minds. And these were not bite marks; these were fingerprints.

Dror went on to conduct similar studies with similar results in a wide range of disciplines, including DNA mixture interpretation, forensic anthropology, and pathology. In his testimony before the NAS committee  Dror discussed the literature on various forms of cognitive bias, which come in many flavors: ‘confirmation bias,’ for example, is our tendency to seek out and interpret evidence to conform to our previously existing beliefs; ‘anchoring bias’ is our tendency to overvalue the first piece of information we receive, no matter how irrelevant; ‘contextual bias’ is broadly defined to include internal and external environments, expectations, motivations and emotions.

Though not the focus of the NAS investigation, cognitive biases aren’t limited to forensic experts led astray by irrelevant case information. Subjective decisions, implicitly influenced by biases, especially racial bias, drive almost every aspect of the judicial system but exploration of the deep historical, generational, cultural and societal forces that resulted in this grim reality is beyond the scope of this book.

For junk science, the fix is much easier: blind the experts.

This, however, requires a willingness to do so and acceptance of science. Many in the forensic community resent the suggestion that they are susceptible to bias, viewing it more as an indictment of their professionalism than a part of what it is to be human. Itiel Dror recognized this obstacle. He closed his 2007 NAS presentation by making the point that the research into the influence of cognitive bias wasn’t’ really in dispute in the scientific community. It was the experts themselves who resisted scientific reality – experts who saw no potential for bias by going into prisons and taking suspect’s dental molds, no bias in learning what other experts had concluded, no bias in learning case ‘facts’ from prosecutors on the eve of the trial.

 …………………………………………………………………….

In science, foundation-free opinions by self-proclaimed experts are rejected as ipse dixit (roughly translated, ‘It is because I say so’) . Cicero coined the term to distinguish evidence-based’ information from what is sometimes referred to as  ‘eminence-based’ knowledge – untested wisdom espoused by respected, often  charismatic, industry leaders.

Lawmakers and state and federal courts have often sought to dismantle collateral appeal mechanisms, bar substantive remedies for constitutional violations, and restrict review of federal habeas corpus allegations. Justification for these measures is rooted primarily in a legal doctrine known as the principle of finality. It mean never looking back once a jury has rendered a guilty verdict. Never correcting mistakes. Never addressing injustice. This, of course, is the opposite of science. Science is a iterative process, always moving forward, always reexamining te evidence underlying previously held beliefs.

 

[the use of ‘never’ in this passage is, of course, a polemical extravagance. It usually a matter of degree, except perhaps in Texas!]

Battle-weary, Lentini began paraphrasing physicist Max Planck’s seminal observation on scientific progress: ‘ Science advances one retirement at a time.’



*This standard comes from the Supreme Court case, Daubert v. Merrell Dow Pharmaceuticals Inc., 509 U.S. 579 (1993).

Under the Daubert standard, the factors that may be considered in determining whether the methodology is valid are: (1) whether the theory or technique in question can be and has been tested; (2) whether it has been subjected to peer review and publication; (3) its known or potential error rate; (4)the existence and maintenance of standards controlling its operation; and (5) whether it has attracted widespread acceptance within a relevant scientific community.

The Daubert standard is the test currently used in the federal courts and some state courts. In the federal court system, it replaced the Frye standard, which is still used in some states.

** the author phrases it as ‘poor people science’,  ‘poor people justice’. The science and justice meted out to poor  Americans