May Contain Lies: How Stories, Statistics, and Studies Exploit Our Biases – And What We Can Do about It

As lawyers, we pride ourselves on dissecting facts and building ironclad arguments. We’re trained to spot inconsistencies and demand evidence in support of claims. But what if some commonly held beliefs about facts, data, and evidence are surprisingly flawed? In his book, May ContainLies: How Stories, Statistics, and Studies Exploit Our Biases – And What We CanDo about It, Alex Edmans uncovers startling findings that challenge common assumptions about the information we consume. The following are some takeaways I found surprising:

• Your Knowledge Might Worsen Your Biases: Counter-intuitively, the book suggests that being more knowledgeable can actually make you more susceptible to confirmation bias (p. 19). More informed individuals are better at conjuring reasons to support their existing beliefs and poking holes in opposing views, leading to a "biased search" for information (p. 37). So, that deep expertise could inadvertently make you less objective.

• "Research Shows That..." is Often Meaningless: Phrases like "Research shows that..." or "A study finds that..." are commonly used as proof, but the source warns they are "often meaningless" (p. 5). This is frequently a result of "data mining", where researchers run "hundreds of different tests, hide those that don’t work and jump on the one that hits the bullseye" (p. 119). Even statistically significant findings can arise purely by chance if enough tests are conducted.

• The "10,000-Hour Rule" is Widely Misunderstood: The popular notion that 10,000 hours of practice is the "magic number of greatness" is largely a misrepresentation (p. 62). The original study it cited found no significant difference in practice hours between the "best" and "good" violinists, and the 10,000-hour figure itself was an average for one specific group at a certain age, not a universal guarantee of expertise.

• "Correlation is Not Causation" is More Complex Than You Think: While this adage is well-known, understanding why is crucial. Even strong correlations in data may not imply causation due to common causes (p. 82). Another factor is "reverse causation", where the outcome affects the input (e.g., patients becoming sicker after going to the doctor isn't because the doctor made them sick; they visited the doctor because they were already unwell) (p. 81).

• Evidence is Not Proof — It May Not Be Universal: Even if a study boasts strong "internal validity" (it accurately finds what it claims in the studied context), it might lack "external validity" (p. 97). This means that evidence gathered in one setting may not apply to different contexts or populations. For example, a successful management technique in a factory might not work in a school due to different goals and structures.

For In-House Legal Counsel:

These insights underscore the critical importance of deep inquiry beyond surface-level claims. As legal professionals working within organizations, our role extends beyond merely accepting presented information or "facts." We are uniquely positioned to challenge the underlying basis of data, studies, and expert opinions, whether it's an internal report supporting a new business strategy, a consultant's recommendation, or "best practices" shaping corporate policy. This involves asking questions like: 

• Are these facts truly representative data?
• Does this data really provide conclusive evidence of causation, or are there alternative explanations?
• Is this evidence truly applicable to our specific context, or is its universality overstated?

This discerning scrutiny can help safeguard our organizations from decisions based on misleading information.

May contain lies : how stories, statistics, and studies exploit our biases - and what we can do about it Edmans, Alex  2024, Book , 328 pages. 9780520405851