Extraordinary Claims - PowerPoint Presentation

Slide1

Extraordinary ClaimsSlide2

“A wise man… proportions his belief to the evidence.” – David HumeSlide3

“Extraordinary claims require extraordinary evidence.” – Carl SaganSlide4

Base rate neglectSlide5

Outcomes

Yes

No

Test = Yes

True Positive

False Positive

Test = No

False Negative

True NegativeSlide6

Base Rates

The

base rate

for any condition is simply the proportion of people who have the condition.

Base rate of dog owners in Hong Kong:

# of Dog Owners in Hong Kong

÷

# of Hong

KongersSlide7

Base Rate

Yes

No

Test = Yes

True Positive

False Positive

Test = No

False Negative

True NegativeSlide8

As the base rate decreases, the number of true positives decreases.Slide9

Base Rate

Yes

No

Test = Yes

True Positive

False Positive

Test = No

False Negative

True NegativeSlide10

Importance of Base Rates

Therefore, the proportion of false positives out of total positives increases:

False Positive

÷

False Positive + True PositiveSlide11

Importance of Base Rates

And the proportion of true positives out of total positives decreases:

True Positive

÷

False Positive + True PositiveSlide12

What Does It Mean?

It means that even very good tests (low false positive rate, low false negative rate) cannot reliably detect conditions with low base rates.Slide13

Example

Suppose the police have a test for telling whether someone is driving drunk:

If you are drunk, the test returns “positive” 100% of the time.

If you are not drunk, the test returns “positive” 95% of the time.

The police randomly stop 1,000 drivers and test them.Slide14

The base rate of drunk drivers is:

# of drunk drivers

÷

# of drunk drivers + # of sober driversSlide15

High Base Rate

The base rate of drunk drivers is:

# of drunk drivers =500

÷

# of drunk drivers =500 + # of sober drivers =500

Base rate = 50%Slide16

Outcomes

Drunk

Not

Drunk

Test = Yes

True Positive

False Positive

Test = No

False Negative

True NegativeSlide17

Outcomes

Drunk

Not

Drunk

Test = Yes

500

x 100%

500

x 5%

Test = No

500

x 0%

500

x 95%Slide18

Outcomes

Drunk

Not

Drunk

Test = Yes

500

25

Test = No

0

475Slide19

Good Test!

Now we have 500 true positives and only 25 false positives:

False Positive

÷

False Positive + True PositiveSlide20

Good Test!

Now we have 500 true positives and only 25 false positives:

25

÷

25 + 500

5% =Slide21

Low Base Rate

The base rate of drunk drivers is:

# of drunk drivers =5

÷

# of drunk drivers =5 + # of sober drivers =995

Base rate = 0.5%Slide22

Outcomes

Drunk

Not

Drunk

Test = Yes

True Positive

False Positive

Test = No

False Negative

True NegativeSlide23

Outcomes

Drunk

Not

Drunk

Test = Yes

5

x 100%

995 x 5%

Test = No

5

x 0%

995 x 95%Slide24

Outcomes

Drunk

Not

Drunk

Test = Yes

5

50

Test = No

0

945Slide25

Bad Test!

Now we have 5 true positives and 50 false positives:

False Positive

÷

False Positive + True PositiveSlide26

Bad Test!

Now we have 5 true positives and 50 false positives:

50

÷

50 + 5

91% =Slide27

We saw this same phenomenon when we learned that most published scientific research is false.Slide28
Slide29

Low base rate of truths.Slide30

High proportion of false positives.Slide31

Base Rate Neglect Fallacy

The base rate neglect fallacy is when we ignore the base rate.

The base rate is very low, so our tests are very unreliable… but we still trust the tests.Slide32

It Matters

The Hong Kong police stop people 2 million times every year.

Only 22,500 of these cases (1%) are found to be offenses.

Even if the police have a very low false positive rate it’s clear that the base rate of offenses does not warrant these stops.Slide33

Prior probabilitiesSlide34

Probabilities

A base rate is a rate (a frequency). # of X’s out of # of Y’s. # of drunk drivers out of # of total drivers.

Sometimes a rate or a frequency doesn’t make sense. Some things only happen once, like an election. So here we talk about

probabilities

instead of rates.Slide35

Probabilities

For things that do happen a lot the probabilities are (or approximate) the frequencies:

Probability of random person being stopped by police

=

# of people stopped

by police

÷ total # of peopleSlide36

Things that Only Happened Once

The 2012 Chief Executive Election

The 1997 Handover

The great plague of Hong Kong

The second world war

The extinction of the dinosaurs

The big bangSlide37

Evidence

Drunk tests are a type of evidence. Testing positive raises the probability that you are drunk. But how much does it raise the probability?

That depends on the base rate

.Slide38

Evidence

You can also have evidence for or against one-time events. For example, you might have a footprint at the scene of a crime. This raises the probability that certain people are guilty. How much?

That depends on the prior probability

.Slide39

Priors and Posteriors

Prior probability = probability that something is true before you look at the new evidence.

Posterior probability = probability that something is true after you look at the new evidence.Slide40

Relativity

Priors and posteriors are relative to evidence. Once we look at new evidence, our posteriors become our new priors when we consider the next evidence.Slide41

Bayes’ Theorem

P (data/

hyp

.) x P(

hyp

.)

÷

P(data)

P(hypothesis/ data) =

Posterior

PriorSlide42

Prior/ Posterior Positively Correlated

P(hypothesis

/ data

) ∝ [P(data/

hyp

.) x P(

hyp

.)]Slide43

What Does It Mean

If the prior probability of some event is very low, then even very good evidence for that event does not significantly increase its probability.Slide44

“Extraordinary claims require extraordinary evidence.” – Carl SaganSlide45

Hume on miraclesSlide46

“A wise man… proportions his belief to the evidence.” – David HumeSlide47

Novel Testimony

Suppose

that we get testimony concerning something we have never experienced.

Hume imagines someone from the equatorial regions being told about frost, and snow, and ice. They have never experienced anything like that before.Slide48

It’s Strange!Slide49

Hume

thinks this person would have reason to disbelieve stories about a white powder that fell from the sky, covered everything by several inches, and then turned to water and went away

.Slide50

It’s

not that they should believe the stories are

not

true, just that they don’t have to believe they

are

true. We need more

evidence, because the prior is so low.Slide51

But

nowsuppose

someone tells us an even stranger story.

It’s

like the snow-story, in that we’ve never experienced anything like it before. But it’s even stranger, because we have

always

experienced the

opposite

before.Slide52

Miracles

For

Hume, this is the definition of a miracle. A miracle is a violation of the laws of nature. Every event or process in the world conforms to the laws of nature (for example, the laws of physics like the law of gravity)– except, if there are any, miracles. Slide53

Example

There are about 100 billion people who have lived and died in the history of humanity (and there are 7 billion more who are alive now).

As far as we know, none of the 100 billion people who have ever died and were dead for four days, later came back to life. It’s a law of nature that when you die, that’s the end, there’s no more.Slide54

Lazarus

Although there is testimony, in at least one religious book– the Christian bible– that such an event occurred at least once in history, when Jesus raised Lazarus from the dead, after he had been dead for four days.Slide55

What Should We Believe?

According to Hume, we should be wise and apportion our belief to the evidence.

Since on the one hand we have 100 billion people who died and never came back, and on the other hand we have an old legend from a book intended to make people believe its religious views, it’s most probable that

the raising of Lazarus never happened

.Slide56

Hume on Miracles

“No testimony is sufficient to establish a miracle, unless the testimony be of such a kind that its falsehood would be more miraculous than the fact which it endeavors to establish.”Slide57

Seeing and Believing

So

, for example, Hume would even say that if you

saw

someone die and come back to life, you should not believe that it really happened. Slide58

Seeing and Believing

Because

it’s always possible that what you saw was a trick, or the person was never really dead, or you were on drugs or…

Since

none of those suppositions are miraculous, you should believe them instead of believing in the miracle. They’re more likely than a violation of nature’s laws.Slide59

No more philosophy!Slide60

OK, Back to Science…

There’s a debate among scientists about Evidence Based Medicine vs. Science Based Medicine.

They sound the same, but they’re very different!Slide61

Modern Medicine

In current modern medicine the following is (one) best estimate:

37% of treatments are based on Randomized Controlled Trials

76% of treatments are based on good evidence (RCTs, observational studies)

The rest should be based on scientific theory (reasonable extension of what we know).Slide62

Evidence Based Medicine

One idea is that the 76% of tested-treatments are the “real” evidence based medicine and the rest is no better than untested alternative medicine. These are equal:

Treatments

based on scientific theory (reasonable extension of what we know

).

Untested pre-scientific or otherwise alternative treatments (e.g. homeopathy).Slide63

Difficult Tests

Some alternative treatments are difficult to test.

Homeopaths claim that their treatments are individualized. So it’s not enough to give everyone suffering from a disease the same magic water… they have to come into the shop for a personalized experience.Slide64

Can’t Placebo a Whole Shop!Slide65

False Equivalence

This means we should let the homeopaths “get away with it.” Sure, their treatments aren’t supported by science, but neither are 24% of modern treatments.Slide66

Science Based Medicine

Science based medicine, on the other hand, says we should take into account

prior probability

.

We have lots of scientific knowledge of water. Nothing about it says that chemically pure water that

in the past

contained other chemicals and

was then shaken

should behave any differently than regular chemically pure waterSlide67

Science Based Medicine

And, science based medicine says that the 24% of treatments that are not evidence based,

while they should still be tested

, are much better because of prior probability.

If science tells us why they should work, then we should believe the science even if we haven’t tested them (yet) or can’t test them.Slide68

Example

It’s immoral not to perform blood transfusions on people who have lost lots of blood.

So we can’t do a RCT on blood transfusions.

But that doesn’t mean they’re as silly as homeopathy.