6. Evidence - PowerPoint Presentation

Slide1

6. Evidence based management:What is the best available evidence?Slide2

5-step approach

EBMgt is a 5-step approach

Formulate an answerable question (PICOC)

Search for the best available evidence Critically appraise the quality of the found evidence Integrate the evidence with managerial expertise and organizational concerns and apply Monitor and evaluate the resultsSlide3

Why are disciplines as such as psychoanalysis, astrology and parapsychology widely regarded as pseudo-science?

IntermezzoSlide4

“It is easy

to

obtain evidence in

favor of virtually any theory, but such ‘corroboration’ should count scientifically only if it is the positive result of a genuinely ‘risky’ prediction, which might conceivably have been false.… A theory is scientific only if it is refutable

by

a conceivable event. Every genuine

test

of a scientific theory, then, is logically an attempt to refute or to falsify it.”

Falsifiability

Carl PopperSlide5

“Inspect every piece of pseudoscience and you will find a security blanket, a thumb to suck, a skirt to hold.

What

have we to offer in exchange? Uncertainty! Insecurity

!”FalsifiabilityIsaac AsimovSlide6

Research designs

What is the BEST car?Slide7

Which design for which question?

Research designsSlide8

Explanation

Which design for which question?Slide9

Best research design?Slide10

Best available?Slide11

The best available evidence =

Studies

with the highest

internal validityStudies with the highest external validitySlide12

1. Best available evidence: internal validitySlide13

internal validity = indicates to what extent the results of the research may be biased and is thus a comment on the degree to which alternative explanations for the outcome found are

possible (confounding).

Internal validitySlide14
Slide15

Three criteria:

the "cause" and the "effect" are

related

the "cause" precedes the "effect" in time there are no plausible alternative explanations for the observed effectWhen do we know there is causal relation?CausalitySlide16

Considerations for research:

Causality

Are the "cause" and the "effect” related:

effect sizeDoes the "cause" precedes the "effect" in time: before and after measurement Are there no plausible alternative explanations for the observed effect:

randomization, control group

INTERNAL VALIDITYSlide17

internal validity = indicates to what extent

the results

of the research may be

biased and is thus a comment on the degree to which alternative explanations for the outcome found are possible (confounding).Internal validitySlide18

Bias Confounding

Reverse Causation

Methodological pitfallsSlide19

Bias: distortion of the outcome due to systematic errors caused by the way the study is designed or conducted.NB: If bias is not taken into account then any conclusions drawn may be wrong!

BiasSlide20

Selection bias Information

(detection) bias

Performance

bias Exclusion (attrition) bias Publication bias… … …..Forms of biasSlide21

Error in the way participants in a study were selected. Because of this comparison groups differ in measured or unmeasured baseline characteristics.

Selection biasSlide22

Distortion of the outcome due to misinterpretation of information or systematic errors in the the measurement of research variables which leads to misclassification. Information bias can be prevented by the use of standardized measurement instruments, hard outcome measures, validated questionnaires and objective, independent and blinded assessors.

Types of information bias:

Reporting bias (recall bias)

Observer bias (interviewer bias, halo-effect)Information biasSlide23

Confounding is the idea that a 3rd variable can distort or confuse (or confound..) a relationship between two other variables. For instance, when factor X causes disease Y, that relationship could be confounded by factor C that is associated with both factor X and disease Y. C would be an alternative explanation for the relationship observed between X and Y. ConfoundingSlide24

What are the confounders?

Shoe size

&

quality of handwritingBody length & body weightNumber of storks & birth rateSmoking youngsters & better lung functionSlide25

ConfoundingSlide26

Correlation does not equal causation!Slide27

http://kill-or-cure.heroku.com/Slide28

Reverse causationSlide29

?

Successful companies

Charismatic leaders

Reverse causationSlide30

Cause and effect can be established only through the proper research design: no amount of statistical hand waving can turn correlations into conclusions about causation

!!!

Internal validitySlide31

Levels of internal validitySlide32

Levels of internal validity

It is shown that …

It is likely that …

Experts are of the opinion that …

There are signs that …Slide33

The levels of internal validity can only be used to determine which type of research is the best method to assess the validity of the cause-and-effect relationship that might exist between an intervention (or moderator) and its outcomes. In this respect, cross-sectional studies and case-studies have the ‘weakest’ design. This of course doesn’t mean that cross-sectional studies and case-studies have a weak design overall. After all, different types of research questions require different types of research

designs.

A case study for instance is clearly a strong design for assessing why or in which way an effect has occurred, but obviously

not the most suitable design for assessing the strength of a possible cause-and-effect relationship.Keep in mind!Slide34

But … sometimes observational studies are as good as RCT’sInternal validity

When the size of effect is very large (swamps the bias)Slide35

These treatments have not been tested in RCTs: are they supported by poor evidence?

Internal validity

Heimlich manoeuvre

Dehydration: drinking water

Cardiac arrest: AED Slide36

2. Best available evidence: external validitySlide37

Ecological validity: Is your

organization

so different from those in the study that its results may be difficult to apply?Population validity: Is your population so different from those in the study that its results may be difficult to apply?

External validity: generalizability

Always ask yourself to what extent the evidence

is generalizable to your

situation:Slide38

Generalizability

Same

P

opulation?Same Intervention?Same Comparison?Same Outcome?Same

C

ontext?Slide39

Generalizability

Keep in mind:

What

works in one narrowly defined setting might not work in another, but some psychological principles are generalizable to all human beings. Slide40

Internal vs external validity

A

ll

research designs are flawed – though each is flawed differently. For instance, research designs with a high internal validity, such as controlled studies, may be less suited to generalization, which restricts their practical usability. Sample surveys and field research, on the other hand, have lower internal validity, but can sometimes be more useful for management practice. So there is always a trade off between internal validity (precision in control and measurements of variables) and external validity (generalizability with respect to populations, setting and context).Slide41

Best available evidence?

internal validity often high

internal validity often low

external validity often low

external validity often …?

external validity sometimes high