based management What is the best available evidence 5step approach EBMgt is a 5step approach Formulate an answerable question PICOC Search for the best available evidence Critically ID: 190515
Download Presentation The PPT/PDF document "6. Evidence" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
6. Evidence based management:What is the best available evidence?Slide2
5-step approach
EBMgt is a 5-step approach
Formulate an answerable question (PICOC)
Search for the best available evidence Critically appraise the quality of the found evidence Integrate the evidence with managerial expertise and organizational concerns and apply Monitor and evaluate the resultsSlide3
Why are disciplines as such as psychoanalysis, astrology and parapsychology widely regarded as pseudo-science?
IntermezzoSlide4
“It is easy
to
obtain evidence in
favor of virtually any theory, but such ‘corroboration’ should count scientifically only if it is the positive result of a genuinely ‘risky’ prediction, which might conceivably have been false.… A theory is scientific only if it is refutable
by
a conceivable event. Every genuine
test
of a scientific theory, then, is logically an attempt to refute or to falsify it.”
Falsifiability
Carl PopperSlide5
“Inspect every piece of pseudoscience and you will find a security blanket, a thumb to suck, a skirt to hold.
What
have we to offer in exchange? Uncertainty! Insecurity
!”FalsifiabilityIsaac AsimovSlide6
Research designs
What is the BEST car?Slide7
Which design for which question?
Research designsSlide8
Explanation
Which design for which question?Slide9
Best research design?Slide10
Best available?Slide11
The best available evidence =
Studies
with the highest
internal validityStudies with the highest external validitySlide12
1. Best available evidence: internal validitySlide13
internal validity = indicates to what extent the results of the research may be biased and is thus a comment on the degree to which alternative explanations for the outcome found are
possible (confounding).
Internal validitySlide14Slide15
Three criteria:
the "cause" and the "effect" are
related
the "cause" precedes the "effect" in time there are no plausible alternative explanations for the observed effectWhen do we know there is causal relation?CausalitySlide16
Considerations for research:
Causality
Are the "cause" and the "effect” related:
effect sizeDoes the "cause" precedes the "effect" in time: before and after measurement Are there no plausible alternative explanations for the observed effect:
randomization, control group
INTERNAL VALIDITYSlide17
internal validity = indicates to what extent
the results
of the research may be
biased and is thus a comment on the degree to which alternative explanations for the outcome found are possible (confounding).Internal validitySlide18
Bias Confounding
Reverse Causation
Methodological pitfallsSlide19
Bias: distortion of the outcome due to systematic errors caused by the way the study is designed or conducted.NB: If bias is not taken into account then any conclusions drawn may be wrong!
BiasSlide20
Selection bias Information
(detection) bias
Performance
bias Exclusion (attrition) bias Publication bias… … …..Forms of biasSlide21
Error in the way participants in a study were selected. Because of this comparison groups differ in measured or unmeasured baseline characteristics.
Selection biasSlide22
Distortion of the outcome due to misinterpretation of information or systematic errors in the the measurement of research variables which leads to misclassification. Information bias can be prevented by the use of standardized measurement instruments, hard outcome measures, validated questionnaires and objective, independent and blinded assessors.
Types of information bias:
Reporting bias (recall bias)
Observer bias (interviewer bias, halo-effect)Information biasSlide23
Confounding is the idea that a 3rd variable can distort or confuse (or confound..) a relationship between two other variables. For instance, when factor X causes disease Y, that relationship could be confounded by factor C that is associated with both factor X and disease Y. C would be an alternative explanation for the relationship observed between X and Y. ConfoundingSlide24
What are the confounders?
Shoe size
&
quality of handwritingBody length & body weightNumber of storks & birth rateSmoking youngsters & better lung functionSlide25
ConfoundingSlide26
Correlation does not equal causation!Slide27
http://kill-or-cure.heroku.com/Slide28
Reverse causationSlide29
?
Successful companies
Charismatic leaders
Reverse causationSlide30
Cause and effect can be established only through the proper research design: no amount of statistical hand waving can turn correlations into conclusions about causation
!!!
Internal validitySlide31
Levels of internal validitySlide32
Levels of internal validity
It is shown that …
It is likely that …
Experts are of the opinion that …
There are signs that …Slide33
The levels of internal validity can only be used to determine which type of research is the best method to assess the validity of the cause-and-effect relationship that might exist between an intervention (or moderator) and its outcomes. In this respect, cross-sectional studies and case-studies have the ‘weakest’ design. This of course doesn’t mean that cross-sectional studies and case-studies have a weak design overall. After all, different types of research questions require different types of research
designs.
A case study for instance is clearly a strong design for assessing why or in which way an effect has occurred, but obviously
not the most suitable design for assessing the strength of a possible cause-and-effect relationship.Keep in mind!Slide34
But … sometimes observational studies are as good as RCT’sInternal validity
When the size of effect is very large (swamps the bias)Slide35
These treatments have not been tested in RCTs: are they supported by poor evidence?
Internal validity
Heimlich manoeuvre
Dehydration: drinking water
Cardiac arrest: AED Slide36
2. Best available evidence: external validitySlide37
Ecological validity: Is your
organization
so different from those in the study that its results may be difficult to apply?Population validity: Is your population so different from those in the study that its results may be difficult to apply?
External validity: generalizability
Always ask yourself to what extent the evidence
is generalizable to your
situation:Slide38
Generalizability
Same
P
opulation?Same Intervention?Same Comparison?Same Outcome?Same
C
ontext?Slide39
Generalizability
Keep in mind:
What
works in one narrowly defined setting might not work in another, but some psychological principles are generalizable to all human beings. Slide40
Internal vs external validity
A
ll
research designs are flawed – though each is flawed differently. For instance, research designs with a high internal validity, such as controlled studies, may be less suited to generalization, which restricts their practical usability. Sample surveys and field research, on the other hand, have lower internal validity, but can sometimes be more useful for management practice. So there is always a trade off between internal validity (precision in control and measurements of variables) and external validity (generalizability with respect to populations, setting and context).Slide41
Best available evidence?
internal validity often high
internal validity often low
external validity often low
external validity often …?
external validity sometimes high