P-values and their limitations & - PowerPoint Presentation

1. P-values and their limitations &Type I and Type II errorsStats Club 8Marnie Brennan

2. ReferencesPetrie and Sabin - Medical Statistics at a Glance: Chapter 17 & 18 GoodPetrie and Watson - Statistics for Veterinary and Animal Science: Chapter 6 GoodKirkwood and Sterne – Essential Medical Statistics: Chapter 8 & 35Dohoo, Martin and Stryhn – Veterinary Epidemiologic Research: Chapter 2 & 6

3. What do you know about P-values?

4. Interesting reads!Sterne, JAC and Davey-Smith, G (2001) Sifting the evidence – what’s wrong with significance tests? British Medical Journal, Vol. 322, 226-231. - GoodAltman, DG and Bland, JM (1995) Absence of evidence is not evidence of absence. British Medical Journal, Vol. 311, 485.Nakagawa, S and Cuthill, IC (2007) Effect size, confidence interval and statistical significance: a practical guide for biologists. Biological Reviews, Vol. 82, 591-605. – I’ve not read this, but it has been recommended

5. Differences between groupsMany different tests to measure the difference between two or more groups of subjects/animals/patientsWe will cover these individually in subsequent weeksHow do we know whether they are truly different from each other?i.e. Is there truly a difference between groups or not?

6. Hypothesis (significance) testingYou have a scientific question you want to answerYou construct a hypothesis to test your questionYou have to have an alternative hypothesis to test it againstThis differs from projects which might be hypothesis generatingi.e. you are exploring possible factors and don’t know yet which ones are more important

7. http://withfriendship.com/user/boss/hypothesis.php

8. Null and alternative hypothesesNull hypothesis – No difference between groups/no association between variablesSometimes written as H0Alternative hypothesis – There is a difference between groups/an association between variablesSometimes written as H1These hypotheses relate to the population of interest, not your sample of the population

9. Thanks to http://dsm1lp.wordpress.com/2012/02/19/what-role-does-the-null-hypothesis-really-play-in-the-scientific-process/ for the example!!!Where have all my socks gone?An example!Aliens have come to Earth specifically to take my socksMy socks are missing because I am disorganisedXALTERNATIVE HYPOTHESISNULL HYPOTHESIS

10. What is a P-value?We do our study, run our statistical tests, and come up with a P-value or Probability‘The P-value is the probability of obtaining our results or something more extreme, if the null hypothesis is true’ (Petrie and Sabin)‘The probability of getting a difference at least as big as that observed if the null hypothesis is true’ (Kirkwood and Sterne)‘The chance of getting the observed effect (or one or more extreme) if the null hypothesis is true’ (Petrie and Watson)

11. What does this mean??!!Basically the probability of getting what you have got with your study results if the null hypothesis is true!If the difference between our groups is largeThe probability would be small, therefore unlikely the null hypothesis is true (and you usually reject the null hypothesis as there is evidence against it)If the difference between our groups is small The probability would be large, therefore likely the null hypothesis is true (there is not enough evidence to reject the null hypothesis)Bad to say you accept the null hypothesis!‘Absence of evidence is not evidence of absence’

12. A value of the test statistic which gives P>0.05A value of the test statistic which gives P<0.05Significant at the 5% levelNot significant at the 5% level

13. Using P-valuesUsually you set your ‘significance’ level before you collect your data – this should be stated in the methodse.g. ‘We set the significance level at P<0.01 for our analysis’P<0.05 is a fairly arbitrary level (one guy’s ponderings!)Read the article by Sterne and Davey SmithBottom line - the smaller the P-value, the more evidence against the null hypothesis

14. A sliding scale.......

15. How does this fit with what you do or have seen/experienced?

16. P-value etiquette (variable!)!Always quote the exact P-value if you canE.g. P = 0.032, not P<0.05Display P-values accurate to two significant figuresE.g. P=0.032, or 0.17When P-values become very small, acceptable to display as P<0.001

17. Limitations of using just P-valuesBy just using P-values, you lose a lot of informationDoesn’t tell you about the magnitude of the effect observedOften researchers only talk about P-values, and nothing elseI am certainly guilty of this!It is also important to determine whether your result is biologically or clinically important (not only that it is ‘significant’) – if you just use a number to interpret outcomes, it may not ‘mean’ anythingYou can use Confidence Intervals (CI’s) to quantify the effect of interestGives you a range of values which represent the difference between your groupsThere is another Stats Club session on these coming up

18. Interpretation of research

19. You Tube video on P-valueshttp://www.youtube.com/watch?v=eyknGvncKLw

20. Errors in hypothesis testingThe rejection of the null hypothesis, or not, can be wrong in studies sometimesPetrie and Watson

21. Type I errorWhen we reject the null hypothesis and it is actually trueAffected by:Significance level chosen (becomes the maximum chance of making a Type I error)If significance level P<0.05 - 1 in 20 chance that a test will be significant by chanceIf P<0.01 - 1 in 100 chance the test is significant by chanceNumber of comparisons – the greater number of comparisons carried out, the more likely you will get a ‘positive’ result that is spurious (multiple testing issue)Comes back to whether the result is biologically or clinically importantCan adjust for this using post-hoc analysis e.g. Bonferroni correctionhttp://illuminutti.com/tag/false-positive/

22. Type II errorWe don’t reject the null hypothesis when there is evidence to do soAffected by:Small sample sizes – more chance of getting Type II errorsPrecision of the measurements – if measurements are precise, less chance of getting Type II errorsEffect of interest – the larger the difference between the groups, the less likely that a Type II error will occurhttp://illuminutti.com/tag/false-positive/

23. Type I and Type II error - relationshipThese two things are related, generally as one increases, the other decreasesBottom line – if your study design is correct, you have carried out a sample size calculation and have recruited the right number of subjects, then the chances of error decrease hugely as the power of your study will be sufficientSample size calculations and power will be discussed in later Stats Club sessions

24. SummarySet your significance level BEFORE you start your data collection, and don’t just go automatically for P<0.05 – think about what you are trying to show with your researchDisplay your P-values correctlyUse P-values but also confidence intervals to get an idea of the magnitude of the difference between groupsSet your study up right to decrease the chances of Type I and Type II errors

25. My kind of hypothesis testing!!!http://www.rootsrundeep.com/hypothesis.html

26. Next timeConfidence intervals beware……