Robert Anderson SAS JMP Quick introduction to modelling and crossvalidation Demo in JMP using simulated data and crossvalidation T o show it working and not working To show the benefit of using multiple validation columns ID: 1047569
Download Presentation The PPT/PDF document "Missing Genuine Effects Is Bad, but Iden..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Missing Genuine Effects Is Bad, but Identifying False Effects Can Be WorseRobert Anderson SAS JMP
2. Quick introduction to modelling and cross-validationDemo in JMP using simulated data and cross-validationTo show it working and not workingTo show the benefit of using multiple validation columnsResults from sensitivity studies on cross-validation successUsing simulated data and many runs under a variety of conditions Today’s talk
3. What do we mean by a model?y2y3x1x2xnFactors/inputsy1Responses/outputsSystem (black box)Equationy = f(x) + ErrorThe model is just an equation or expression that defines the relationship between the inputs and the outputsScientists and engineers need to be able to find the best possible model and correctly identify which factors are genuinely important and which are notOften the greatest concern is that an important or vital factor will be missed. However, statistical modelling methods frequently identify factors which are statistically significant but not genuinely active and that can be an even worse problem.
4. Prediction Profiler Allows the Model to be VisualizedThis is the prediction profiler for a model obtained from analysing historical dataModel equation: Y = 2*X1 – 2.5*X2 + 3*X3 + 3*(X3*X4) – 2*(X5)2 Linear termsInteraction termSquared term
5. Identifying which terms to include in modelTrue situation (Actual)Include or exclude a variable or term in a model Include variable or termExclude variable or termVariable or term is genuinelyimportant??Variable or term is not genuinely important??Implications of finding the incorrect model terms?
6. Identifying which terms to include in modelTrue situation (Actual)Include or exclude a variable or term in a model Include variable or termExclude variable or termVariable or term is genuinelyimportantTrue PositiveCorrect decision madeNo adverse implications?Variable or term is not genuinely important?True NegativeCorrect decision madeNo adverse implicationsImplications of finding the incorrect model terms?
7. Identifying which terms to include in modelTrue situation (Actual)Include or exclude a variable or term in a model Include variable or termExclude variable or termVariable or term is genuinelyimportantTrue PositiveCorrect decision madeNo adverse implicationsFalse NegativeImportant effect is missedPoorer understandingCan’t explain all the variationNeed to continue lookingVariable or term is not genuinely important?True NegativeCorrect decision madeNo adverse implicationsImplications of finding the incorrect model terms?Missing a real effect
8. Identifying which terms to include in modelTrue situation (Actual)Include or exclude a variable or term in a model Include variable or termExclude variable or termVariable or term is genuinelyimportantTrue PositiveCorrect decision madeNo adverse implicationsFalse NegativeImportant effect is missedPoorer understandingCan’t explain all the variationNeed to continue lookingVariable or term is not genuinely importantFalse PositiveNon-genuine effect includedIncorrect understandingWastes time and effortUnexplained variation missedTrue NegativeCorrect decision madeNo adverse implicationsImplications of finding the incorrect model terms?Missing a real effectIdentifying a false effect
9. “Training” sample“Validation” sample“Test” sampleHow the data will be used (Validation methodology)Cross-validation in JMP ProMost of the data will be used to build (or train) the modelSome data will be held back to ensure that the model is not ‘over fitted’ and is the best possible model using that model building techniqueSome data will be held back and not used in the model building process at all. This data will allow a fair comparison of how accurate the predictions from competing models are likely to be.Data randomly split into 3 samplesCross-validation is a way to suppress over-fitting and to reduce the chance of a model containing non-genuine or false effects
10. R2 used to measure the performance of your modelJMP stops adding terms to the model when the validation R2 reaches a maximum. This suppresses over-fitting.Measuring your model’s performanceTraining sampleValidation sample
11. R2 used to measure the performance of your modelJMP stops adding terms to the model when the validation R2 reaches a maximum. This suppresses over-fitting.Measuring your model’s performanceTraining sampleValidation sampleExplanatory power of modelhighlow
12. R2 used to measure the performance of your modelJMP stops adding terms to the model when the validation R2 reaches a maximum. This suppresses over-fitting.Measuring your model’s performanceTraining sampleValidation sampleModel complexitylowhighExplanatory power of modelhighlow
13. R2 used to measure the performance of your modelJMP stops adding terms to the model when the validation R2 reaches a maximum. This suppresses over-fitting.Measuring your model’s performance8 model terms gives the maximum validation R2Training sampleValidation sampleModel complexitylowhighExplanatory power of modelhighlow
14. Let’s look at an example in JMP now
15. Over-fitted model includes many statistically significant terms which are non-genuine and false signalsOver-fitted model obtained when validation isn’t usedCorrect model is obtained when validation is usedBenefit of Using Cross-ValidationActual model used to simulate the dataSimulated data was used so that the correct model was known
16. Some simulation studies to see how sensitive the validation method is to certain parametersThe results of the following simulation studies were obtained by drawing random samples from a 1000 row randomly generated dataset in which the response Y was simulated using a column formula of the form shown below.In each of the simulation studies, a single validation column was tried and the number of times the correct model was obtained was recorded.Model equation: Y =
17. The Effect of Sample Size on Cross-validation SuccessSample size = variedEffect size S/N ratio = 2Training/Validation ratio = 0.7/0.3Number of active terms = 3Number of columns = 30Each data point represents the percentage of correct models obtained from 10 trials using simulated data and a single validation column
18. The Effect of Effect Size on Cross-validation SuccessSample size = 50Effect size S/N ratio = variedTraining/Validation ratio = 0.7/0.3Number of active terms = 3Number of columns = 30
19. The Effect of Training\Validation Proportions on Cross-validation SuccessSample size = 50Effect size S/N ratio = variedTraining/Validation ratio = variedNumber of active terms = 3Number of columns = 30
20. The Effect of Model Complexity on Cross-validation SuccessSample size = 50Effect size S/N ratio = variedTraining/Validation ratio = 0.7/0.3Number of active terms = variedNumber of columns = 30
21. The Effect of the Number of Variables on Cross-validation SuccessSample size = 50Effect size S/N ratio = variedTraining/Validation ratio = 0.7/0.3Number of active terms = 3Number of columns = varied
22. ConclusionsIf you are building models from historical or observational data, you should be using cross-validationIf you use cross-validation, you shouldn’t rely on a single validation column, you should try multiple validation columnsThe simplest and most frequently occurring model using multiple validation columns is likely to be the ‘correct’ modelCross-validation suppresses overfitting (or finding non-genuine effects) but it doesn’t always prevent it.