Mogalanyane Statistics South Africa 8 May 2019 Missing in action testing alternative imputation methods in price statistics Aim Why prices go missing and why do we care 1 What others have said about dealing with missing prices ID: 1047817
Download Presentation The PPT/PDF document "Patrick Kelly and Matlhatsi" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Patrick Kelly and Matlhatsi MogalanyaneStatistics South Africa8 May 2019Missing in actiontesting alternative imputation methods in price statistics
2. AimWhy prices go missing and why do we care?1What others have said about dealing with missing prices2Test the most effective ways to deal with missing prices3
3. Why do prices go missing?And why do we care?
4. Why do prices go missing?Temporarily missingOut of stockSeasonalPermanently missingDiscontinuedTemp unavailable > 2 months
5. Aim of imputingMissing price creates a gap in sampled items – could create biasObjectives of imputing“First do no harm” – Minimise biasComplete matrix of prices for immediate use of next price when available
6. Imputation methodsOverall meanPrice relatives of all matched varieties in elementary indexTargeted meanPrice relatives of a subset of matched varieties in elementary indexCarry forwardPrice of the variety from the previous monthOmitDo nothing
7. What about regression?Time product dummy commonly used for quality adjustment when substituting for permanently missing pricesUses longer time series than traditional imputation techniquesTest it as a method for imputing temporarily missing prices (will it help with seasonal products?)
8. LiteratureWhat have others said?
9. LiteratureVery little quantitative analytical workException is Swiss study – test performance of overall and class mean for quality adjustment when substituting permanently missing varieties of clothingClass mean performed better than overall mean but needs bigger sample
10. Methodology
11. Methodology25 month dataset for 7 products using South African CPI Imputed all missing prices to get complete matrix of dataRandomly deleted 10% of prices in each product groupCOICOP groupProductPricing behaviourNumber of observationsBread and cerealsBreadStable758FruitPeachesStrong seasonal10VegetablesBroccoliWeak seasonal57Milk, cheese and eggsEggsStable414ClothingMen’s shortsStrong seasonal197ClothingMen’s jeansStable180FurnitureBedroom suitesSticky135Description of data
12. MethodologyImputed missing prices using 4 methodsOverall mean - all available price relatives for productTargeted mean - price relatives of a subset of varieties for that product (specific geographic area)Carry forward - use price in t-1 as the price in t.TPD - all data from the product group for the current and previous 12 months.
13. MethodologyCompare imputed price and index to:Original dataData with missing prices
14. Results
15. ResultsMeasure deviation of price and index from the original data set
16. Price level: Difference between actual and imputed price
17. Index level: Deviation of index by imputation type
18. Index level: Deviation of index by imputation type – Do nothing
19. Index level: Deviation of index by imputation type – Carry forward
20. Index level: Deviation of index by imputation type – TPD
21. Index level: Deviation of index by imputation type – Overall
22. Index level: Deviation of index by imputation type – Targeted
23. Index level: Deviation of index by imputation type
24. Conclusions
25. ConclusionsConfirm overall and targeted mean as most reliable methodsOverall mean performed better with seasonal items Overall mean has larger sample – ensure targeted mean has adequate observationsPerformance of TPD was mixed – needs further investigationCarry forward performed well for sticky prices – but will miss change when it does happenDo nothing creates biggest bias – Not meeting criteria of ‘do no harm’
26. The endThank you