Sequential Data Cleaning: A Statistical Approach
Description: Sequential Data Cleaning: A Statistical Approach Aoqian Zhang1, Shaoxu Song1 , Jianmin Wang1 1Tsinghua University, China 120 SIGMOD 2016 Outline Motivation Problem Solutions Exact Solution Approximate Solution Experiments Conclusion 220
Related Topics
Download Presentation
"Sequential Data Cleaning: A Statistical Approach" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Sequential Data Cleaning: A Statistical Approach Aoqian Zhang1, Shaoxu Song1 , Jianmin Wang1
1Tsinghua University, China 1/20 SIGMOD 2016<br>
slide2. Outline Motivation
Problem
Solutions
Exact Solution
Approximate Solution
Experiments
Conclusion 2/20 SIGMOD 2016<br>
slide3. Stream Data Erroneous Stream data are often dirty
Unreliable sensor reading
Large spike and small errors
Stock and Flight
Accuracy of Stock in Yahoo! Finance is 0.93[1]
Accuracy of Travelocity is 0.95[1]
Reasons
Ambiguity in information extraction
Pure mistake 3/20 SIGMOD 2016 [1] X. Li, X. L. Dong, K. Lyons, W. Meng, and D. Srivastava. Truth finding on the deep web: Is the problem solved? PVLDB, 6(2):97-108, 2012.<br>
slide4. Data Cleaning 4/20 SIGMOD 2016 Constraint based methods
Constraints on speeds of value changes[1]
Minimum change principle
Large spike error: max/min values allowed
Small error: fail to identify [1] Song, Shaoxu, et al. "SCREEN: Stream Data Cleaning under Speed Constraints." SIGMOD’15<br>
slide5. Probability 5/20 SIGMOD 2016<br>
slide6. Likelihood 6/20 SIGMOD 2016<br>
slide7. Outline Motivation
Problem
Solutions
Exact Solution
Approximate Solution
Experiments
Conclusion 7/20 SIGMOD 2016<br>
slide8. Problem 8/20 SIGMOD 2016<br>
slide9. Problem Budget 9/20 SIGMOD 2016<br>
slide10. Outline Motivation
Problem
Solutions
Exact Solution
Approximate Solution
Experiments
Conclusion 10/20 SIGMOD 2016<br>
slide11. DP-based Solution 11/20 SIGMOD 2016<br>
slide12. Other Approximation 12/20 SIGMOD 2016<br>
slide13. Solutions 13/20 SIGMOD 2016 Summary<br>
slide14. Outline Motivation
Problem
Solutions
Exact Solution
Approximate Solution
Experiments
Conclusion 14/20 SIGMOD 2016<br>
slide15. Experiment Setting 15/20 SIGMOD 2016 1. http://finance.yahoo.com/q/hp?s=AIP.L+Historical+Prices<br>
slide16. Comparison 16/20 SIGMOD 2016 Constraint based
SCREEN1
Different Algorithm
DP: Exact solution
DPC: Constant factor approximation
DPL: Linear time approximation
QP: Continuous probabilistic distribution approximation
SG: Simple greedy Song, Shaoxu, et al. "SCREEN: Stream Data Cleaning under Speed Constraints." Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. ACM, 2015.<br>
slide17. Analysis-STOCK 17/20 SIGMOD 2016 Scalability
Budget<br>
slide18. Outline Motivation
Problem
Solutions
Exact Solution
Approximate Solution
Experiments
Conclusion 18/20 SIGMOD 2016<br>
slide19. Conclusion Repair
Precisely handle large spike errors
Small errors can be detected and repaired
Various methods accustomed in different situations
Better performance in both repairing and application accuracies compared to the state-of-art data constraint-based repairing 19/20 SIGMOD 2016<br>
slide20. Q & A Thanks！ 20/20 SIGMOD 2016<br>
1Tsinghua University, China 1/20 SIGMOD 2016<br>
slide2. Outline Motivation
Problem
Solutions
Exact Solution
Approximate Solution
Experiments
Conclusion 2/20 SIGMOD 2016<br>
slide3. Stream Data Erroneous Stream data are often dirty
Unreliable sensor reading
Large spike and small errors
Stock and Flight
Accuracy of Stock in Yahoo! Finance is 0.93[1]
Accuracy of Travelocity is 0.95[1]
Reasons
Ambiguity in information extraction
Pure mistake 3/20 SIGMOD 2016 [1] X. Li, X. L. Dong, K. Lyons, W. Meng, and D. Srivastava. Truth finding on the deep web: Is the problem solved? PVLDB, 6(2):97-108, 2012.<br>
slide4. Data Cleaning 4/20 SIGMOD 2016 Constraint based methods
Constraints on speeds of value changes[1]
Minimum change principle
Large spike error: max/min values allowed
Small error: fail to identify [1] Song, Shaoxu, et al. "SCREEN: Stream Data Cleaning under Speed Constraints." SIGMOD’15<br>
slide5. Probability 5/20 SIGMOD 2016<br>
slide6. Likelihood 6/20 SIGMOD 2016<br>
slide7. Outline Motivation
Problem
Solutions
Exact Solution
Approximate Solution
Experiments
Conclusion 7/20 SIGMOD 2016<br>
slide8. Problem 8/20 SIGMOD 2016<br>
slide9. Problem Budget 9/20 SIGMOD 2016<br>
slide10. Outline Motivation
Problem
Solutions
Exact Solution
Approximate Solution
Experiments
Conclusion 10/20 SIGMOD 2016<br>
slide11. DP-based Solution 11/20 SIGMOD 2016<br>
slide12. Other Approximation 12/20 SIGMOD 2016<br>
slide13. Solutions 13/20 SIGMOD 2016 Summary<br>
slide14. Outline Motivation
Problem
Solutions
Exact Solution
Approximate Solution
Experiments
Conclusion 14/20 SIGMOD 2016<br>
slide15. Experiment Setting 15/20 SIGMOD 2016 1. http://finance.yahoo.com/q/hp?s=AIP.L+Historical+Prices<br>
slide16. Comparison 16/20 SIGMOD 2016 Constraint based
SCREEN1
Different Algorithm
DP: Exact solution
DPC: Constant factor approximation
DPL: Linear time approximation
QP: Continuous probabilistic distribution approximation
SG: Simple greedy Song, Shaoxu, et al. "SCREEN: Stream Data Cleaning under Speed Constraints." Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. ACM, 2015.<br>
slide17. Analysis-STOCK 17/20 SIGMOD 2016 Scalability
Budget<br>
slide18. Outline Motivation
Problem
Solutions
Exact Solution
Approximate Solution
Experiments
Conclusion 18/20 SIGMOD 2016<br>
slide19. Conclusion Repair
Precisely handle large spike errors
Small errors can be detected and repaired
Various methods accustomed in different situations
Better performance in both repairing and application accuracies compared to the state-of-art data constraint-based repairing 19/20 SIGMOD 2016<br>
slide20. Q & A Thanks！ 20/20 SIGMOD 2016<br>