"Sequential Data Cleaning: A Statistical Approach" is the property of its rightful owner. Permission is granted to
download and print the materials on this website for personal, non-commercial use only, and to display it
on your personal computer provided you do not modify the materials and that you retain all copyright
notices contained in the materials. By downloading content from our website, you accept the terms of this
agreement.
Presentation Transcript
01
Sequential Data Cleaning: A Statistical Approach Aoqian Zhang1, Shaoxu Song1 , Jianmin Wang1
Stream Data Erroneous Stream data are often dirty
Unreliable sensor reading
Large spike and small errors
Stock and Flight
Accuracy of Stock in Yahoo! Finance is 0.93[1]
Accuracy of Travelocity is 0.95[1]
Reasons
Ambiguity in information extraction
Pure mistake 3/20 SIGMOD 2016 [1] X. Li, X. L. Dong, K. Lyons, W. Meng, and D. Srivastava. Truth finding on the deep web: Is the problem solved? PVLDB, 6(2):97-108, 2012.<br>
04
Data Cleaning 4/20 SIGMOD 2016 Constraint based methods
Constraints on speeds of value changes[1]
Minimum change principle
Large spike error: max/min values allowed
Small error: fail to identify [1] Song, Shaoxu, et al. "SCREEN: Stream Data Cleaning under Speed Constraints." SIGMOD’15<br>
Comparison 16/20 SIGMOD 2016 Constraint based
SCREEN1
Different Algorithm
DP: Exact solution
DPC: Constant factor approximation
DPL: Linear time approximation
QP: Continuous probabilistic distribution approximation
SG: Simple greedy Song, Shaoxu, et al. "SCREEN: Stream Data Cleaning under Speed Constraints." Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. ACM, 2015.<br>
