PPT-Anomaly detection of large scale distributed storage system based on machine learning

Institute of High Energy Physics CAS Wang Lu LuWangihepaccn Agenda Introduction Challenges and requirements of anomaly detection in large scale storage systems Definition and category of anomaly. Anomaly Detection for. Cyber Security. Presentation by Mike Calder . Anomaly Detection. Used for cyber security. Detecting threats using network data. Detecting threats using host-based data. In some domains, anomalies are detected so that they can be removed/corrected. Jimeng. Sun, . Huiming. . Qu. , . Deepayan. . Chakrabarti. & Christos . Faloutsos. Presented By. Bhavana. . Dalvi. Outline. Motivation. Problem Definition. Neighborhood formation. Anomaly detection. 2. /86. Contents. Statistical . methods. parametric. non-parametric (clustering). Systems with learning. 3. /86. Anomaly detection. Establishes . profiles of normal . user/network behaviour . Compares . Problem motivation. Machine Learning. Anomaly detection example. Aircraft engine features:. . = heat generated. = vibration intensity. …. (vibration). (heat). Dataset:. New engine:. Density estimation. Craig Buchanan. University of Illinois at Urbana-Champaign. CS 598 MCC. 4/30/13. Outline. K-Nearest Neighbor. Neural Networks. Support Vector Machines. Lightweight Network Intrusion Detection (LNID). &. Intrusion . Detection Systems. 1. Intruders. Three classes of intruders:. Examples of Intrusion. Performing a remote root compromise of an e-mail server. Defacing a Web server. Guessing and cracking passwords. Detection. Carolina . Ruiz. Department of Computer Science. WPI. Slides based on . Chapter 10 of. “Introduction to Data Mining”. textbook . by Tan, Steinbach, Kumar. (all figures and some slides taken from this chapter. © 2015 Blueocean Market Intelligence. 1. Implemented real time anomaly detection system for a leading PC manufacturer globally(1/2). © 2015 Blueocean Market Intelligence. 2. Client: . Leading PC Manufacturer in the US. On-Orbit Anomaly Research. NASA IV&V Facility. Fairmont, WV, USA. 2013 Annual Workshop on Independent Verification & Validation of Software. Fairmont, WV, USA. September 10-12, 2013. Agenda. September 10, 2013. Lecture Notes for Chapter 10. Introduction to Data Mining. by. Tan, Steinbach, Kumar. New slides have been added and the original slides have been significantly modified by . Christoph F. . Eick. Lecture Organization . Project Lead: . Farokh. . Bastani. , I-Ling Yen, . Latifur. Khan. Date: April 7, 2011. 2010/Current Project Overview. Self-Detection of Abnormal Event Sequences. 2. Tasks:. Prepare Cisco event sequence data for analysis tools.. “Anomaly Detection: A Tutorial”. Arindam. . Banerjee. , . Varun. . Chandola. , . Vipin. Kumar, Jaideep . Srivastava. , . University of Minnesota. Aleksandar. . Lazarevic. , . United Technology Research Center. Kai Shen, Christopher Stewart, . Chuanpeng Li, and Xin Li. 6/16/2009. SIGMETRICS 2009. 1. University of Rochester. Performance Anomalies. 6/16/2009. SIGMETRICS 2009. 2. Complex software systems (like operating systems and distributed systems):. Hierarchical Temporal Memory (and LSTM). Jaime Coello de Portugal. Many thanks to . Jochem. . Snuverink. Motivation. Global outlier. Level change. Pattern deviation. Pattern change. Plots from: Ted .