Machine Learning Module for Big Data Analysis in Kepler Mai H. Nguyen1, Daniel Crawl1, Jianwu Wang2, Ilkay Altintas1 1San Diego Supercomputer Center University of California, San Diego 2Department of Information Systems University of
"Machine Learning Module for Big Data Analysis in" is the property of its rightful owner. Permission is granted to
download and print the materials on this website for personal, non-commercial use only, and to display it
on your personal computer provided you do not modify the materials and that you retain all copyright
notices contained in the materials. By downloading content from our website, you accept the terms of this
agreement.
Presentation Transcript
01
Machine Learning Module for Big Data Analysis in Kepler Mai H. Nguyen1, Daniel Crawl1, Jianwu Wang2, Ilkay Altintas1
1San Diego Supercomputer Center
University of California, San Diego
2Department of Information Systems
University of Maryland, Baltimore County<br>
02
Kepler Kepler is a scientific workflow system.
Built on Ptolemy II framework.
GUI-based, drag-and-drop approach to designing scientific workflows.
Environment for integrating disparate software components.
Framework for distributed execution.
Allows scientists with minimal computing knowledge to build workflows for scientific applications. www.kepler-project.org<br>
03
Sample Kepler Workflow<br>
04
Machine Learning ML techniques use data-driven approach for analyzing problems.
Goal: Build systems that learn from data, not explicitly programmed.
Sample applications:
Detecting weather patterns
Image recognition
Speech processing
Fraud detection<br>
05
Machine Learning in Kepler ML module in Kepler:
Python
ExternalExecution
RExpression
MLlib<br>
06
Big Data Analysis Spark
Platform for cluster computing
Distributed in-memory architecture
Provides fast and scalable processing of iterative operations, ideal for ML algorithms.
MLlib
Scalable ML library built on Spark<br>
07
Big Data Analysis in Kepler Spark and MLlib are incorporated into Kepler.
Actors implement MLlib algorithms
Use MLlib’s Java API to invoke scalable ML algorithms
Examples:
k-means clustering
random forest
support vector machine<br>
08
Workflows ML actors can be connected to other actors in Kepler to build complex workflows:<br>
09
ML Algorithm with Different Implementations Build actor for single ML algorithm based on different implementations
ExecutionChoice actor
One functionality
Several implementations
Advantage:
Compare different implementations of same technique or algorithm without having to change workflow.
Example: kmeans-all
Implements k-means clustering on 4 different platforms: R, MLlib, Mahout, KNIME<br>
10
kmeans-all Top Level<br>
11
kmeans-all Shared Options<br>
12
kmeans-all – R K-means clustering implemented in R:<br>
13
kmeans-all – MLlib Workflow for Mllib<br>
14
kmeans-all – Mahout Workflow for Mahout<br>
15
kmeans-all – KNIME Workflow for KNIME<br>
16
kmeans-all – Key Points Actor provides ML functionality as implemented on different platforms.
Can be used compare accuracy and processing speed of different implementations of k-means.
Implementation choice can be changed with minimal changes to workflow.<br>
17
Summary ML module in Kepler provides way to incorporate machine learning functionality into scientific workflow.
ML actors
RExpression, Python
ExternalExecution
ExecutionChoice
One functionality with several implementations
Change implementation choice without changing workflow structure.<br>
