Mastering Cloud Computing - PowerPoint Presentation

sherrill-nordquist . @sherrill-nordquist

448 views
Uploaded On 2017-04-10

Mastering Cloud Computing - PPT Presentation

Rajkumar Buyya Christian Vecchiola Thamarai Selvi Chapters Part I Foundations Chapter 1 Introduction Chapter 2 Principles of Parallel and Distributed Computing Chapter 3 Virtualization ID: 535995

data cloud storage service cloud data service storage management application user applications file server amp task amazon services aneka

Link:

Copy

Embed:

<iframe width="560" height="315" src="https://www.docslides.com/embed/535995" frameborder="0" allowfullscreen></iframe>

Download Presentation from below link

Download Presentation The PPT/PDF document "Mastering Cloud Computing" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Presentation Transcript

Slide1

Mastering Cloud Computing

Rajkumar Buyya, Christian Vecchiola, Thamarai SelviSlide2

Chapters

Part I: FoundationsChapter 1 — IntroductionChapter 2 — Principles of Parallel and Distributed ComputingChapter 3 — Virtualization Chapter 4 — Cloud Computing Architecture Part II: Cloud Application Programming and the Aneka Platform Chapter 5 — Aneka: Cloud Application Platform Chapter 6 — Concurrent Computing: Thread Programming

Chapter 7 — High-Throughput Computing: Task ProgrammingChapter 8 — Data Intensive Computing: Map-Reduce Programming

Part III: Industrial Platforms and New Developments

Chapter 9 — Cloud Platforms in Industry

Chapter 10 — Cloud Applications

Chapter 11 — Advanced Topics in Cloud ComputingSlide3

Chapter 1 - IntroductionSlide4

I need to grow my infrastructure, but I do not know for how long…

I cannot invest in infrastructure, I just started my business….

I want to focus on application logic and not maintenance and scalability issues

I want to access and edit my documents and photos from everywhere..

I have a surplus of infrastructure that I want to make use of

I have a lot of infrastructure that I want to rent …

I have infrastructure and middleware and I can host applications

I have infrastructure and provide application services

Global Cloud MarketplaceSlide5
Slide6

Compute

Storage

Applications

Development and Runtime Platform

Private Resources

Cloud Manager

Private Cloud

Private Cloud (Government)

Public Clouds

Government Agencies

Organization Personnel

All users, on any deviceSlide7

Cloud Deployment Models

Private/Enterprise

Clouds

* A public Cloud

model

within

a company’s

own Data

Center

infrastructure for

internal and/or

partners use.

Public/Internet

Clouds

* 3rd

party,

multi-tenant Cloud

infrastructure

& services:

* available on

subscription

basis to all.

Hybrid/Inter

Clouds

* Mixed

usage of

private and public

Clouds: Leasing public

cloud services

when private cloud

capacity is

insufficientSlide8

Virtualization

IT outsourcing

Pay as you go

Provisioning

on demand

Elasticity

Utility computing

Scalability

IaaS

PaaS

SaaS

Virtual Datacenters

Green computing

Internet

No capital investments

Quality of Service

Cloudbusrting

Security

Privacy & Trust

IT outsourcing

Service Level Agreement

Billing

Cloud Computing?Slide9

Runtime Environment for Applications

Development and Data Processing Platforms

Examples: Windows Azure,

Hadoop

, Google

AppEngine

, Aneka

Platform as a Service

Virtualized Servers

Storage and Networking

Examples: Amazon EC2, S3,

Rightscale

vCloud

Infrastructure as a Service

End user applications

Scientific applications

Office automation, Photo editing,

CRM, and Social Networking

Examples: Google Documents,

Facebook

Flickr

Salesforce

Software as a Service

Web 2.0 InterfacesSlide10

1950

1960

1970

1980

1990

2000

2010

Mainframes

Clusters

1999

Grid Computing

Grids

Clouds

1966

: Flynn’s Taxonomy

SISD

SIMD

MISD

MIMD

1969

: ARPANET

1970

: DARPA’s

TCP/IP

1984: DEC’sVMScluster1984: IEEE 802.3Ethernet & LAN1975: Xerox PARCInvented Ethernet

1990: Lee-Calliau WWW, HTTP, HTML2004: Web 2.02005: Amazon AWS (EC2, S3)1960: Cray’s First Supercomputer 2010: Microsoft Azure1997: IEEE 802.11 (Wi-Fi)1989: TCP/IPIETF RFC 1122

2007: Manjrasoft Aneka2008: Google AppEngine1951: UNIVAC I, First MainframeSlide11

Chapter 2 – Parallel and Distributed ComputingSlide12

Compilers, PVM, MPI, ….

Parallel Programming Environments

Cluster Middleware:

Single System Image and Availability Infrastructure

Network Interface HW

Operating System

Common

SW & Applications

PC / Workstation

Network Interface HW

Operating System

Common

SW & Applications

PC / Workstation

Network Interface HW

Operating System

Common

SW & Applications

PC / Workstation

Network Interface HW

Operating System

Common

SW & Applications

PC / Workstation

High speed network connection

Sequential Applications

ParallelApplicationsSlide13

Computer and network hardware

Hardware

Support for heterogeneous resource sharing, communication, and programming environments for application development

Middleware

User interface for presentation

Applications

Distributed System Stack

Execution platform including network connectivity services

Operative SystemSlide14

Paradigms /

Architectural Models

Message Based

Communication

Client - Server

Object Space

RPC

Distributed Objects

Peer –to-Peer

Java Spaces

Java RMI,

Remoting

, CORBA, ORBIX.DCOM, COM+

RPC / Specific Tasks

MPI Programming, JMS, MSMQ, MQS

Istant

Messaging,

Video Conferencing

DistributedAgents

Mobile AgentsSlide15

1940

1950

Applications

Parallel Era

Sequential Era

Compilers

Architectures

1960

1970

1980

1990

2000

2010

2020

2030

Applications

Problem Solving Environments

Compilers

Architectures

Problem Solving EnvironmentsSlide16

Processor

Data Input

Data Output

Instruction StreamSlide17

Processor N

Data Output N

Single Instruction Stream

Processor 2

Processor 1

Data Input 1

Data Input 2

Data Input N

Data Output 2

Data Output 1Slide18

Processor N

Instruction

Stream 1

Processor 2

Processor 1

Single Data Input Stream

Single Data Output Stream

Instruction

Stream 2

Instruction

Stream NSlide19

Processor N

Instruction

Stream 1

Processor 2

Processor 1

Instruction

Stream 2

Instruction

Stream N

Data Input 1

Data Input 2

Data Input N

Data Output 1

Data Output 2

Data Output 3Slide20

Core 2

Cache L1

Processor 1

Global System Memory

Memory Bus

Processor 2

Processor N

Processor 1

Local

Memory

Memory Bus

Processor 2

Local

Memory

Memory Bus

Processor 2

Local

Memory

Memory Bus

IPC Channel

IPC ChannelSlide21

Large Level

(Processes, Tasks)

Task 1

Task 2

Task N

Function 1

Function 2

Function J

Statements

load

unction f1()

{…}

unction f2()

{…}

unction

()

{…}

[0] = …

b[0] = …

a[1] = …b[1] = …

a[k] = …b[k] = …Shared MemoryShared MemoryMessages IPCMessages IPCMedium Level(Threads, Functions)Fine Level(Processor, Instructions)

Very Fine Level(Cores, Pipeline, Instructions)Slide22

Hardware

Operating System

Middleware

Applications

Networking and Parallel Hardware

IPC primitives for control and data.

Frameworks for distributed programmingSlide23

Hardware

and OS (

IaaS

)

Middleware (

PaaS

)

Applications (

SaaS

)

Social Networks, Scientific Computing, Enterprise Applications

Virtual hardware, networking, OS images, and storage.

Frameworks for

Cloud Application DevelopmentSlide24

Client-server

client

Server/client

server

Server/client

server

client

Server/client

server

client

server

request

response

Two Tier

(Classic Model)

Three Tier

N Tier Slide25

P2P

peer

peerSlide26

RPC

RPC Service

Main Procedure

Procedure A

Procedure B

Procedure C:Node B

RPC Library

Node A

Program A (RPC Client)

Procedure C

Node B

Program C (RPC Server)

Procedure Registry

Parameters Marshaling and Procedure Name

Return Value

Marshaling

Parameters Unmarshaling and Procedure Name

Return Value

Unmarshaling

NetworkSlide27

DOF interaction

Instance

Object Proxy

Remote Reference Module

Node A

Application A

Node B

Application B

Network

Remote Reference Module

Object Skeleton

Remote

Instance

1: Ask for Reference

5: Object Activation

21Slide28

Application

Client

Invocation

Query

Response (WSDL)

Web

Server

UDDI Registry

WSDL(s)

Web

Server

WSDL

Web Service

Application

WSDL

Web Service

Application

ClientSlide29

WS technology Stack

Web Service Flow

Service Discovery

Service Description

XML-based Messaging

Network

Security

Service Publication

Quality of Service

Management

WSDL

SOAP

HTTP,

FTP,e

-mail, MQ, IIOP, ….

Direct

 UDDI

Static

 UDDI

WSFLSlide30

SOAP Messages

Envelope

POST /

InStock

HTTP/1.1

Host: www.stocks.com

Content-Type: application/

soap+xml

; charset=utf-8

Content-Length: <Size>

<?xml version=“1.0”>

soap:Envelope

xmlns:soap

=“http//www.w3.org/2001/12/soap-envelope”

soap:encondingStyle

=“

http//

www.w3.org/2001/12/soap-encoding”>

<soap:Header></soap:Header

> <soap:Body xmlns:m

=http://www.stocks.org/stock> <m:GetStockPrice>

<m:StockName>IBM<m:StockName>

</m:GetStockPrice> </

soap:Body></soap:Envelope>Header: Metadata & AssertionsBody: Method CallSlide31

SOAP Messages

Envelope

POST /

InStock

HTTP/1.1

Host: www.stocks.com

Content-Type: application/

soap+xml

; charset=utf-8

Content-Length: <Size>

<?xml version=“1.0”>

soap:Envelope

xmlns:soap

=“http//www.w3.org/2001/12/soap-envelope”

soap:encondingStyle

=“

http//

www.w3.org/2001/12/soap-encoding”>

<soap:Header></soap:Header

> <soap:Body xmlns:m

=http://www.stocks.org/stock> <m:GetStockPriceResponse>

<m:Price>34.5<m:Price>

</m:GetStockPriceResponse> </

soap:Body></soap:Envelope>Header: Metadata & AssertionsBody: Execution ResultSlide32

Chapter 3 – VirtualisationSlide33

Virtualization Layer

Virtual Hardware

Virtual Networking

Virtual Storage

Software Emulation

Host

Physical Hardware

Physical Storage

Physical Networking

Guest

Applications

Virtual ImageSlide34

Operative Systems

Execution Stack

Hardware - level Virtualization

Hardware

Programming Languages

Applications

OS- level Virtualization

Programming Language level Virtualization

Application - level VirtualizationSlide35

Aggregation

Sharing

Emulation

Isolation

Virtualization

Physical Resources

Virtual ResourcesSlide36

Virtualization

Execution Environment

Storage

Network

….

Emulation

High-Level VM

Multiprogramming

Hardware-assisted

Virtualization

Process Level

System Level

Paravirtualization

Full Virtualization

How it is done?

Technique

Virtualization Model

Application

Programming Language

Operating System

Hardware

Partial VirtualizationSlide37

Libraries

API

ABI

Hardware

Operative System

ISA

Applications

Operative System

Hardware

Libraries

Applications

API calls

System calls

ISA

User

ISA

User

ISASlide38

Ring 3

Ring 2

Ring 1

Ring 0

Least privileged mode

(user mode)

Privileged modes

Most privileged mode

(supervisor mode)Slide39

Host

VMM

Virtual Machine

binary translation

instruction mapping

interpretation

……

Guest

In memory

representation

Storage

Virtual Image

Host emulationSlide40

ABI

Hardware

Operative System

ISA

Virtual Machine Manager

ISA

Hardware

ISA

Virtual Machine Manager

ISA

VMSlide41

Virtual Machine Manager

ISA

Virtual Machine Instance

Instructions (ISA)

Interpreter

Routines

Interpreter

Routines

Allocator

DispatcherSlide42

User Instructions

Sensitive Instructions

Privileged InstructionsSlide43

Server A

(running)

Server B

(running)

Virtual Machine Manager

Server A

(running)

Server B

(inactive)

Virtual Machine Manager

Before Migration

After MigrationSlide44

Xen

Hypervisor (VMM)

Memory management CPU state registers

Devices I/O

User Domains (Domain U)

Guest OS

Modified codebase

Hypercalls

into

Xen

VMM

User Applications

(unmodified ABI)

Management Domain (Domain 0)

VM Management

HTTP interface

Access to the

Xen

Hypervisor

Ring 3

Ring 2

Ring 1

Ring 0

Hardware (x86)

Privileged instructions

Hardware trapSlide45

Hypervisor

Binary translation

Instruction caching

Guest Operating System

Unmodified codebase

VMM unaware

User Applications

(unmodified ABI)

Ring 3

Ring 2

Ring 1

Ring 0

Hardware (x86)

Hardware trap (sensitive instructions)

Dynamic / cached translation

(sensitive instructions)Slide46

Hardware (x86)

Host Operating System

VMware Hypervisor (VMM

)

Direct access to hardware

I/O, memory, networking for guests

Save/Restore CPU state for host OS

VMware Driver

Virtual Machine Instance

User Applications

VMware Workstation

Guest Operating System

User Applications

I/OSlide47

Hardware (x86)

Host Operating System

VMware Hypervisor (VMM

)

Direct access to hardware

I/O, memory, networking for guests

Save/Restore CPU state for host OS

VMware Driver

VM Instance

serverd

(daemon)

VMware

Web Server

VM Instance

VM InstanceSlide48

Hardware

VMkernel

hostd

VMX

CIM broker

User world API

Resource

scheduling

Device drivers

Storage stack

Network stack

Distributed VM file system

Virtual Ethernet adapter and switch

VMM

VMX

DCUI

syslog

vxpa

SNMP

Third-party CIM plug-insSlide49

Server

ESXi

ESX

vSphere

Server

ESXi

ESX

vSphere

Data Center

vCenter

Server

ESXi

ESX

vSphere

Server

ESXi

ESX

vSphere

Data Center

vCenter

vCloud

Cloud

Infrastructure

Virtualization

vFabric

Platform

Virtualization

Zimbra

Application

VirtualizationSlide50

Latest Pictures.

Hardware (x86)

Hypervisor

(Ring -1)

Hypercalls

MSRs

APIC

Scheduler

Address Management

Partition

Management

Root / Parent Partition

VMWPs

VMMS

WMI

Hypervisor-aware

ernel (Ring 0)

VSPs

VID

WinHv

I/O Stack

Drivers

VMBus

Enlightened Child Partition

User Applications

(Ring 3)

Hypervisor-aware

Wndows Kernel (Ring 0)VSCs / ICsWinHvI/O StackDrivers

VMBusEnlightened Child PartitionUser Applications(Ring 3)Hypervisor-aware Linux Kernel (Ring 0)VSCs / ICsLinuxHv

I/O StackDriversVMBusUnenlightened Child PartitionUser Applications(Ring 3)Hypervisor-unaware Kernel (Ring 0)

Processor

MemorySlide51

Chapter 4 – Cloud Computing ArchitectureSlide52

Virtual Machine (VM), VM Management and Deployment

QoS

Negotiation, Admission Control,

Pricing,

SLA Management, Monitoring, Execution Management, Metering, Accounting

Cloud Hosting Platforms

Web 2.0,

Mashups

, Concurrent and Distributed Programming, Workflows , Libraries, Scripting

Cloud Programming Environment and Tools

User

Applications

Social Computing, Enterprise ISV, Scientific Computing, CDNs

Cloud Applications

Cloud Resources

Autonomic Cloud Economy

Adaptive Management

User-level

Middleware

Core

Middleware

System

Infrastructure

SaaS

PaaS

IaaS

(M)

Pure

PaaS

Pure

SaaSSlide53

Physical Infrastructure

Datacenter

Cluster

Desktop /

Heterogeneous Resources

Third Party

IaaS

Cloud

Infrastructure Management Software

Scheduling

Pricing / Billing

Monitoring

QoS

SLA Management

Reservation

VM Pool Management

VM Image Repository

Provisioning

Web Services, Portals, REST API

Web based Management InterfaceSlide54

Programming API / Libraries

Physical Infrastructure

IaaS

Providers

PaaS

Core Middleware

Resources Management

Application

Management

User

Management

QoS

/ SLA Management &

Billing

Elasticity & Scaling

Runtime

Web Services, Portals, REST API

Web based InterfaceSlide55

DataSynapse

Zimory

Pools,

Elastra

CloudServer

, Aneka, ….

Platform as a Service Solutions

PaaS

IaaS

(M)

Datacenters, Clusters, Desktop Grids

Physical Infrastructure

KVM,

Xen

VMWare

, …

Virtual Machine Technology

Eucalyptus,

OpenNebula

VMWare

vCloud

, OpenPEX, InterGrid, … Infrastructure Management SoftwareSlide56

Datacenter

Public Cloud

Cluster

Desktop Grids /

NOWs

Public Cloud

Dynamic

Provisioning

IaaS

(M)

Paas

Private Cloud

Hybrid / Heterogeneous CloudSlide57

Private

Users

Public Services

Industries

Enterprises

Party Cloud

Public Cloud

Private Cloud

Public Cloud

Community Cloud

Federal and

Government Bodies

Development

Platforms

Infrastructure

Application

ServicesSlide58

Chapter 5 – Aneka

: Cloud Application Platform Slide59

Multiple Infrastructures

Multi-core

Cluster

Grid

Cloud

Thread

Task

...

MapReduce

2100

Aneka

Multiple Applications

1. SDK

2. RuntimeSlide60

ECMA 334: .NET or Mono / Windows, Linux, & Mac

Infrastruscture

Enterprise Desktop Grid

Data Centers

Clusters

Public Cloud

Persistence and Security

Fabric Services

Hardware Profiling

Dynamic Resource Provisioning

Foundation Services

Billing & Reporting

Resource Reservation

Storage

Licensing & Accounting

Execution Services

Independent Bag of Tasks

Distributed Threads

MapReduce

other models…

Middleware - Container

Application Development & Management

SDK: APIs & Tools

Management: Tools, Interfaces and APIs

PAL – Platform Abstraction LayerSlide61

Aneka Repository

Aneka

Containers

Node

Manager

Node

Manager

Aneka

Containers

Aneka

Containers

Node

Manager

Node

Manager

Aneka

Containers

Management Console

update

HTTP

File Share

……Slide62
Slide63
Slide64
Slide65

ECMA 334: .NET or Mono / Windows, Linux, & Mac

Infrastruscture

Enterprise Desktop Grid

Data Centers

Clusters

Public Cloud

Persistence & Security

Fabric Services

Hardware Profiling

Resource Provisioning

High-Availability

Membership

Foundation Services

Billing & Reporting

Resource Reservation

Storage

Licensing & Accounting

Application Services

Bag of Tasks

Distributed Threads

MapReduce

Other models…

PSM

Middleware - Container

Application Development & Management

Software Development Kit: APIs & Tools

Management: Tools, Interfaces and APIs

PAL – Platform Abstraction LayerSlide66

Reservation

Provisioning

Reporting

Accounting

Mandatory

Index (master)

Scheduling

Failover

Mandatory

Index (slave)

Storage

Storage Node

Allocation

Mandatory

Index (slave)

Execution

Worker Node

Reservation

Provisioning

Reporting

Accounting

Mandatory

Index (master)

Scheduling

Master Node

Allocation

Mandatory

Index (slave)

Execution

Worker Node

Allocation

MandatoryIndex (slave)ExecutionWorker NodeMandatory

Index (slave)StorageStorage NodeSlide67

Dynamic Resources

Amazon EC2

Storage Service

Provisioning

Service

Task Scheduling Service

Reporting, Billing,

Accouting

Master Node

Slave Node

Task Execution Service

Static Resources

AWSSlide68

Provisioning

Service

Application

Management & Scheduling

Reporting, Billing,

Accouting

Desktops & Workstations

Low priority workloads

High

Performance

and

QoS

Public Clouds

Cluster Resources

Aneka Hybrid Cloud

QoSSlide69

Resource Provisioning

Application

Management &

Scheduling

Resource Reservation

Master Node

Desktop Machines

Virtual Clusters

Workstations

ClustersSlide70

Resource Provisioning

Application

Management &

Scheduling

Master Node

Reporting,

Billing,

Accouting

Amazon EC2 Instance

Amazon EC2 Instances

Slave Nodes

EC2 BoundariesSlide71

Provisioning

Service

Application

Management & Scheduling

Reporting, Billing,

Accouting

Public Clouds

Desktops & Workstations

Clusters

Virtual Cluster Resources

Resource

Reservation

Master Node

Local InfrastructureSlide72

Unknown

Initialized

Running

Paused

Stopped

Pausing

Resuming

Starting

Stopping

IService.Start

()

IService.Stop

()

IService.Continue

()

IService.Pause

()

<Error>Slide73

Chapter 6 – Concurrent

Computing: Thread ProgrammingSlide74

CPU: single die

Core 1

Core 2

Core N

Cache L1

Cache L2

To MemorySlide75

Execution Timeline

Thread Local Storage

Instructions

(program counter)

Thread

Main Thread

Process

Shared MemorySlide76

Process

Units of work

Result

Process

Units of work

Result

a. Embarrassingly parallel

b. Inherently sequentialSlide77

4 x 6

x 3

4 x 3

2,1Slide78

Process

Units of work

Result

Unit 1

Unit 2

Unit 5

Unit 4

Unit 3Slide79

Unstarted

Running

Stopped

Wait

SleepJoin

Start()

Interrupt()

Abort()

Suspended

Suspend

Requested

Suspend()

Resume ()

Abort

Requested

Wait()/Sleep()/Join()

Normal Completion

Unstarted

Started

Running

Completed

StagingIn

Queued

Staging

Out

Aborted

Rejected

Failed

Start()

System.Threading.Thread

life cycle.b. Aneka.Threading.AnekaThread life cycle.Slide80

Chapter 7 – High-Throughput Computing: Task ProgrammingSlide81

Middleware

Access

Node(s)

Computing Nodes

Local Node

Tasks

Remote EnvironmentSlide82

4215

6810

MPI_COMM_WORLD

Group A

Group B

Communicator A

Communicator BSlide83

MPI Include File

MPI Environment Initialization

MPI Environment Shutdown

Prototypes declaration

Do Work and Message Passing

Do Work

Serial Code

Parallel Code

Code Section

Header

SectionSlide84

Workflow Application Modeling & Definition Tools

Workflow

Specification

Task + Data Scheduler

Data Movement

Data Provenance

Dependency Manager

Runtime

Manager

Language Interpreter

Fault Management

Workflow Enactment Engine

Middleware Supporting Clusters, Grids, and Clouds

Distributed Computing Resources

Interaction with Distributed Resources

Interaction with Information Services

Users

Workflow Design & Definition

Workflow Execution & Control

Resource Info Service

Application Info Service

Information Services

Run Time

Build TimeSlide85

Local Node

(… more tasks)

AnekaApplication

Task1 :

ITask

Task2 :

ITask

AnekaTask

Index

Node

Membership

Catalogue

StorageService

TaskScheduler

TaskExecution

Service

Aneka Cloud

TaskManagerSlide86

Client Machine

File dependencies

Index

Node

Membership

Catalogue

StorageService

TaskScheduler

TaskExecution

Service

Aneka Cloud

Web Service

TaskManager

AnekaTask

Remote FTP

Result Files

Input/Output

FilesSlide87

IJobManager

TaskManager

AnekaApplication

AnekaTask

TaskManager

PSM Model

PSMJobInfo

PSMCommandInfo

PSMParameterInfo

PSMFileInfo

PSMContext

AnekaTask

ITask

Aneka

ApplicationModelSlide88

Offspring Environment

IDistributionEngine

AppInstance

IOffspringPlugin

DefaultPlugin

LogPlugin

AutoPlugin

Strategy

Controller

ITaskStrategy

Workflow Logic 1

Workflow Logic N

TaskStrategyBase

Local

Engine

AnekaEngine

DistributionEngineBase

TaskManager

Aneka CloudSlide89

StrategyController

Control Thread

(Strategy Management)

Monitoring Thread

(Feedback from Distribution Engine)

Init Application

Init Strategy

Submit Tasks

Wait

Release Strategy

Shutdown Application

Yes

Complete or Stop?

Task Feedback

Task Failed?

Invoke

OnSuccess

Invoke

OnFailed

Yes

New Task?

Signal

Iteration Complete?

Yes

Submit

Yes

NoSlide90

Chapter 8 – Data Intensive Computing: Map-Reduce ProgrammingSlide91

Data/compute

intensive problems

Compute

intensive problems

Current Problems

Data

intensive problems

Heterogeneous Formats

Distributed Petabytes

Data Complexity

(volume/format/distribution)

Homogeneous Formats

Non-distributed Megabytes or Gigabytes

Computational

Complexity

Model Solvers

Simulations

Simple search

Statistical Models

Decision support

Knowledge generationSlide92

Compute Site

Instrument

Storage Facility

Compute Site

Scientist

Instrument

Scientist

Storage Facility

Information & Discovery

Compute SiteSlide93

Storage Peer

Key Ring

Pluggable Storage Engine (BDB,

MySQL

)

Failure and Membership

Detection

Storage Peer

Request

Coordinator

Storage Peer

Storage PeerSlide94

Tablet Server

Master Server

Chubby

GFS

Logs

Tablet Files

Logs

Tablet Files

Logs

Tablet Files

Logs

Tablet FilesSlide95

MapReduce Infrastructure / Runtime

Map

Task

Reduce

Task

Map

Task

Map

Task

Map

Task

Map

Task

Reduce

Task

Aggregation by Key

Input

Output

Input Data File

Output Data File

Distributed File SystemSlide96

Worker

Master

Split 0

Split 1

Split 2

Split 3

Split 4

Output File 0

Output File 1

User Program

) fork

(2) assign map

(2) assign reduce

(3) read

(4) local

write

(5) remote read

(6) write

nput files

ap phase

reduce phase

output files

ntermediate

filesSlide97

Stage

Input File

Output File

Vertices (processes)

ChannelsSlide98

Aneka Cloud

MapReduce

Scheduling Service

MapReduceApplication

Reducer

Mapper

Input Data File

Client Machine

MapReduce

Execution Service

Map

Task

Reduce

Task

MapReduce

Execution Service

Map

Task

Reduce

Task

MapReduce

Execution Service

Map

Task

Reduce

Task

DFS Implementation

Split 0Split 1Split 2

Split 3Split 4

intermediate filesOutput File 0Output File 1Slide99

MapReduceSchedulerService

MapReduceScheduler

upload queue

initialization queue

ispatch queue

failure queue

resubmission queue

reporting queue

upload thread

initialization thread

dispatch thread

eport thread

DFS Implementation

Message translation

and Event Triggering

Aneka Middleware

MapReduceApplication

MapReduce

Execution ServicesSlide100

MapReduceExecutorService

ExecutorManager

job and task

information

ommand thread

report thread

ompletion thread

DFS Implementation

Message translation

and Event Triggering

Aneka Middleware

MapReduce

Scheduling Service

MapReduceExecutor

Map Task

MapReduceExecutor

Reduce Task

Sandbox

ommand

queueSlide101

Header: 3 bytes magic number, 1 byte version

…….

Record Block:

Integer (4 bytes)

Length of the block

Integer (4 bytes)

Length of the key

Key byte array

Value byte arraySlide102
Slide103
Slide104

User Requests

Wireless / Mobile 3G Network

ECG Sensor

ECG Sensor Module

Embedded Bluetooth Enabled

Data Communication & Processor Module

Bluetooth connectivity

IaaS

aaS

Dynamically Scalable Runtime

(

QoS

-based scaling of computing resources)

Security Runtime

SaaS

ECG Data Analysis as a Service

User Requests

User

(detailed view)

Large Number

5Slide105

Jeeva

Portal

A : BLAST

B : Create Data Vector

C : HH Classifier

D : SS Classifier

E : TT Classifier

F : HS Classifier

G : ST Classifier

H : TH Classifier

: Predict Final Secondary

Structure

Initial Phase

Classification

Phase

Final Phase

Task Graph

AnekaSlide106

Chapter 9 – Cloud Platforms in IndustrySlide107

Communication Services

Amazon Simple Queue Service (SQS)

Amazon Simple Notification Service (EBS)

Amazon Simple Email Service (SES)

Amazon Route 53

Amazon Virtual Private Cloud (VPC)

Amazon Direct Connect

Amazon Elastic Load Balancing

Storage Services

Amazon Simple Storage Service (S3)

Amazon Elastic Block Store (EBS)

Amazon

ElastiCache

Amazon

SimpleDB

Amazon Relational Database Service (RDS)

Amazon

CloudFront

Amazon Import/Export

Compute Services

Amazon Elastic Compute Cloud (EC2)

Amazon Elastic

MapReduce

AWS Elastic Beanstalk

AWS

Cloudformation

Autoscaling

Amazon AWS Platform

Additional Services

Amazon

GovCloud

Amazon Flexible Payment Service (FPS)

Amazon

DevPay

Amazon

Fullfillment

Web Service (FWS)

Amazon Mechanical Turk

Alexa

Web Information Service

Amazon

CloudWatch

Alexa

Top SitesSlide108

Development

Runtime

Local Machine

Python SDK

Google

AppEngine

Infrastructure

Sandboxed Runtime Environment

Data Store

Url

Fetch

Image Manipulation

Task Queue

Cron

Jobs

Web App

Web AppSlide109

Compute

Enterprise Level Infrastructure

Storage

Networking

Identity

Marketplace

Management portal

Windows Azure Platform

Local Development Environment

Performance

Development Tools

Windows Azure Compute

Windows Azure Storage

Windows Azure Connect

Content Delivery Network (CDN)

AppFabric

Caching

AppFabric

Service Bus

AppFabric

Integration

AppFabric

Access Control

Tools

Windows Azure

SDK

AppFabric

SDK

SQL Azure

DataMarket

Applications

Marketplace

SQL Azure

Windows Azure

AppFabric

Windows AzureSlide110

Infrastructure Layer

Platform Layer

Tabular Data Stream (TDS)

TDS + Secure Socket Layer

SQL Server

SQL Azure Fabric

Management Services

SQL Server

SQL Azure Fabric

Management Services

SQL Server

SQL Azure Fabric

Management Services

Services Layer

Provisioning

Billing and Metering

Connection Routing

Provisioning

Billing and Metering

Connection Routing

Provisioning

Billing and Metering

Connection Routing

ODBC

ADO.NET

PHP

SQL Server Applications and Tools

WCF Data Services

HTTP / REST

Client Layer

Client Premises or Windows Azure PlatformSlide111

Chapter 10 – Cloud ApplicationsSlide112

Aneka

Cloud-

CoXCS

(Task orchestrator)

Elastic Sizing (EC2,…, )

CoXCS

Tasks

DatasetsSlide113

Local Storage

Aneka

Private Cloud

Public Cloud

Archiving

Distribution

Portal (

SaaS

)Slide114

Force.com

Platform

Metadata Cache

Bulk Processing Engine

Multitenant-aware Query

Optimiser

Runtime Application Generator

Full-text Search Engine

Shared Database

Data Tables

Pivot Tables

Metadata Tables (UDD)

Virtual Application Components

COMMON APPLICATION SCREENS

TENANT-SPECIFIC

SCREENS

OBJECTS (TABLES)

Indexes

User-base 3: Application 3

User-base 1: Application 1

User-base 2: Application 2Slide115

User

EyeOS

Server

eyeos

apps

extern

extras

system

users

workgroups

Client-side scriptsSlide116

Scalability engine

EC2 instances: video rendering

EC2 instances:

web front end

Amazon SQS

Amazon S3:

Video storageSlide117

Aneka

Networked LAN

Aneka

MayaRender

Maya: Train Design

Render Files

Scenarios

Render TasksSlide118

Gamer

Titan Gaming Portal

Game Server

Aneka

Log

Processing Task

Log

Processing Task

Log

Processing TaskSlide119

Dropbox

Folder

Dropbox

Folder

Dropbox

Folder

Automatic

synchronizationSlide120

Chapter 11 – Advanced Topics in Cloud ComputingSlide121

Reducing Power Cost and Carbon Emission

Reducing Power Cost and Maximizing Revenue

Revenue

Power Consumption

Cloud Data Center

User Applications

Green Cloud SolutionSlide122

Users

Brokers

Compute Cloud

Pool Node

Cluster (VM Pool)

Compute Cloud

Storage Cloud

Cloud Exchange

Mastering Cloud Computing - PowerPoint Presentation

Mastering Cloud Computing - PPT Presentation

Share:

Link:

Embed:

Related Contents