Grid Computing Definitions The term Grid computing originated in the early 1990s as a metaphor for making computer power as easy to access as an electric power grid The definitive definition of a Grid is provided ID: 723896
Download Presentation The PPT/PDF document "Grid Computing Net-535 Fall 2013" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Grid Computing
Net-535 Fall 2013Slide2
Grid Computing Definitions
The term
Grid computing
originated in the early 1990s as a metaphor for making computer power as easy to access as an electric power grid.
The definitive definition of a Grid is provided
by Ian Foster
in his article "What is the Grid?
Computing resources are not administered centrally.
Open standards are used.
Non-trivial quality of service is achieved.
IBM
: "A Grid is a type of parallel and distributed system that enables the sharing, selection, and aggregation of resources distributed across multiple administrative domains based on the resources availability, capacity, performance, cost and users' quality-of-service requirements" Slide3
What is Grid Computing
“
Grid computing, most simply stated, is distributed computing taken to the next
evolutionary level
. The goal is to create the illusion of a simple yet large and powerful self managing
virtual computer out of a large collection of connected heterogeneous systems sharing various combinations of resources” IBM Redbook .Slide4
Electrical Power Grid Analogy
Electrical power grid
users (or electrical appliances) get access to electricity through wall sockets with no care or consideration for where or how the electricity is actually generated.
“The power grid”
links together power plants of many different kinds
The
Grid Computing
users (or client applications) gain access to computing resources (processors, storage, data, applications, and so on) as needed with little or no knowledge of where those resources are located or what the underlying technologies, hardware, operating system, and so on are"the Grid" links together computing resources (PCs, workstations, servers, storage elements) and provides the mechanism needed to access them.
4Slide5
Properties of the Grid
• Distributed
• Dynamic
• Heterogeneous
• Virtual environment
• Collaborative environment• Transparent access to all the available resources
5Slide6
Who needs Grid Computing?
6
Not just computer scientists…
scientists “hit the wall”
when faced with situations:
The amount of data they need is huge and the data is stored in different institutions.
The amount of similar calculations the scientist has to do is huge.
Other areas:
Government
BusinessEducationIndustrial design etcSlide7
What grid computing can do(1)
Exploiting underutilized
resources:
The easiest use of grid computing is to run an existing application on a different
machine
The machine on which the application is normally run might be unusually busy due to an unusual
peak in activity
.
The job in question could be run on an idle machine elsewhere on the grid.There are at least two prerequisites for this
scenarioFirst, the application must be executable remotely and without undue overheadSecond, the remote machine must meet any special hardware, software, or resource requirements imposed by the applicationThe processing resources are not the only ones that may be underutilized also grid dataOften, machines may have enormous unused disk drive capacity. Grid computing, more specifically, a “data grid”, can be used to aggregate this unused storage into a much larger virtual data store
Another function of the grid is to better balance resource utilization. An organization may
have
occasional unexpected peaks of activity that demand more resources. If the
applications
are grid enabled, they can be moved to underutilized machines during such
peaksSlide8
What grid computing can do
Parallel CPU capacity
is one of the most attractive features of a
grid
Subjobs
on different machinesBarriers often exist to perfect scalability.Slide9
What grid computing can do
Applications
Grid-enabled applications
no practical tools for transforming arbitrary applications to exploit the parallel capabilities of a grid.Slide10
What grid computing can do
Virtual resources and virtual organizations for collaboration
Another important grid computing contribution is to enable and simplify collaboration among a
wider
audience
. Grid computing takes these capabilities to an even wider audience, while offering important standards that enable very heterogeneous systems to work together to
form
the image of a large virtual computing system offering a variety of virtual resources, as
illustratedSlide11
What grid computing can do
Access to additional resources
special equipment, software, licenses, and other
services
Some machines on the grid may have special devices
Resource balancing An unexpected peak can be routed to relatively idle machines in the grid. If the grid is already fully utilized, the lowest priority work being performed on the grid can
be
temporarily suspended or even cancelled and performed again later to make room for
the higher priority work.Slide12
What grid computing can do
Reliability
Now: redundancy in hardware
Future: Software
Utilize “autonomic computing”
Management
More disperse IT infrastructure
Priority among projectsSlide13
Grid concepts and components
Types of resources
Computation
Storage
Primary/secondary storage
Mountable networked filed systemAFS, NFS, DFS, GPFSCapacity increase (multiple machine )
Uniform name space
Data StrippingSlide14
Grid concepts and components(2)Types of resources (cont)
Communications
Redundant communication paths
Software and licenses
License management software
Special equipment, capacities, architectures, and policies
different architectures, operating systems, devices, capacities, and equipment.
Jobs and applications
Application is a collection of jobsSpecific dependenciesSlide15
Grid concepts and components(3)Types of resources (cont)
Scheduling, reservation, and scavenging
scheduler
automatically finds the most appropriate machine on which to run any given job
scavenging
report its idle status to the grid management node. This management node would assign to this idle machine the next job that is satisfied by the machine’s resources.
Reserved
Reserve
of resources in advance to improve the quality of serviceSlide16
Grid concepts and components(4)
Intragrid to Intergrid
cluster
same hardware/software
Intragrid
heterogeneous machines/softwaremultiple department/same organization
Intergrid
heterogeneous machines/software
multiple department/multiple organizationSlide17
Grid components
Management
components
First, there is a component that keeps
track
of the resources available to the grid and which users are members of the grid.Second, there are measurement components that determine both the capacities of the nodes on the grid and their current utilization rate at any given time.
Third, advanced grid management software can automatically manage many aspects of the
gridSlide18
Grid components
Donor
software
Each machine contributing resources typically needs to enroll as a member of the grid and
install
some software that manages the grid’s use of its resources. Usually, some sort of identification and authentication procedure must be performed before a machine can join the gridThe donor machine will usually have some sort of monitor that determines or measures how busy the machine is and the rate or amount of resources utilized. This
information
is “bubbled up” to the management software of the grid and used to schedule use
of those resources accordingly.Slide19
Grid components
Submission
software
Software to submit the job
Schedulers
Most grid systems include some sort of job scheduling software. This software locates a machine on which to run a grid job that has been submitted by a user. In the simplest cases, it may just blindly assign jobs in a round-robin fashion to the next machine matching the resource requirements. However, there are advantages to using a more advanced scheduler.
Some schedulers implement a job priority system. This is sometimes done by using several
job queues, each with a different priority. As grid machines become available to execute jobs,
the jobs are taken from the highest priority queues first. Policies of various kinds are also
implemented using schedulers. Policies can include various kinds of constrains on jobs, users, and resources. For example, there may be a policy that restricts grid jobs from executing at certain times of the day.Slide20
Grid components
Communications
A grid system may include software to help jobs communicate with each other.
For
example, an application may split itself into a large number of
subjobs. Each of these subjobs is a separate job in the grid
.
However, the application may implement an algorithm that requires that the
subjobs communicate some information among them. The subjobs
need to be able to locate other specific subjobs,establish a communications connection with them, and send the appropriate data. The open standard Message Passin Interface (MPI) and any of several variations is often included as part of the grid system for just this kind of communication.Slide21
Grid components
Observation, management, and measurement
We mentioned above the schedulers react to current loads on the grid.
Usually, the donor
software
will include some tools that measure the current load and activity on a given machine using either operating system facilities or by direct measurement. This software is sometimes referred to as a “load sensor.” Some grid systems provide the means for
implementing
custom load sensors for other than CPU
or storage resources.Slide22
Grid User Roles
---A User’s Perspective
22
Enrolling and installing grid software
Logging onto the grid
Queries and submitting jobs
Data configuration
Monitoring progress and recovery
Reserving resourcesSlide23
Grid User Roles
---An Administrator’s Perspective
23
Planning
Installation
Managing enrollment of donors and users
Certificate authority
Resource management
Data sharingSlide24
Using a grid: An application developer’s perspective(1)
Applications that are not enabled for using multiple processors but can be executed on different machines.
Applications that are already designed to use the multiple processors of a grid setting.
Applications that need to be modified or rewritten to better exploit a grid
Tools for debugging and measuring the behavior of grid applicationsSlide25
Using a grid: An application developer’s perspective(2)
Globus
developer’s toolkit
Manage grid operations
Measurement
Repair Debug grid applicationsOpen Grid Services Architecture (OGSA)Slide26
Grid Architecture
26
Application
Collective
Resource
Connectivity
Fabric
Application
Transport
Internet
Link
GRID
InternetSlide27
Grid Architecture
Grid Computing
27
Fabric layer:
Provides the resources to which shared access is mediated by Grid protocols.
Connectivity layer:
Defines the core communication and authentication protocols required for grid-specific network functions.
Resource layer:
Defines protocols, APIs, and SDKs for secure negotiations, initiation, monitoring control, accounting and payment of sharing operations on individual resources.
Collective Layer: Contains protocols and services that capture interactions among a collection of resources.Application Layer: These are user applications that operate within VO (Virtual Organization ) environment
.Slide28
Standards for Grid Environments
28
Global Grid Forum (GGF)
http://www.ggf.org
Organization for the Advancement of Structured
Information Standards (OASIS)
http://www.oasis-open.org/
World Wide Web Consortium (W3C)
http://www.w3.org/
Distributed Management Task Force (DMTF)http://www.dmtf.org/ Web Services Interoperability Organization (WS-I)http://www.ws-i.org/Slide29
Globus Toolkit v
5
The
Globus
Alliance is made up of organizations and individuals that develop and make available various technologies applicable to grid computing.
The Globus toolkit v5
includes software for security, information infrastructure, resource management, data management, communication, fault detection, and portability. It is packaged as a set of components that can be used either independently or together to develop applications.
For more information visit http://www.globus.orgSlide30
Challenges
Trust (security is built on trusted parties or trusted third-party CA)
Problem
: how to trust VO members and its agents (autonomous apps)
Solution: proxy credentials provided by a CA in public key infrastructureSharing of applications and data
Problem
: incompatible machines and OS, need to limit access Solution
: virtualization, Grid resource allocation policiesCommunication of Grid policies and metadata: Grid interoperability Problem: incompatible protocols Solution: XML-based protocols and open standardsReliability and robustness (a non-functional requirement) Problem: Grid-based systems can be brittle (network connections) Solution: two-phase commit, transaction-based protocolsQuality of service (
QoS
) (a non-functional requirement)
Problem
: need end-to-end resource management, transactions
Solution
: budgeting of cycles, bandwidth, and storage capacitySlide31
Finally
grid computing assumes and/or requires technologies that include:
Support for executing programs on a variety of platforms
A secure infrastructure
Data movement/replication/federation
Resource discoveryResource managementSlide32
References
Introduction to Grid Computing
IBM Redbook
The Grid 2: Blueprint for a New Computing Infrastructure
,
2nd Edition from Ian Foster, Carl Kesselman.The Anatomy of the Grid: Enabling Scalable Virtual Organizations by I. Foster, C.
Kesselman
and S.
Tuecke , International J. Supercomputer Applications, 15(3), 2001 http://www.globus.org/alliance/publications/papers/anatomy.pdf