COMP3211 Advanced Databases Dr Nicholas Gibbins nmgecssotonacuk 20142015 Transactions Revisited 2 Complete units of work ACID Transactions may be distributed across more than one processor ID: 582466
Download Presentation The PPT/PDF document "Message Queues" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Message Queues
COMP3211 Advanced Databases
Dr
Nicholas Gibbins –
nmg@ecs.soton.ac.uk
2014-2015Slide2
Transactions Revisited
2
Complete units of work – ACID
Transactions may be distributed across more than one processor
Dedicated link between programs
Operations synchronized
Distributed two phase commit
Many transactions
need not be
completed synchronously
Communication must still be guaranteed
Work can be completed later Slide3
Synchronous versus Asynchronous
3
HTTP is a
synchronous
protocol
Request from client to server is followed by reply from server to client in the same TCP connection
SMTP
is an
asynchronous
protocol
Email messages are sent on a store-and-forward basis
Final message recipient need not be available when message is
sentSlide4
Messaging and Queuing (MQ)
4
MQ enables programs to communicate across a network asynchronously
No private, dedicated link required
Systems can be heterogeneous
Message delivery is guaranteed
Messages are placed on queues by one system and taken off by another
Similar idea to email, but more sophisticated mechanism
MQ can also be used between programs running on the same systemSlide5
Direct Transaction Processing
5
Conventional (direct) transaction processing has weaknesses:
Designed for synchronous processing
Has difficulties with long-lived transactions and communication errors
Difficult to balance loads between several servers carrying out the same tasks
Difficult to
prioritise
one request over anotherSlide6
6
If server is down, client does not receive an immediate answer
Cannot distinguish between:
request not delivered to server
server failure
reply not delivered to client
Direct TP Problems: Server Failure
Client
ServerSlide7
7
Cannot tell if response has been received by client
Direct TP Problems: Client Failure
Client
ServerSlide8
8
Cannot ensure that servers carrying out a task have an even load
Direct TP Problems: Unbalanced Load
Server 1
Server 2
Server 3
requestsSlide9
9
Tasks handled on a first-come, first-served basis; cannot process high-priority requests early
Direct TP Problems: No
Prioritisation
Server
!
requests
urgent requestSlide10
10
Client adds request to queue (
enqueues
)
Server removes request from queue (
dequeues
)
If server is unavailable (busy, down, disconnected), the request is stored in the queue until the server is able to process it
Message Queues
Client
Server
message queue
enqueue
request
dequeue
requestSlide11
11
Bidirectional communication between client and server requires a separate queue for communications in each direction
Bidirectional Queues
Client
Server
request queue
enqueue
request
dequeue
request
reply queue
enqueue
reply
dequeue
replySlide12
12
Communication may be one-to-many
Communication may be many-to-one
Application Structure
Client
Server
message queue
Server
Server
Client
message queue
Server
Client
ClientSlide13
13
When a server finishes processing a request, it takes the next from the queue
Load Balancing
Client
Server
message queue
enqueue
dequeue
Client
ServerSlide14
Queued Transaction Processing
14
transaction
start
dequeue
(request queue)
process
request
enqueue
(reply
queue)
commit
request
queue
reply
queueSlide15
Queued Transaction Processing
15
If transaction aborts:
request returned to input queue
changes made by transaction are rolled back
if necessary, reply removed from output queue
Repeated aborts (due to a poisoned message) may be prevented with a maximum limit on retriesSlide16
Queued Transaction Processing
16
transaction 1: submit request
start
construct
request
enqueue
(request
queue)
commit
transaction 3: process reply
start
dequeue
(reply queue)
process
output
commit
transaction 2: execute request
start
dequeue
(request queue)
process
request
enqueue
(reply
queue)
commit
client
server
request
queue
reply
queueSlide17
Message Ordering
17
Description of message queues so far does not consider how messages are ordered in a queue
First-come, first-served
Highest-priority-first
Aborted transactions may lead to out-of-order processing:
T1
dequeues
M1
T2
dequeues
M2
T2 commits
T1 aborts, returns M1 to queueSlide18
Benefits
Simple APIs hide communications complexityFewer constraints on inter-program operation
Programs
communicate indirectly –
asynchronous
Client
and server do not need to be running at the same
time
Fewer network sessions needed, and programs are less vulnerable to network failures
Business change easier to handle
Assured message delivery
Asynchronous
t
ransaction processing
18Slide19
Implementations and Standards
Message Queuing is not a new idea!Used for, example, within IBM's Information Management System (IMS) 30 years ago
Both proprietary and open source APIs and platforms
IBM
Websphere
MQ, Microsoft Message Queuing, Oracle Advanced Queuing
Apache
ActiveMQ
, Rabbit MQ
Some products inter-operate via common standards
AMQP,
MQTT
19Slide20
Example 1: Client-Server Applications
20
Insurance agents throughout the country
request insurance
quotations using an
online system
.
This system is
implemented as
a traditional client-server application, with client programs (the insurance agents) sending requests for quotations to a central server program.
The server does some calculations using data from a central insurance database, then sends a quotation to the requesting agent. Slide21
21
Client
programs put request messages on a single queue, from which the server program takes them
.
Responses may be sent back to clients via extra message queues (one per client)
Example 1: Client-Server Applications
Insurance
Agent
message queue
Insurance
Quotation
Insurance
Agent
Insurance
Agent
insurance
dataSlide22
Example 2: Output-only Devices
22
A device providing a service is output-only, and does not send messages back to the requester. Only one-way message flows are required.
Output-only devices:
Printing devices
Displays:- Stock exchange, Flight arrivals and departures
Factory floor roboticsSlide23
23
Robotics Controller is
in control of an automated manufacturing process. It puts messages
on:
Queue 1 for the Robotics
A
program, which directs some
welding
machinery
Queue
2 for the Robotics
B
program, which controls a paint sprayer.
Example 2: Output-only Devices
Robotics
Controller
Robotics
A
Robotics
B
queue 1
queue 2Slide24
Example 3: The Batch Window
24
A department store writes its sales figures to a file throughout the
day
’
s
trading. Overnight, a report of the
day’s
sales is produced using this file of data as input. The report must be on the Sales
Manager’s
desk before the next
day
’
s
trading
begins.
T
he
amount of time available for producing the report is limited to the “window” of time between the end of business on one day and the start of business on the nextThe start and finish times for the activity are fixedSlide25
25
Instead of operating in sequence and communicating via a file, the two programs could run independently of each other and communicate using
a message queue
Example 3: The Batch Window
Sales
Recording
Sales
Reporting
message queueSlide26
Summary
Message queuing allows an alternative way to distribute applications
Many applications involve databases, so the technologies intersect
Conceptually and practically, MQ is often a more straightforward approach than direct connections between systems, and synchronous transaction processing
26