Brian F Cooper Adam Silberstein Erwin Tam Raghu Ramakrishnan Russell Sears Yahoo Research Presenter Duncan Benchmarking Cloud Serving Systems with YCSB Benchmarking vs Testing ID: 172879
Download Presentation The PPT/PDF document "Benchmarking Cloud Serving Systems with ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Benchmarking Cloud Serving Systems with YCSB
Brian F. Cooper, Adam Silberstein, Erwin Tam,
Raghu
Ramakrishnan
, Russell Sears
Yahoo!
Research
Presenter DuncanSlide2
Benchmarking Cloud Serving Systems with YCSB
Benchmarking
vs
Testing
Any difference?
My opinion
Benchmarking: Performance
Testing: usability test, security test, performance etc…Slide3
Motivation
A lot of new systems in Cloud for data storage and management
MongoDB
,
MySQL
,
Asterix
, etc..
Tradeoff
E.g. Append update to a sequential disk-log
Good for write, bad for read
Synchronous replication
copies up to date, but high write latency
How
to choose?
Use benchmark to model your scenario!Slide4
Evaluate Performance =?
Latency
Users don’t want to wait!
Throughput
Want to serve more requests!
Inherent tradeoff between latency and throughput
More requests => more resource contention=> higher latencySlide5
Which system is better?
“
Typically application designers must decide on an acceptable latency, and provision enough servers to achieve the desired throughput
”
achieve the desired latency and throughput with fewer servers.
Desired latency:0.1 sec, 100 request/sec
MongoDB
, 10 server
Asterix
DB, 15 serverSlide6
What else to evaluate?
Cloud platform
Scalability
Good scalability=>performance proportional to # of servers
Elasticity
Good elasticity=>performance improvement with small disruptionSlide7
A Short Summary
Evaluate performance = evaluate latency, throughput, scalability, elasticity
A better system= less machine to achieve the performance goalSlide8
YCSB
Data generator
Workload generator
YCSB client
Interface to communicate with DBSlide9
YCSB Data Generator
A
table with
F
fields and
N
records
Each field => a random string
E.g. 1,000 byte records,
F
=10, 100 bytes per fieldSlide10
Workload Generator
Basic operations
Insert, update, read, scan
No join, aggregate etc.
Able to control the
distributions
of:
Which operation to perform
E.g. 0.95 read, 0.05 update, 0 scan => read-heavy workload
Which record to read or write
Uniform
Zipfian
: some records are extremely popular
Latest: recent records are more popularSlide11
YCSB Client
A script
Use the script to run the benchmark
Workload parameter files
You can change the parameter
Java program
DB interface layer
You can implement the interface for your DB systemSlide12
Experiments
Experiment Setup:
6 servers
YCSB client on another server
Cassandra,
HBase
,
MySQL
, PNUTS
Update heavy, read heavy, read only, read latest, short range scan workload.Slide13
Future Work
Availability
Impact of failure on the system performance
Replication
Impact to performance when increase replicationSlide14
4 criteria
Author’s 4 criteria for a good benchmark:
Relevance to application
Portability
Not just for 1 system!
Scalability
Not just for small system, small data!
simplicitySlide15
Reference
Benchmarking Cloud Serving Systems with
YCSB,
Brian
F. Cooper, Adam Silberstein, Erwin Tam,
Raghu
Ramakrishnan
, Russell
Sears,
SOCC
10
BG: A Benchmark to Evaluate Interactive
Social Networking Actions,
Sumita
Barahmand
,
Shahram
Ghandeharizadeh
, CIDR 13
http://
en.wikipedia.org/wiki/Software_testing
http://en.wikipedia.org/wiki/Benchmark_(computing)Slide16
Thank You
!
Questions?Slide17
Why a new benchmark?
Most cloud systems do not have a SQL interface =>
hard to implement complex queries
Benchmark only for specific applications
TPC-W for E-commerce
TPC-C for apps that mange, sell, distribute product/service