Romit Girdhar Software Engineer P4016 Wee Hyong Tok Program Manager CLOUD MOBILE INTERNET CONNECTED DIGITAL ANALOG VALUE Complex implementations Spreadmarts Siloed data Transactional systems ID: 647799
Download Presentation The PPT/PDF document "Demystifying Cloud Data Services for an ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1Slide2
Demystifying Cloud Data Services for an App Developer
Romit GirdharSoftware Engineer
P4016
Wee Hyong Tok
Program ManagerSlide3
CLOUD
MOBILE
INTERNET CONNECTED
DIGITAL
ANALOG
VALUE
Complex implementations
Spreadmarts
Siloed data
Transactional systems
Enterprise data warehouse
OLAP
ETL
Hadoop
Dashboards
Ad hoc analysis
Operational reporting
Machine learning and AI
Any data
In-memory
Internet of Things
Growth of data
Evolving
Data Processing Requirements
Evolving RequirementsSlide4
Transforming Data into Intelligent Action
#
MSBuild
OLTP, ERP, LOB, ...
Devices, social,
sensors, web
BI tools
Data marts
Apps
Dashboards
TRANSFORM
ETL tool
(SSIS, etc.)
EXTRACT
Original data
Transformed data
INGEST
Original data
LOAD
(SQL Sever, Teradata, etc.)
EDW
Scale-out, storage,
and compute
(HDFS, Blob storage, etc.)
TRANSFORM AND LOAD
(On-premise and in the cloud)Slide5
Common Components of Data-driven Solutions
Ingestion
– Incoming dataProcessing – Analyze/Act on your data
Storing
– Access your raw/processed dataServing – Serve your dataSlide6
Ingestion Patterns
Real-time Ingestion
- Capable of ingesting millions of messages/sec
Near-real time Ingestion
– Data received from web forms, etc., ingesting less than 1000msg/secOne-time Load
Scheduled Periodic LoadsSlide7
Processing Patterns
Real-time Processing
– Need to process each event as it arrives.Batch Processing
– Process large amounts of data on a periodic scheduleSlide8
Data Serving Patterns
Dashboarding/Reporting
Ad-hoc Data Access – Access your data in your app
Interactive Data Access
– Slice/Dice, Drill-downSlide9
Data Storage Patterns
OLTP/Hot Store
– Great for quick access of data & light-weight Interactive queries
Warm Store
– Cheap to store ; Great for batch processingArchival/Cold Store – Cheap to store ; Data only needs to be accessed as an exception.
OLAP/Analytical Store – Similar to a hot store, but, provides fast and interactive access to your dataSlide10
Types of Data Stores
Transient / Staging
Big Data RelationalNon-RelationalJSON DocumentsColumn
Key-Value
GraphTime-seriesData Volume and VelocityReal time vs BatchRead/Write LatencyTypes of data to be stored
RequirementsSlide11
Solution PatternsSlide12
Solution Pattern #1 - Real-time Serving
Inbound
Data
Hot/
OLTP Store
Application
Application
Cold/ArchivalSlide13
Store
Ingest
Serve and
Consume
Solution Pattern #1 - Real-time Serving
Inbound
Data
Hot/
OLTP Store
Application
Application
Cold/
ArchivalSlide14
Solution Pattern #2 – Real-time Processing
Inbound
Data
Message Queue
Near real-time processing engine
Hot/OLTP Store
Application
Dashboarding/Reporting
Warm Store
(for later processing)Slide15
Store
Process
Serve and
Consume
Ingest
Solution Pattern #2 – Real-time Processing
Inbound
Data
Message Queue
Near real-time processing engine
Hot/OLTP Store
Application
Dashboarding/Reporting
Warm Store
(for later processing)Slide16
Solution Pattern #3 - Batch Processing
Data in the
warm store
Data Processing
Application
Reports and
Dashboard
OLAP/Analytical Store
Advanced
Analytics
Inbound
Data
Other Data SourcesSlide17
Solution Pattern #3
Other Data Sources
Data in the
warm store
Data Processing
Application
Reports and
Dashboard
OLAP/Analytical Store
Advanced
Analytics
Inbound
Data
Store
Process
Ingest
Serve and ConsumeSlide18
What are my Technology Options?Slide19
What are my tech options?Slide20
Let’s test our knowledge…
Support Call Logs
Service Telemetry Data
Social Data
?
?
Ingest
Process
Store
Serve and
ConsumeSlide21
Ingest
Process
Store
Serve and
Consume
Let’s test our knowledge…
Azure SQL
Data Warehouse
PolyBase
over
Azure Data Lake Store
Azure Analysis Services
Federated
U-SQL Join Query
(Reference Data)
Intelligence @ Scale
Built-in Cognition Capabilities
Azure Data Lake Store and Analytics
Azure SQL Database
Power BI
Event Hubs
Stream Analytics
Real-time Insights and Alerts
Support Call Logs
Service Telemetry Data
Social DataSlide22
How should I get started?Slide23
Data Processing Flow – Real-time processingSlide24
Data Processing Flow – Batch Processing
No
No
Yes
NoSlide25
Reference architecture for common scenarios
Built on best practice design patterns
Automated deployment on your Azure subscription
Customizable for your needs
Supported by a global partner ecosystem
Get started in minutes
http://aka.ms/cisolutions
Call to action
Code Labs – Building your First Apphttps://github.com/Microsoft-Build-2016/CodeLabs-DataSlide26
Related sessions
Learn more about building Data-driven Apps for Scale:
B8040: How JCI built next-gen Data-driven applications at scaleThursday, May 11th, 2017: 4.00pm – 5.00pm PST
B8081
: How to serve AI with Data: The future of the data platform Wednesday, May 10th, 2017: 2.00pm – 3.00pm PST [Live streaming available!]B8018: How to build global scale IoT applications with Azure SQL DB
Thursday, May 11th, 2017: 3.30pm – 4.30pm PSTCode Stories Theater sessions#MSBuildSlide27