Service with Azure Storage Copyright 2016 Monza Clound LLC Agenda Who we are Monza Cloud LLC Note Were not here to read MSDN articles code snippets to you Were here to demonstrate production code techniques and share lessons ID: 748320
Download Presentation The PPT/PDF document "Azure, Caas and Storage Practical Data D..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Azure, Caas and Storage
Practical Data Driven Configuration as a Service with Azure Storage
Copyright © 2016 Monza Clound, LLCSlide2
Agenda
Who we are, Monza Cloud, LLC.
Note: We’re not here to read MSDN articles / code snippets to you. We’re here to demonstrate production code techniques and share lessons
learned
How we use CaaS (Configuration as a Service) with database driven configuration repositories
Quick introduction to AzStudio, our cloud APIs and Tools
Demonstrate the basic steps to work with storage using MSDE code
Practical Demonstrations :
CaaS at work – Shortcuts and environments explained
Blob storage creation, routing, and testing
Queue creation, routing and testing. Demo BigItem
Table Storage creation, routing and testing. Demo BigItem
Closing statement
Q&A, reference to resourcesSlide3
Monza Cloud, LLC
Founded in 2015Team worked for several years on various Azure projects before forming company
We specialize in rapid development of SaaS offerings that take advantage of our Azure framework
We’re not a big “move this VM” or “Modify this existing app” team, we’re here to write your apps in the cloud for the cloud
Extensive experience deploying “internet of things” solutionsSlide4
What is CaaS
It is a web service that examines the client’s DNS information and optional attributes to determine it’s config informationIt delivers either a JSON master configuration file or a database connection string to the configuration database
CaaS can be considered the “bootstrap” for our environments
Configuration manager then brokers the config information to the rest of the codeSlide5
CaaS Advantages
No more config files. No need for specialized tools to manage these filesDatabase driven config means referential integrity checks and data enforcement, easy to back up and restore
Configuration data can be segmented and management of specific portions of configuration can be distributed to different personnel
Configuration changes can be picked up on-the-fly (runtime) by your services without recycling or redeploying the service
Can be made exceptionally redundant, can gracefully handle multiple layers of failure without redeploymentSlide6
CaaS - Bootstrap
Service Starts
List of
Master endpoints
Pop next endpoint
Call endpoint, asking for configuration
data
Configuration service uses reverse DNS / keys to lookup config
Missing lookup or service request fails
Return ENVIRONMENT settings and DATABASE connection
Service Runs
Service sets environment values and config database info (singleton)
No more endpoints!
Service Fails
Note: JSON file can replace database connectionSlide7
Service – Post Bootstrap
CaaS response has now set the global property to an environment id. This drives ALL configuration lookup operationsApplication now consumes services / functionality as necessary
and a connection manager dictates routing
Programmers ignore all of this, they use
shortcuts
Control and management are abstracted across multiple projects, teams and responsibilities are segmented, making cloud configuration a breezeSlide8
AzStudio : Quick Overview
For today’s demos we’re using our the Monza Cloud product suite for Azure, AzStudioAzStudio C# API’s == JQuery or Angular in the JavaScript world
AzStudio == Visual toolset used by programmers and DevOps
If you’re developing in Azure you WILL develop something like this in your environment. Either using a collection of tools / scripts or something consolidated like our product
You will use these patterns and codify them in your organization in order to survive the rapid decomposition of resources in a SaaS /
Paas
EnvironmentSlide9
Common Pattern in Azure Storage
Create your storage accountUse connection string to connect to account, preserving this connection reference in memory (singleton pattern)
Get a reference to the storage type (blob, queue, table storage). Keep that reference in memory as well
Get a reference to the specific resource (container, queue client , table client). Yes, preserve the reference for future operations
Perform actions. Using singleton patterns will speed up subsequent steps dramatically. Using “infrequent refresh” techniques will allow you to update the above information without recycling rolesSlide10
BigItem
Before we go to practical demos, let’s talk about overcoming limitations in Queues and Table StorageBoth systems have a maximum item size (64K and 1MB)
The work around to this is to synchronize additional information in a blob storage container
The trick to this work around is abstracting the mechanics of this extension out so developers are not constantly having to be aware of this arrangement
Our solution is called “BigItem” and we’ll demonstrate it during our practical demosSlide11
Topic #1: Blobs
Blobs are any binary (file) data you want to store in the cloudBlobs are stored in a
container
in a
storage account
Containers can be set public (with or without browsing) or private.
Most errors occur with containers that are private and you want public access
Blob performance depends on many things, size and how you access will effect performance the most
SAS (Shared Access Service) allows you to grant limited-time access to a file on a private container
CORS is now available at the storage account level, allows you to publish web services / scripts to storage containersSlide12
Practical Demonstration #1: Blobs
We’re going to set up a storage container type (Crowd picks name)Set up routing table elements for DEV and QA (but not production)
We’ll create a blob “shortcut” to consolidate routing
Test 1 – Read, write and delete contents in DEV
Test 2 – Read, write and delete contents in QA
Test 3 – Demonstrate error raised when attempting to hit productionSlide13
Blob Code Examples
Here’s how our code looks when using Blob ServicesYou’ll want to create a basic set of library calls for your product(s)
Makes inheritance of things like
StorageAccount
connections reusable and cacheable
Upload
:
Az.BlobStorageManager.UploadFileToStorage
(
path,shortcutName
);
Download:
Az.BlobStorageManager.
DownloadFileFromStorage(
sourcePath, destPath
, shortcutName);Delete: Az.BlobStorageManager.DeleteFile
(fileName,shortcutName);List:
Az.BlobStorageManager.GetContainerFiles(shortcutName);Slide14
Blobs – MSDN Links and AzStudio Example
How to set up a storage account: https
://azure.microsoft.com/en-us/documentation/articles/storage-create-storage-account
/
Blobs
and Containers:
https://
msdn.microsoft.com/library/azure/dd179376.aspx
These links are raw MSDN documentation.
AzStudio’s
APIs wrap these in much easier code. For example, to write a file to a blob is as simple as:
Az.BlobStorageManager.UploadFileToStorage
(
pathToBlob
,”
myshortcutname”);Slide15
Topic 2: Queues
Queues are used to pipe information between systemsQueues reside in a
container
in a
storage account
“Queue Driven Workflow” is a basic tenant of cloud architecture, so you’ll need to use queues quite a bit
Queue throughput: (very roughly) around 5000 1K transactions a minute
Size of transaction has a high impact on queue throughput
Have plans in place to federate your queues and a robust library around this to reduce complexity in
your systemsSlide16
Practical Demonstration #2: Queues
Set up a queue container type and routing informationCreate queue shortcut for DEV and QA, but not production
Demonstrate writing the reading with the queue.
Explain our wrapper and encryption in queues “at rest”
Demonstrate encryption of queue item contents
Talk about federation and BigItem. Demonstrate BigItemSlide17
Queues – MSDN Links and AzStudio Example
Setting up Queue and using it in MSDN:
https
://
msdn.microsoft.com/library/azure/dd179353.aspx
Using queues in AzStudio uses the same concepts as blobs. Just use a shortcut (encryption and BigItem are transparent). Here’s an example.
Az.QueueManager.GetQueue
(“
myqueueshortcut
”).
Enqueue
(
myObject
);Slide18
Topic 3: Table Storage
Table storage is high-volume schema-less entity storageTables store any class that implements
ITableEntity
, which has several restrictions
Two properties make up the primary key for a row
PartitionKey
and
RowKey
You can only really effectively query against these two keys. While you can query against any of you’re classes properties, not using these two properties will invoke a full table scan
Careful planning around keys is the core principle of using table storage
See
https://azure.microsoft.com/en-us/documentation/articles/storage-table-design-guide/#
table-design-patterns
for detailed pattern / anti-pattern informationSlide19
Practical Demonstration #3: Table Storage
Create the table as a container type
Map the table in DEV and QA
Write and read from DEV
Write and Read from QA
Show that PROD is not responding (errors due to missing table routing)
Explain how LINQ and FILTERS work
Explain BigItem with table storageSlide20
Conclusion
You’re cloud development and deployment can get overly complicated with configuration filesEven with specialized scripts and tools the files just don’t lend themselves to clean segmentation and security practices
The tools and techniques demonstrated tonight should give you ideas about your own efforts
Often overlooked is having these tools enforces practices, which help you overcome business culture issues
Now we’ll open up to Q&A