on Windows Phone 8 Rob Miles University of Hull Microsoft MVP WPHB301 Common core and security architecture Great consistent experience across devices Productive and connected Robust platform for ID: 408558
Download Presentation The PPT/PDF document "Speechifying Applications" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1Slide2
Speechifying Applicationson Windows Phone 8
Rob MilesUniversity of HullMicrosoft MVP
WPH-B301Slide3
Common core and security architecture
Great, consistent experience across devices
Productive
and connected
Robust platform for
mobile apps
Unified app and
device management
Windows Phone The Right Choice for BusinessSlide4
Topics
Speech on Windows Phone 8Speech synthesisControlling applications using speech
Voice command definition files
Building conversations
Selecting application entry points
Simple speech input
Speech input and grammarsUsing Grammar ListsSlide5
Speech on Windows Phone 8Slide6
Windows Phone Speech Support
Windows Phone 7.5 had voice support built into the operating system Programs and phone features could be activated by voice commands
Incoming SMS messages could be read to the user
The user could compose and send SMS messages
Windows 8 builds on this to allow applications to make full use of speechSlide7
Windows Phone 8 Speech
Applications can speak messages using the Speech Synthesis featureApplications can be started and given commands
Applications can accept commands using voice input
Free speech recognition requires an internet connection, but Speech Synthesis and “programmed recognition” does notSlide8
Speech SynthesisSlide9
Enabling Speech Synthesis
If an application wishes to use speech output the ID_CAP_SPEECH_RECOGNITION capability must be enabled in WMAppManifest.xml
The application can also bring in the Synthesis namespace
using
Windows.Phone.Speech.Synthesis
;Slide10
Simple Speech
The
SpeechSynthsizer
class provides a simple way to produce speech
The
SpeakTextAsync
method speaks the content of the string using the default voice
Note that the method is an asynchronous one, so the calling method must be flagged as asyncSpeech output does not require a network connectionasync void CheeseLiker
(){ SpeechSynthesizer synth = new SpeechSynthesizer(); await synth.SpeakTextAsync("I like cheese.");}Slide11
Speech Exceptions
If the application is interrupted during speech output the speech method will throw an exception if the application is resumed
try
{
SpeechSynthesizer
synth =
new SpeechSynthesizer(); await
synth.SpeakTextAsync("I like cheese.");}catch ( Exception ex) { if (((uint)ex.HResult == 0x80045508)) { // System Call Interrupted thrown by Speech }
}Slide12
Speech Exceptions
Your program must catch this exception or it will fail certification
try
{
SpeechSynthesizer
synth =
new SpeechSynthesizer(); await
synth.SpeakTextAsync("I like cheese.");}catch ( Exception ex) { if (((uint)ex.HResult == 0x80045508)) { // System Call Interrupted thrown by Speech } }Slide13
Selecting a language
The default speaking voice is selected automatically from the locale set for the phone
The
InstalledVoices
class provides a list of all the voices available on the phone
The above code selects a French voice
// Query for a voice that speaks French.
var frenchVoices = from voice in
InstalledVoices.All where voice.Language == "fr-FR" select voice;// Set the voice as identified by the query.synth.SetVoice(frenchVoices.ElementAt(0));Slide14
Speech Synthesis Markup Language
You can use Speech Synthesis
Markup
Language (SSML) to control the
spoken
output
Change the voice, pitch, rate, volume, pronunciation and other characteristicsAlso allows the inclusion of audio files into the spoken outputYou can also
speak the contents of a file <?xml version="1.0" encoding="ISO-8859-1"?><speak version="1.0"
xmlns=http://www.w3.org/2001/10/synthesis xml:lang="en-US"> <p> Your <say-as interpret-as="ordinal">1st</say-as> request was for <say-as
interpret-as="cardinal">1</say-as> room on <say-as interpret-as="date" format="mdy">10/19/2013</say-as>
, arriving at <
say-as
interpret-as="time" format="hms12">12:35pm</say-as>. </p></speak>Slide15
“Speech Markup”
Rob MilesSlide16
SSML Languages
The markup langauge behaviour is sensitive to the locale where the phone is being used
I had some problems when I used en-GB to decode the mark-up file
It works fine with en-US howeverSlide17
Controlling Applications using Voice CommandsSlide18
Voice command Application Launching
The Voice Command feature of Windows Phone 7 allowed users to start applicationsIn Windows Phone 8 the feature has been expanded to allow the user to request data from the application in the start command
The data will allow a particular application page to be selected when the program starts and can also pass request information to that pageSlide19
Adding Voice Commands
To start using Voice Commands you must first request the speech capability for your applicationThe application then calls a method to register “trigger” words and phrases the first time it is runSlide20
A Voice Command Definition (VCD) file
The VCD file can be loaded from the application or from any URIIn the above case it is a file that has been added to the project and marked as Content
The voice commands for an application are loaded into the voice command service when the application runs
The application must run at least once to configure the voice commands
The
voice command behaviour
can also be changed by the application
async void setupVoiceCommands(){
await VoiceCommandService.InstallCommandSetsFromFileAsync( new Uri("ms-appx:///VCDCommands.xml", UriKind.RelativeOrAbsolute));}Slide21
The Cheese Reminder Program
The Cheese Reminder program will remind you to eat some cheese at a slightly later timeThe time is set by using voice commands
Do not use this application if you are lactose intolerantSlide22
<
CommandPrefix
>
Cheese Reminder
</
CommandPrefix
>
<Example> ten minutes </Example>
<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>
<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>
<
PhraseList
Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Voice Command Definition (VCD) fileThis is the VCD content to set the reminder:“Cheese Reminder ten minutes”Slide23
<
CommandPrefix
>
Cheese Reminder
</
CommandPrefix
>
<Example> ten minutes </Example>
<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>
<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>
<
PhraseList
Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Trigger PhraseThis is the phrase the user says to trigger the Cheese Reminder applicationSlide24
<
CommandPrefix
>
Cheese Reminder
</
CommandPrefix
>
<Example> ten minutes </Example>
<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>
<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>
<
PhraseList
Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Example TextThis text is used to show a sample phrase that can be used with the applicationSlide25
<
CommandPrefix
>
Cheese Reminder
</
CommandPrefix
>
<Example> ten minutes </Example>
<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>
<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>
<
PhraseList
Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Command NameThe command name is passed into the application so that it knows what the user has saidSlide26
<
CommandPrefix
>
Cheese Reminder
</
CommandPrefix
>
<Example> ten minutes </Example>
<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>
<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>
<
PhraseList
Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Command ExampleAn example for this particular commandSlide27
<
CommandPrefix
>
Cheese Reminder
</
CommandPrefix
>
<Example> ten minutes </Example>
<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>
<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>
<
PhraseList
Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Command to Listen ForSound to listen forElements in square brackets are optionalSlide28
<
CommandPrefix
>
Cheese Reminder
</
CommandPrefix
>
<Example> ten minutes </Example>
<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>
<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>
<
PhraseList
Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Phrase ListThis is a list of phrases that can follow the commandThe program can determine which phrase was spoken by the userSlide29
<
CommandPrefix
>
Cheese Reminder
</
CommandPrefix
>
<Example> ten minutes </Example>
<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>
<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>
<
PhraseList
Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>FeedbackThis gives the format of spoken feedback for the commandSlide30
<
CommandPrefix
>
Cheese Reminder
</
CommandPrefix
>
<Example> ten minutes </Example>
<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>
<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>
<
PhraseList
Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Navigation URLThis identifies the page in the application which will be loaded when the command is recognisedSlide31
<
CommandPrefix
>
Cheese Reminder
</
CommandPrefix
>
<Example> ten minutes </Example>
<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>
<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>
<
PhraseList
Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Phraselist contentsThese are the phrases that can follow the commandThe program can determine which phrase was givenSlide32
Picking up Voice Commands
This code runs in the OnNavigatedTo
method of a target page
Can also check for the voice command phrase that was used
if
(
e.NavigationMode
== System.Windows.Navigation.NavigationMode.New) { if (
NavigationContext.QueryString.ContainsKey("voiceCommandName")) { switch (NavigationContext.QueryString["voiceCommandName"]) { case "SetReminder": string
delayNumberString = NavigationContext.QueryString["number"]; break; } }}Slide33
Cheese Reminder
Rob MilesSlide34
Identifying phrases
The navigation context can be queried to determine the phrase used to trigger the navigationIn this case the program is selecting the units used in setting the reminder
<
PhraseList
Label
=
"units"> <Item
> minute </Item> <Item> minutes </Item> <Item> hour </Item> <Item> hours </
Item></PhraseList>string delayUnits = NavigationContext.QueryString["units"];Slide35
Modifying the phrase list
An application can modify a phrase list when it is runningIt cannot add new commands howeverThe program must be able to make use of these modified names
VoiceCommandSet
fortuneVcs
=
VoiceCommandService
.InstalledCommandSets["en-US"];await
fortuneVcs.UpdatePhraseListAsync("units", new string[] { "hours", "minutes", "seconds", "millenia" });Slide36
Finding the phrase list
The command set name in the InstalledCommandSets
maps onto the
Name
field in the voice command xml file
VoiceCommandSet
fortuneVcs = VoiceCommandService.InstalledCommandSets["en-gb
"];<CommandSet xml:lang="en-gb" Name="en-gb">Slide37
Handling Languages
You can create customised command files for each languageWhen your application runs it can detect which language is in use on the phone and find a matching command fileSlide38
Language identification
I use a list of supported languages and then check the CultureInfo
setting on the phone
string
[]
supportLangauges
=
new string[] { "en-us", "en-
gb" };string language = CultureInfo.CurrentCulture.Name.ToLower(); foreach (string supportedLanguage in supportLangauges) if (language == supportedLanguage) return
true;// got the languageSlide39
First Time Run
The first time your program runs the customised phrase list may not be ready for useYou have to handle this situation gracefullyDisplay a message and ask the user to run the program againSlide40
Installation checking
If you assume the command set is present your program will throw an exception the first time it runs
if
(!
VoiceCommandService
.InstalledCommandSets.ContainsKey
(language))
{ MessageBox.Show("Voice Commands Loading" +
"Please restart the program to set up the voice commands." ); return;}Slide41
Voice Music
Voice Music builds a phrase list from the music on your phoneYou can then select music and playlists using voice commandsYou can get it from the Windows Phone Marketplace
It’s free
Slide42
Creating the album name phrase list
This method adds all the album names to the phrase recognition
async
void
addAlbumCommands
(VoiceCommandSet widgetVcs){
using (MediaLibrary library = new MediaLibrary()) { string[] albumNames = new string[library.Albums.Count]; for (int i = 0; i < library.Albums.Count
; i++) albumNames[i] = library.Albums[i].Name; await widgetVcs.UpdatePhraseListAsync("albumName", albumNames); }}Slide43
Voice Music
Rob MilesSlide44
Free Speech InputSlide45
Recognizing Free Speech
A Windows Phone application can recognise words and phrases and pass them to your programFrom my experiments it seems quite reliable
Note that a network connection is required for this feature
Your application can just use the speech string directly
The standard “Listening” interface is displayed over your applicationSlide46
Simple Speech Recognition
The above method checks for a successful responseBy default the system uses the language settings on the Phone
SpeechRecognizerUI
recoWithUI
;
async
private void ListenButton_Click
(object sender, RoutedEventArgs e){ this.recoWithUI = new SpeechRecognizerUI(); SpeechRecognitionUIResult recoResult = await recoWithUI.RecognizeWithUIAsync(); if
( recoResult.ResultStatus == SpeechRecognitionUIStatus.Succeeded ) MessageBox.Show(string.Format("You said {0}.", recoResult.RecognitionResult.Text));
}Slide47
Customizing Speech Recognition
InitialSilenceTimeout The time
that
the speech recognizer will wait
until it hears speech.
The
default setting is 5 seconds.BabbleTimeoutThe time
that the speech recognizer will listen while it hears background noiseThe default setting is 0 seconds (the feature is not activated). EndSilenceTimeout The time interval during which the speech recognizer will wait before finalizing the recognition operationThe default setting is 150 milliseconds. Slide48
Customizing Speech Recognition
A program can also select whether or not the speech recognition echoes back the user input and displays it in a message boxThe code above also sets timeout values
recoWithUI.Settings.ReadoutEnabled
=
false
;
// don't read the saying back
recoWithUI.Settings.ShowConfirmation = false; // don't show the confirmation
recoWithUI.Recognizer.Settings.InitialSilenceTimeout = TimeSpan.FromSeconds(6.0);recoWithUI.Recognizer.Settings.BabbleTimeout = TimeSpan.FromSeconds(4.0);recoWithUI.Recognizer.Settings.EndSilenceTimeout = TimeSpan.FromSeconds(1.2);Slide49
Handling Errors
An application can bind to events which indicate problems with the audio inputThere is also an event fired when the state of the capture changes
recoWithUI.Recognizer.AudioProblemOccurred
+=
Recognizer_AudioProblemOccurred
;
recoWithUI.Recognizer.AudioCaptureStateChanged
+= Recognizer_AudioCaptureStateChanged;...
void Recognizer_AudioProblemOccurred(SpeechRecognizer sender, SpeechAudioProblemOccurredEventArgs args){ MessageBox.Show("Please speak more clearly");}Slide50
Important Note
Just a reminder that if you use Free Speech input the phone will require a network connection to decode the text
This is not the case with commands you build into the application
They can be used at any time, even if you have customised themSlide51
Using GrammarsSlide52
Grammars and Speech input
The simple speech recognition we have seen so far uses the “Short Dictation” grammar which just captures the text and returns it to the applicationYou can add your own grammars that will structure the conversation between the user and the applicationSlide53
Grammars and Speech input
Grammars can be created using the Speech Recognition Grammar Specification (SRGS) Version 1.0 and stored as XML files loaded when the application runs
This is a little complex, but worth the effort if you want to create applications with rich language interaction with the user – there are examples on MSDN
If the application just needs to identify particular commands you can use a grammar list to achieve thisSlide54
Using Grammar Lists
To create a Grammar List an application defines an array of strings that form the words in the listThe Grammar can then be added to the recognizer and given a name
Multiple grammar lists can be added to a grammar recognizer
The recognizer will now resolve any of the words in the lists that have been supplied
string
[]
strengthNames
= { "weak", "mild", "medium",
"strong", "english"};recoWithUI.Recognizer.Grammars.AddGrammarFromList("cheeseStrength", strengthNames);Slide55
Enabling and Disabling Grammar Lists
An application can enable or disable particular grammars before a recognition actionIt is also possible to set relative weightings of grammar lists
The text displayed as part of the listen operation can also be set, as shown above
recoWithUI.Settings.ListenText
=
"How strong do you like your cheese?"
;
recoWithUI.Recognizer.Grammars["cheeseStrength"
].Enabled = true;SpeechRecognitionUIResult recoResult = await recoWithUI.RecognizeWithUIAsync();Slide56
Result confidence
An application can determine the confidence that the speech system has in the result that was obtainedResult values are High, Medium, Low, Rejected
SpeechRecognitionUIResult
recoResult
=
await
recoWithUI.RecognizeWithUIAsync();if
( recoResult.RecognitionResult.TextConfidence == SpeechRecognitionConfidence.High ){ // select cheese based on strength value}Slide57
Matching Multiple Grammars
If the spoken input matches multiple grammars a program can obtain a list of the alternative results using
recoResult.RecognitionResult.GetAlternatives
The list is supplied in order of confidence
The application can then determine the best fit from the context of the voice request
This list is also provided if the request used a more complex grammar
var
alternatives = recoResult.RecognitionResult.GetAlternates(3);Slide58
Profanity
Words that are recognised as profanities are not displayed in the response from a recognizer commandThe speech system will also not repeat themThey are enclosed in
<Profanity> </Profanity>
when supplied to the program that receives the speech dataSlide59
Review
Applications in Windows 8 can use speech generation and recognition to interact with usersApplications can produce speech output from text files which can be marked up with Speech Synthesis Markup
Language (SSML)
to include text
Applications can be started and provided with initial commands by registering a Voice Command Definition File with the Windows Phone
The commands can be picked up when a page is loaded, or the commands specify a particular page to load
An application can modify the phrase part of a command to change the activation commands
Applications can recognise speech using complex grammars or simple word listsSlide60
Related content
Breakout Sessions (session codes and titles)
Hands-on Labs (session codes and titles)
Product Demo Stations (demo station title and location)
Related Certification Exam
Find Me Later At...Slide61
Related content
WPH-B301: Speechifying
your Windows Phone 8
Applications – it’s been and gone, but the recording should be worth watching
Find me on the
TechED
booth on Thursday morning
All the demo content and a really funny blog:
www.robmiles.comSlide62
Windows Phone Breakout Sessions
Monday, June 3:
1:15pm - The phone that has everything the enterprise needs: Windows Phone 8
3:00pm – The top down guide for developers: Windows Phone 8
4:45pm – Radical perspectives on mobility strategy
Tuesday, June 4:
8:30am – The power of collaboration: Integrating Windows Phone with Office 365, Exchange and SharePoint Online
10:15am – All aboard for the future of HTML5 mobile & hybrid web apps for Windows Phone 8 and Windows Tablets
1:30pm – The Windows Phone 8 networking survival kit3:15pm – Build it once for both: Writing code and designing for Windows 8 and Windows Phone 85:00pm – Secrets of using background agents for Windows Phone 8Slide63
Windows Phone Breakout Sessions
Wednesday, June 5:
8:30am – Mobile security in the enterprise: Windows Phone 8 answers the call
10:15am – Using C and C++ in your Windows Phone 8 applications
1:30pm – Developing large-scale enterprise mobile apps for Windows Phone 8
3:15pm – Speechifying your Windows Phone 8 applications
Thursday, June 6:
8:30am – Manage Windows Phone enterprise apps
10:15am – Support your demanding LOB apps with SQLite and Windows Phone 81:00pm – Creating Windows Phone 8 apps for SharePointSlide64
Windows Phone Hands On Labs
Porting Windows 8 to Windows Phone 8
Windows Phone 8: File and Protocol Association
Windows Phone 8: Lock Screen Wallpaper
Windows Phone 8: Voice Command
Windows Phone 8: Tiles
Windows Phone 8: Purchase
Windows Phone 8: Wallet
Windows Phone 8: Running TrackerSlide65
Windows Phone Booth
Device Bar featuring the latest Windows Phones in the marketplace
Expert Area – Stop by the booth to get all your Windows Phone questions answered by Windows Phone experts
Theater Presentations:
The Windows connected experience
Get Hands on with Windows Phone for Business
Windows Phone 8 Developer focused sessionsSlide66
Windows Phone Promotions
Windows Phone Hats – Stop by the Windows Phone booth to get your free hat. Wear the hat for a chance to win a Windows Phone.
Attend a Windows Phone theater session to be entered into a drawing
for Windows Phones and other prizes.
Windows Phone Breakout Sessions – Attend Windows Phone breakout sessions and be entered into a drawing to win a Windows Phone.
Windows Phone Hoodie’s – Get a demo from the Windows Phone roaming demo team in the expo hall. Wear the hoodie for a chance
to win a Windows Phone.Slide67
Track Resources
For more information about Windows Phone:
http://www.windowsphone.com/businessSlide68
msdn
Resources for Developers
http://microsoft.com/msdn
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
TechNet
Resources
Sessions on Demand
http://channel9.msdn.com/Events/TechEd
Resources for IT Professionals
http://microsoft.com/technet Slide69
Evaluate this session
Scan
this QR code
to
evaluate this session.
Required Slide
*delete this box when your slide is finalized
Your MS Tag will be inserted here during the final scrub. Slide70
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.