Speechifying Applications - PowerPoint Presentation

423 views
Uploaded On 2016-07-17

Speechifying Applications - PPT Presentation

on Windows Phone 8 Rob Miles University of Hull Microsoft MVP WPHB301 Common core and security architecture Great consistent experience across devices Productive and connected Robust platform for ID: 408558

windows phone item speech phone windows speech item command voice application reminder phraselist minutes feedback ten cheese number commandprefix

Link:

Copy

Embed:

<iframe width="560" height="315" src="https://www.docslides.com/embed/408558" frameborder="0" allowfullscreen></iframe>

Download Presentation from below link

Download Presentation The PPT/PDF document "Speechifying Applications" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Presentation Transcript

Slide1
Slide2

Speechifying Applicationson Windows Phone 8

Rob MilesUniversity of HullMicrosoft MVP

WPH-B301Slide3

Common core and security architecture

Great, consistent experience across devices

Productive

and connected

Robust platform for

mobile apps

Unified app and

device management

Windows Phone The Right Choice for BusinessSlide4

Topics

Speech on Windows Phone 8Speech synthesisControlling applications using speech

Voice command definition files

Building conversations

Selecting application entry points

Simple speech input

Speech input and grammarsUsing Grammar ListsSlide5

Speech on Windows Phone 8Slide6

Windows Phone Speech Support

Windows Phone 7.5 had voice support built into the operating system Programs and phone features could be activated by voice commands

Incoming SMS messages could be read to the user

The user could compose and send SMS messages

Windows 8 builds on this to allow applications to make full use of speechSlide7

Windows Phone 8 Speech

Applications can speak messages using the Speech Synthesis featureApplications can be started and given commands

Applications can accept commands using voice input

Free speech recognition requires an internet connection, but Speech Synthesis and “programmed recognition” does notSlide8

Speech SynthesisSlide9

Enabling Speech Synthesis

If an application wishes to use speech output the ID_CAP_SPEECH_RECOGNITION capability must be enabled in WMAppManifest.xml

The application can also bring in the Synthesis namespace

using

Windows.Phone.Speech.Synthesis

;Slide10

Simple Speech

The

SpeechSynthsizer

class provides a simple way to produce speech

The

SpeakTextAsync

method speaks the content of the string using the default voice

Note that the method is an asynchronous one, so the calling method must be flagged as asyncSpeech output does not require a network connectionasync void CheeseLiker

(){ SpeechSynthesizer synth = new SpeechSynthesizer(); await synth.SpeakTextAsync("I like cheese.");}Slide11

Speech Exceptions

If the application is interrupted during speech output the speech method will throw an exception if the application is resumed

try

{

SpeechSynthesizer

synth =

new SpeechSynthesizer(); await

synth.SpeakTextAsync("I like cheese.");}catch ( Exception ex) { if (((uint)ex.HResult == 0x80045508)) { // System Call Interrupted thrown by Speech }

}Slide12

Speech Exceptions

Your program must catch this exception or it will fail certification

try

{

SpeechSynthesizer

synth =

new SpeechSynthesizer(); await

synth.SpeakTextAsync("I like cheese.");}catch ( Exception ex) { if (((uint)ex.HResult == 0x80045508)) { // System Call Interrupted thrown by Speech } }Slide13

Selecting a language

The default speaking voice is selected automatically from the locale set for the phone

The

InstalledVoices

class provides a list of all the voices available on the phone

The above code selects a French voice

// Query for a voice that speaks French.

var frenchVoices = from voice in

InstalledVoices.All where voice.Language == "fr-FR" select voice;// Set the voice as identified by the query.synth.SetVoice(frenchVoices.ElementAt(0));Slide14

Speech Synthesis Markup Language

You can use Speech Synthesis

Markup

Language (SSML) to control the

spoken

output

Change the voice, pitch, rate, volume, pronunciation and other characteristicsAlso allows the inclusion of audio files into the spoken outputYou can also

speak the contents of a file <?xml version="1.0" encoding="ISO-8859-1"?><speak version="1.0"

xmlns=http://www.w3.org/2001/10/synthesis xml:lang="en-US"> <p> Your <say-as interpret-as="ordinal">1st</say-as> request was for <say-as

interpret-as="cardinal">1</say-as> room on <say-as interpret-as="date" format="mdy">10/19/2013</say-as>

, arriving at <

say-as

interpret-as="time" format="hms12">12:35pm</say-as>. </p></speak>Slide15

“Speech Markup”

Rob MilesSlide16

SSML Languages

The markup langauge behaviour is sensitive to the locale where the phone is being used

I had some problems when I used en-GB to decode the mark-up file

It works fine with en-US howeverSlide17

Controlling Applications using Voice CommandsSlide18

Voice command Application Launching

The Voice Command feature of Windows Phone 7 allowed users to start applicationsIn Windows Phone 8 the feature has been expanded to allow the user to request data from the application in the start command

The data will allow a particular application page to be selected when the program starts and can also pass request information to that pageSlide19

Adding Voice Commands

To start using Voice Commands you must first request the speech capability for your applicationThe application then calls a method to register “trigger” words and phrases the first time it is runSlide20

A Voice Command Definition (VCD) file

The VCD file can be loaded from the application or from any URIIn the above case it is a file that has been added to the project and marked as Content

The voice commands for an application are loaded into the voice command service when the application runs

The application must run at least once to configure the voice commands

The

voice command behaviour

can also be changed by the application

async void setupVoiceCommands(){

await VoiceCommandService.InstallCommandSetsFromFileAsync( new Uri("ms-appx:///VCDCommands.xml", UriKind.RelativeOrAbsolute));}Slide21

The Cheese Reminder Program

The Cheese Reminder program will remind you to eat some cheese at a slightly later timeThe time is set by using voice commands

Do not use this application if you are lactose intolerantSlide22

CommandPrefix

Cheese Reminder

CommandPrefix

<Example> ten minutes </Example>

<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>

<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>

PhraseList

Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Voice Command Definition (VCD) fileThis is the VCD content to set the reminder:“Cheese Reminder ten minutes”Slide23

CommandPrefix

Cheese Reminder

CommandPrefix

<Example> ten minutes </Example>

<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>

<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>

PhraseList

Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Trigger PhraseThis is the phrase the user says to trigger the Cheese Reminder applicationSlide24

CommandPrefix

Cheese Reminder

CommandPrefix

<Example> ten minutes </Example>

<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>

<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>

PhraseList

Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Example TextThis text is used to show a sample phrase that can be used with the applicationSlide25

CommandPrefix

Cheese Reminder

CommandPrefix

<Example> ten minutes </Example>

<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>

<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>

PhraseList

Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Command NameThe command name is passed into the application so that it knows what the user has saidSlide26

CommandPrefix

Cheese Reminder

CommandPrefix

<Example> ten minutes </Example>

<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>

<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>

PhraseList

Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Command ExampleAn example for this particular commandSlide27

CommandPrefix

Cheese Reminder

CommandPrefix

<Example> ten minutes </Example>

<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>

<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>

PhraseList

Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Command to Listen ForSound to listen forElements in square brackets are optionalSlide28

CommandPrefix

Cheese Reminder

CommandPrefix

<Example> ten minutes </Example>

<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>

<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>

PhraseList

Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Phrase ListThis is a list of phrases that can follow the commandThe program can determine which phrase was spoken by the userSlide29

CommandPrefix

Cheese Reminder

CommandPrefix

<Example> ten minutes </Example>

<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>

<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>

PhraseList

Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>FeedbackThis gives the format of spoken feedback for the commandSlide30

CommandPrefix

Cheese Reminder

CommandPrefix

<Example> ten minutes </Example>

<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>

<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>

PhraseList

Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Navigation URLThis identifies the page in the application which will be loaded when the command is recognisedSlide31

CommandPrefix

Cheese Reminder

CommandPrefix

<Example> ten minutes </Example>

<Command Name="SetReminder"> <Example> ten minutes </Example> <ListenFor> [set] {number} {units} </ListenFor>

<Feedback> Setting a reminder... </Feedback> <Navigate Target="ReminderSetPage.xaml"/> </Command>

PhraseList

Label="number"> <Item> one </Item> <Item> two </Item> </PhraseList>Phraselist contentsThese are the phrases that can follow the commandThe program can determine which phrase was givenSlide32

Picking up Voice Commands

This code runs in the OnNavigatedTo

method of a target page

Can also check for the voice command phrase that was used

(

e.NavigationMode

== System.Windows.Navigation.NavigationMode.New) { if (

NavigationContext.QueryString.ContainsKey("voiceCommandName")) { switch (NavigationContext.QueryString["voiceCommandName"]) { case "SetReminder": string

delayNumberString = NavigationContext.QueryString["number"]; break; } }}Slide33

Cheese Reminder

Rob MilesSlide34

Identifying phrases

The navigation context can be queried to determine the phrase used to trigger the navigationIn this case the program is selecting the units used in setting the reminder

PhraseList

Label

"units"> <Item

> minute </Item> <Item> minutes </Item> <Item> hour </Item> <Item> hours </

Item></PhraseList>string delayUnits = NavigationContext.QueryString["units"];Slide35

Modifying the phrase list

An application can modify a phrase list when it is runningIt cannot add new commands howeverThe program must be able to make use of these modified names

VoiceCommandSet

fortuneVcs

VoiceCommandService

.InstalledCommandSets["en-US"];await

fortuneVcs.UpdatePhraseListAsync("units", new string[] { "hours", "minutes", "seconds", "millenia" });Slide36

Finding the phrase list

The command set name in the InstalledCommandSets

maps onto the

Name

field in the voice command xml file

VoiceCommandSet

fortuneVcs = VoiceCommandService.InstalledCommandSets["en-gb

"];<CommandSet xml:lang="en-gb" Name="en-gb">Slide37

Handling Languages

You can create customised command files for each languageWhen your application runs it can detect which language is in use on the phone and find a matching command fileSlide38

Language identification

I use a list of supported languages and then check the CultureInfo

setting on the phone

string

[]

supportLangauges

new string[] { "en-us", "en-

gb" };string language = CultureInfo.CurrentCulture.Name.ToLower(); foreach (string supportedLanguage in supportLangauges) if (language == supportedLanguage) return

true;// got the languageSlide39

First Time Run

The first time your program runs the customised phrase list may not be ready for useYou have to handle this situation gracefullyDisplay a message and ask the user to run the program againSlide40

Installation checking

If you assume the command set is present your program will throw an exception the first time it runs

VoiceCommandService

.InstalledCommandSets.ContainsKey

(language))

{ MessageBox.Show("Voice Commands Loading" +

"Please restart the program to set up the voice commands." ); return;}Slide41

Voice Music

Voice Music builds a phrase list from the music on your phoneYou can then select music and playlists using voice commandsYou can get it from the Windows Phone Marketplace

It’s free

 Slide42

Creating the album name phrase list

This method adds all the album names to the phrase recognition

async

void

addAlbumCommands

(VoiceCommandSet widgetVcs){

using (MediaLibrary library = new MediaLibrary()) { string[] albumNames = new string[library.Albums.Count]; for (int i = 0; i < library.Albums.Count

; i++) albumNames[i] = library.Albums[i].Name; await widgetVcs.UpdatePhraseListAsync("albumName", albumNames); }}Slide43

Voice Music

Rob MilesSlide44

Free Speech InputSlide45

Recognizing Free Speech

A Windows Phone application can recognise words and phrases and pass them to your programFrom my experiments it seems quite reliable

Note that a network connection is required for this feature

Your application can just use the speech string directly

The standard “Listening” interface is displayed over your applicationSlide46

Simple Speech Recognition

The above method checks for a successful responseBy default the system uses the language settings on the Phone

SpeechRecognizerUI

recoWithUI

;

async

private void ListenButton_Click

(object sender, RoutedEventArgs e){ this.recoWithUI = new SpeechRecognizerUI(); SpeechRecognitionUIResult recoResult = await recoWithUI.RecognizeWithUIAsync(); if

( recoResult.ResultStatus == SpeechRecognitionUIStatus.Succeeded ) MessageBox.Show(string.Format("You said {0}.", recoResult.RecognitionResult.Text));

}Slide47

Customizing Speech Recognition

InitialSilenceTimeout The time

that

the speech recognizer will wait

until it hears speech.

The

default setting is 5 seconds.BabbleTimeoutThe time

that the speech recognizer will listen while it hears background noiseThe default setting is 0 seconds (the feature is not activated). EndSilenceTimeout The time interval during which the speech recognizer will wait before finalizing the recognition operationThe default setting is 150 milliseconds. Slide48

Customizing Speech Recognition

A program can also select whether or not the speech recognition echoes back the user input and displays it in a message boxThe code above also sets timeout values

recoWithUI.Settings.ReadoutEnabled

false

;

// don't read the saying back

recoWithUI.Settings.ShowConfirmation = false; // don't show the confirmation

recoWithUI.Recognizer.Settings.InitialSilenceTimeout = TimeSpan.FromSeconds(6.0);recoWithUI.Recognizer.Settings.BabbleTimeout = TimeSpan.FromSeconds(4.0);recoWithUI.Recognizer.Settings.EndSilenceTimeout = TimeSpan.FromSeconds(1.2);Slide49

Handling Errors

An application can bind to events which indicate problems with the audio inputThere is also an event fired when the state of the capture changes

recoWithUI.Recognizer.AudioProblemOccurred

Recognizer_AudioProblemOccurred

;

recoWithUI.Recognizer.AudioCaptureStateChanged

+= Recognizer_AudioCaptureStateChanged;...

void Recognizer_AudioProblemOccurred(SpeechRecognizer sender, SpeechAudioProblemOccurredEventArgs args){ MessageBox.Show("Please speak more clearly");}Slide50

Important Note

Just a reminder that if you use Free Speech input the phone will require a network connection to decode the text

This is not the case with commands you build into the application

They can be used at any time, even if you have customised themSlide51

Using GrammarsSlide52

Grammars and Speech input

The simple speech recognition we have seen so far uses the “Short Dictation” grammar which just captures the text and returns it to the applicationYou can add your own grammars that will structure the conversation between the user and the applicationSlide53

Grammars and Speech input

Grammars can be created using the Speech Recognition Grammar Specification (SRGS) Version 1.0 and stored as XML files loaded when the application runs

This is a little complex, but worth the effort if you want to create applications with rich language interaction with the user – there are examples on MSDN

If the application just needs to identify particular commands you can use a grammar list to achieve thisSlide54

Using Grammar Lists

To create a Grammar List an application defines an array of strings that form the words in the listThe Grammar can then be added to the recognizer and given a name

Multiple grammar lists can be added to a grammar recognizer

The recognizer will now resolve any of the words in the lists that have been supplied

string

[]

strengthNames

= { "weak", "mild", "medium",

"strong", "english"};recoWithUI.Recognizer.Grammars.AddGrammarFromList("cheeseStrength", strengthNames);Slide55

Enabling and Disabling Grammar Lists

An application can enable or disable particular grammars before a recognition actionIt is also possible to set relative weightings of grammar lists

The text displayed as part of the listen operation can also be set, as shown above

recoWithUI.Settings.ListenText

"How strong do you like your cheese?"

;

recoWithUI.Recognizer.Grammars["cheeseStrength"

].Enabled = true;SpeechRecognitionUIResult recoResult = await recoWithUI.RecognizeWithUIAsync();Slide56

Result confidence

An application can determine the confidence that the speech system has in the result that was obtainedResult values are High, Medium, Low, Rejected

SpeechRecognitionUIResult

recoResult

await

recoWithUI.RecognizeWithUIAsync();if

( recoResult.RecognitionResult.TextConfidence == SpeechRecognitionConfidence.High ){ // select cheese based on strength value}Slide57

Matching Multiple Grammars

If the spoken input matches multiple grammars a program can obtain a list of the alternative results using

recoResult.RecognitionResult.GetAlternatives

The list is supplied in order of confidence

The application can then determine the best fit from the context of the voice request

This list is also provided if the request used a more complex grammar

var

alternatives = recoResult.RecognitionResult.GetAlternates(3);Slide58

Profanity

Words that are recognised as profanities are not displayed in the response from a recognizer commandThe speech system will also not repeat themThey are enclosed in

when supplied to the program that receives the speech dataSlide59

Review

Applications in Windows 8 can use speech generation and recognition to interact with usersApplications can produce speech output from text files which can be marked up with Speech Synthesis Markup

Language (SSML)

to include text

Applications can be started and provided with initial commands by registering a Voice Command Definition File with the Windows Phone

The commands can be picked up when a page is loaded, or the commands specify a particular page to load

An application can modify the phrase part of a command to change the activation commands

Applications can recognise speech using complex grammars or simple word listsSlide60