Nick Waggoner Senior Program Manager 4022 Demo Photo annotation Input and hittesting Windows touch l anguage Panning and zooming Pen and ink Agenda Where we are today Deep and m ature ID: 509559
Download Presentation The PPT/PDF document "Make my desktop app great with touch, mo..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1Slide2
Make my desktop app great with touch, mouse, and pen
Nick Waggoner
Senior Program Manager
4-022Slide3
Demo: Photo annotationSlide4
Input and hit-testingWindows touch l
anguage
Panning and zooming
Pen and ink
AgendaSlide5
Where we are today
Deep
and m
ature
Diverse ecosystem
Primarily mouse and keyboardSlide6
PC Form
Factors
have evolved. Slide7
Key platform APIs
Direct2D,
DirectWrite
Direct3D
Dynamic
DPI
Desktop Applications
DirectComposition
Remainder of Win32
Realtime
Stylus
Ink Recognition
InteractionContext
WM_POINTER Family
DirectManipulation
Win32 apps
Windows
kernel
s
ervices
Input and Interaction
Rendering and composition
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer
family
DirectManipulation
Input and interactionSlide8
I
nput
and hit-testingSlide9
WM_POINTER
Unified messages for touch, mouse, and pen.
EnableMouseInPointer
(…)
GetPointerInfo
(…) /
GetPointerInfoHistory
(…)
GetPointerTouchInfo
(…)
/
GetPointerPenInfo
(…)
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer
family
DirectManipulationSlide10
Touch hit testing
RegisterTouchHitTestingWindow
(…)
WM_TOUCHHITTESTING
EvaluateProximityToRect
(…) or
EvaluateProximityToPolygon
(…)
PackTouchHitTestingProximityEvaluation
(…)
Control C
Button 1
Button 2
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer
family
DirectManipulationSlide11
Demo: Moving to WM_POINTER & WM_TOUCHHITTESTINGSlide12
Windows touch
l
anguageSlide13
Windows touch language
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer
family
DirectManipulationSlide14
InteractionContext
Initialization
CreateInteractionContext
(…)
SetInteractionConfigurationInteractionContext
(…)
RegisterOutputCallbackInteractionContext
(…)
Feeding input
AddPointerInteractionContext
(…)
ProcessPointerFramesInteractionContext
(…)
Handling
output
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer family
DirectManipulation
void
INTERACTION_CONTEXT_OUTPUT_CALLBACK( _In_opt
_ void *clientData,
_In_ const
INTERACTION_CONTEXT_OUTPUT *output ); Slide15
typedef
struct
INTERACTION_CONTEXT_OUTPUT {
INTERACTION_ID
interactionId
;
INTERACTION_FLAGS
interactionFlags
;
POINTER_INPUT_TYPE
inputType
;
float
x;
float
y;
union
{ INTERACTION_ARGUMENTS_MANIPULATION manipulation; INTERACTION_ARGUMENTS_TAP tap; INTERACTION_ARGUMENTS_CROSS_SLIDE
crossSlide; } arguments; } INTERACTION_CONTEXT_OUTPUT;
InteractionContext – Output structuresSlide16
typedef
struct
INTERACTION_ARGUMENTS_MANIPULATION {
MANIPULATION_TRANSFORM delta;
MANIPULATION_TRANSFORM cumulative;
MANIPULATION_VELOCITY velocity;
MANIPULATION_RAILS_STATE
railsState
;
} INTERACTION_ARGUMENTS_MANIPULATION;
InteractionContext
– Output structuresSlide17
Demo: Integrating the touch
l
anguageSlide18
Panning and zoomingSlide19
Drives panning and zooming in the system
Slide and pinch
Encapsulates the personality
Inertia
Bounce effects Snap-points
DirectManipulation
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer
family
DirectManipulationSlide20
Pre-declare your behaviors/interactions
DirectManipulation
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer
family
DirectManipulation
Harness the power of Windows
with
DirectCompositionSlide21
Initialization
IDirectManipulationManager
IDirectManipulationViewport
IDirectManipulationContent
Hit-testing
SetContact
()
WM_POINTERDOWN / DM_POINTERHITTEST
Manipulation begins
WM_POINTERCAPTURECHANGED
DirectManipulation
– Key interfaces/messages
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer
family
DirectManipulationSlide22
Adding panning and zoomingSlide23
Best performance by integrating
DirectManipulation
with a 3 thread model:
UI thread
Delegate thread
Independent hit-test
t
hread
IDirectManipulationManager
::
RegisterHitTestTarget
(…)
DirectManipulation
- threading
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer
family
DirectManipulationSlide24
Pen and inkSlide25
Independent of the UI thread
Handwriting recognition
RealTimeStylus
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer
family
DirectManipulationSlide26
Initialization
IRealTimeStylus
AddStylusSyncPlugin
(…)
Processing Input
IStylusSyncPlugin
StylusDown
(…), Packets(…),
StylusUp
(…)
IStrokeBuilder
BeginStroke(…),
AppendPackets(…), EndStroke(…)
Handwriting Recognition InkRecognizerContext
Recognize(…), BackgroundRecognize(…),
BackgroundRecognizeWithAlternates(…)
RealTimeStylus - Key interfaces/methods
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer family
DirectManipulationSlide27
RealTimeStylus
-
Key interfaces/methods
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer
family
DirectManipulation
IRealTimeStylus
IStylusSyncPlugin
IStrokeBuilder
InkRecognizerContextSlide28
Adding ink and recognitionSlide29
Performance is critical for both inking as well as panning and zooming experiences.Slide30
RecapSlide31
Use
WM_POINTER
for touch, mouse, and pen.
Use the
InteractionContext
to incorporate the Windows touch language into your app.
Use
DirectManipulation
for great panning and zooming.
Differentiate your app by making ink pop with the
RealTimeStylus
APIs.
Takeaways
Realtime
stylus
Ink recognition
InteractionContext
WM_Pointer
family
DirectManipulationSlide32
Resources
Related
talks
3-017 - Power
up your desktop app with Windows
4-021 -
DirectComposition
: Smooth composition and animation for desktop
apps
4-9147 – Exploring customized touch interactions for your Windows store app
Reference documentation
WM_POINTER
-
http://msdn.microsoft.com/en-us/library/windows/desktop/hh454916(v=vs.85).
aspx InteractionContext - http://msdn.microsoft.com/en-us/library/windows/desktop/hh437192(v=vs.85).
aspx DirectManipulation - http://msdn.microsoft.com/en-us/library/windows/desktop/hh446972(v=vs.85).
aspx RealTimeStylus - http://msdn.microsoft.com/en-us/library/windows/desktop/ms701683(v=vs.85).
aspx InkRecognizerContext - http://msdn.microsoft.com/en-us/library/windows/desktop/ms696371(v=vs.85).
aspxProcess touch input with DirectManipulation sample http://go.microsoft.com/fwlink/?LinkID=309852&clcid=0x409Design guidelines
Touch target sizes
- http://msdn.microsoft.com/en-us/library/windows/apps/hh465415.aspx#touch_targets Slide33
Device distribution starts after sessions conclude today (
approximately
6:00pm) in the Big Room, Hall D.
If you choose not to pick up your devices tonight, distribution
will
continue for the duration of the conference at Registration
in
the North
Lobby.
Get your goodies
Acer
Iconia
W3, Surface Pro,
and Surface
Type CoverSlide34
Evaluate this session
Scan this QR code
to evaluate this session and be automatically entered in a
drawing
to
win
a
prize!
Required Slide
*delete this box when your slide is finalized
Your MS Tag will be inserted here during the final scrub. Slide35