Stan Pennington Program Manager Microsoft Corporation CL34 Agenda Media Foundation MF Hardware Codecs and Asynchronous Media Foundation Transforms MFTs in general MF video capture support with focus on UVC 11 HD webcams ID: 150268
Download Presentation The PPT/PDF document "Media Foundation: Supporting Hardware C..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Media Foundation: Supporting Hardware Codecs and Cameras
Stan PenningtonProgram ManagerMicrosoft Corporation
CL34Slide2
Agenda
Media Foundation (MF) Hardware Codecs, and Asynchronous Media Foundation Transforms (MFTs) in general
MF video capture support, with focus on UVC 1.1 HD webcams
But first, let’s all get up to speed on MF…
MFT
MFSlide3
MF Background: Media Infrastructures in Windows
Perspective
Team focus on unmanaged APIs
Performance concerns for managed data flow components
Non-deterministic finalization of GC
Can expose configuration, control, and status through managed. Legacy and roadmap issues with wrappersFour infrastructuresVideo for Windows (VfW)
DirectShow (DShow)Windows Media Format SDK (WM FSDK)MFSlide4
MF Background: Media Infrastructures in Windows
Perspective (cont’d)
One or both of two development models for ISVs
Applications built on APIs
New data flow components using extensibility points – often driven by media format or DSP needsSlide5
MF Background: Media Infrastructures in Windows
Evolution
1990 1995 2000 2005 2010
VfW
WM FSDK
MF
ACM Codecs
DVD, TV (Filter, DMO, ACM, VCM Codecs)
WM True Streaming (DMO, ACM Codecs)
Key Mainstream Scenarios & New Features (MFT Codecs)
DVD
TV
Current Usage
DShow
Windows 7Slide6
MF Background: Media Infrastructures in Windows
Typical Microsoft usage in Windows 7
MF
Playback of non-streaming ASF, MP3, MPEG-4, and selected AVI permutations for digital still cameras
New metadata handling in Shell Property Handlers
Transcoding and new media sharingDShowDVD, TV, MPEG-2, MPEG-1, custom I/O, video editing
FSDKBest compatibility for ASF true streamingHistoric integration of ASF read, writeSlide7
MF Background: Media Infrastructures in Windows
Use by Windows Media Experiences
Information provided to set expectations, subject to change in servicing or major releases
Windows Media Player (WMP) and WMP OCX
Opportunistically use MF, FSDK, or DShow as necessary for best end-user playback experience
MF (+ DShow source hybrid) for Windows 7 transcodingWindows Media Center (WMC)WMP OCX with custom presenter - for playback other than TV, DVD
Shell drag-and-drop transcoding for devices (Windows 7 only)MF (+ DShow source hybrid)Slide8
MF Background: Media Infrastructures in Windows
Use by Windows Media Experiences (cont’d)
Change: Protected Media Path (PMP) in WMP and OCX
Windows Vista: PMP used for MF even with unprotected content, providing seamless cross-fade in mixed DRM/non-DRM playlists
Windows 7: PMP only used for DRM-protected content in MFSlide9
MF Background: Media Infrastructures in Windows
Use by Windows Media Experiences: Summary
Cannot assume WMP or WMC extensibility from media infrastructure documentation
Only WMP SDK plug-ins are documented as extensibility points, per
http://msdn.microsoft.com/en-us/library/dd564682(VS.85).aspx
WMC extensibility is per http://msdn.microsoft.com/en-us/library/bb895967.aspx
Ask Microsoft for technical clarifications or documentation enhancementsSlide10
MF Background: Why does Microsoft use MF?
Stream objects (IStream) for media input, output
Most connectivity and data flow divorced from streaming components, moved to higher layer
No push-pull component issues
Better potential for dynamic changes
Better control of threadingMore constrained component roles and graphs topologiesSlide11
MF Background: MF Features
All COM, C++
Windows Vista
High-level API: Media Session
Formats: ASF w/ WMV and WMA, MP3Windows 7
Backward compatibilityNew high-level APIs: Source Reader, Sink Writer,
Transcode API (part of “Fast Transcode” feature)Formats: + AVI w/ DV and MJPEG, + MPEG-4 container w/ MPEG-4 ASP, H.264/AVC, AAC LC, HE v1, HE v2, key ACM wrappingAsynchronous MFT and Hardware CodecsCapture support: Device Proxy Slide12
TopoEdit = GraphEdt for MF
demo Slide13
Async MFT
Model
Sync MFT refresher
DMO enhanced for MF media types and samples
Simplest use is to call ProcessInput, ProcessOutput, repeat until output ready status
Need for adding async modelInefficient for high level to poll for hardware completionAsync changes
Locked as sync for backward compatibilityLimited interface use until unlockedEvents fired on input needed, output ready, drain complete, marker processedSlide14
Async MFT
Hardware Acceleration
Candidates
Video and audio codecs, processors
Why codecs as a new model?
No DXVA support for encodeNo efficient hybrid of DXVA decode and non-GPU encode for transcodingSlide15
Async MFT
Hardware Acceleration Usage
Generically, any ISV can write, any ISV can use
Subject to any licensing rules, of course
Otherwise, use local registration (MFTRegisterLocal)Slide16
Async MFT
Hardware Acceleration Usage (cont’d)
Picked up by high-level MF APIs, no need to code directly to Async MFTs,
if:
Overall enable/disable registry keys enabled
MF API opt-in when not using WMP or OCX. Set MF_TOPOLOGY_HARDWARE_MODE attribute on topology Codec has Trusted Merit if overlapping in-box functionality, and preferred media subtypes match, or fallback subtypes not satisfied by in-box component
Can specify directly in own MF topology (MFTEnumEx)Slide17
Async MFT
Hardware Acceleration Usage (cont’d)
For Windows Media Experiences and automatic Media Session selection, must constrain preference
Codec Trusted Merit cryptographic challenge/response
Keys and certificates are managed through Microsoft Hardware Codecs license programSlide18
Hardware Codecs
License Program
Only for video codecs, video processors
Preferred over in-box MFT codecs for Windows Media Experiences if properly licensed through
wmla@microsoft.com
Only for developer of actual codec coreTwo hardware models supported
GPU – vendor writes complete MFTAVStream – vendor uses Microsoft-provided proxy MFT, or writes complete MFTTest suite protects preferenceShares test content with Media Performance Test KitUnder
the HoodSlide19
Hardware Codecs
License Program (cont’d)
Two codec types
Transcoding only
General – for transcoding and playback
Proper MF attributes must be set (see additional resources links)Directly on vendor MFTsThrough KS property sets for AVStream
AVStream MFTs can use hardware data flow via Hardware MediumsUnder
the HoodSlide20
Hardware Codecs
Windows Usage
Windows Media Experiences
WMP and Shell leverage the Transcode API for transcoding to device
Note: No sink extensibility
Several events documented for Hardware Codecs in general are optional in WMP usage. 3rd-party applications may be able to make hardware state machines more efficient by using optional events
Under the HoodSlide21
Hardware Codecs
Merit Rules
Video media subtypes in scope for Trusted Merit *
Under
the Hood
Media subtype
Transcoding
decoderTranscoding
encoderPlayback decoderH.264
YesYesYes *
MPEG-2Yes
No
Yes
*
MPEG-4 Part 2 Advanced Simple Profile (ASP, compatible with DivX/Xvid/3ivx)
Yes
No
Yes
MPEG-4 Part 2 Simple Profile (SP)
Yes
No
Yes
VC-1 and compatible WMV9
Yes
Yes
No
*
WMV not compatible with VC-1
No
No
No
MJPEG
No
No
No
* Test suite requirements are subject to change per the licensing program
** Existing technology may already provide hardware accelerationSlide22
Hardware Codecs
Fast Transcoding
Focused on conversion for portable devices
Based on MF Media Session, implemented in WMP and Shell layers
Non-public DShow wrapper Media Source used for broad source format support, can output compressed or uncompressed
Under
the HoodSlide23
Hardware Codecs
Media Session-Based T
ranscoding
Byte Stream
Media Session
Source Resolver
Media Source
Scheme Handler
Byte Stream
Byte Stream Handler
MFT Decoder
Media Sink
Hardware Acceleration
Topology & Loader
File,
Network
Custom Input
Webcam
Custom Output
Pipeline Topology
MFT Encoder
Proc
Encoding Profile Management
File,
Network
or
or
or
Video Scaling/Conversion, Other Processing
Under
the Hood
GPU-Based Hardware Codec Model Shown
Provided by:
Microsoft
Microsoft
or ISV
IHVSlide24
Hardware Codecs: New Quick Transcoding Sample
demo Slide25
MF Capture
Overview
Device Proxy (DevProxy) – internal object for lighter-weight use of Kernel Streaming (KS) than typical DShow
AVStream driver model supported
USB Video Class (UVC) 1.1 Webcams
Audio capture through WASAPIOther, but not full tuner capabilitiesSlide26
MF Capture
Overview (cont’d)
Picked up by MF high-level APIs through MF Media Source, no need to code directly to DevProxy
Create Media Source and pass in to Source Reader using MFCreateSourceReaderFromMediaSource
Hardware codecs supported by setting
MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS attribute
Focus here on UVC capture1.1 includes value-add plug-in model for camera vendorsDiagram: http://msdn.microsoft.com/en-us/library/ee349147.aspxSlide27
MF Capture
Use with Source Reader
Source Reader high-level API benefits
Easy way to obtain uncompressed or compressed samples from media or device
Has constrained extensibility
Async and sync APIs. Async is bestGood model for feeding a composition engine or adding to existing media app designSlide28
MF Capture
Dev Proxy + Source Reader
Source Reader
Media Source
MFT Decoder *
Hardware Acceleration
Specific Conversions
*
File,
Network
Custom Input
Webcam
Specify URL, Byte Stream, or Media Source
Async or Sync Control and Status
Direct access to consume and modify media samples
*
Optional
Provided by:
Microsoft
IHV
AVStream Hardware Codec Model ShownSlide29
MF Capture: SDK Sample
demo Slide30
MF
Development Tools
Tracing
Challenges
Async programming
Worker queues for Media SourcesConstrained ETW eventsSome Windows Vista WPP events removed for performance optimizations. Windows TMFs not provided
SolutionNew MFTrace tool, planned for release in 2010Detours hooking of public MF interfaces, adding eventsConsolidation of existing eventsUsageLive console outputRedirect to file, process with tool such as TextAnalysisNETTopology and timestamp graphing cmdsSlide31
MF Development
Tools (cont’d)
General
TopoEdit – now aware of hardware-accelerated MFTsSlide32
Summary
MF high-level APIs in Windows 7 provide integration for:
Hardware-accelerated transforms, including Hardware Codecs
Resultant fast transcoding
Simple AVStream-based audio/video capture, including HD webcams compliant with UVC 1.1
Development and tracing tools are being improvedSlide33
Call to Action
Evaluate MF for new designs
Consider MF in opportunities for new functionality or improvements in existing designsSlide34
Additional Resources
MSDN MF documentation in Windows SDK
Top level:
http://msdn.microsoft.com/en-us/library/ms694197(VS.85).aspx
Async MFTs:
http://msdn.microsoft.com/en-us/library/dd317909(VS.85).aspxWriting a custom MFT: http://msdn.microsoft.com/en-us/library/dd940441(VS.85).aspx
Source Reader: http://msdn.microsoft.com/en-us/library/dd940436(VS.85).aspxVideo capture: http://msdn.microsoft.com/en-us/library/dd940439(VS.85).aspxTopoEdit help: http://msdn.microsoft.com/en-us/library/ms694197(VS.85).aspxSlide35
Additional Resources (cont’d)
MSDN AVStream documentation in Windows DDK
Hardware Codec support in AVStream:
http://msdn.microsoft.com/en-us/library/dd568169.aspx
MSDN MF Forum:
http://social.msdn.microsoft.com/forums/en-US/mediafoundationdevelopmentBlog: http://blogs.msdn.com/mfSlide36
YOUR FEEDBACK IS IMPORTANT TO US!
Please fill out session evaluation forms online at
MicrosoftPDC.comSlide37
Learn More On Channel 9
Expand your PDC experience through Channel 9
Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses
channel9.msdn.com/learn
Built by Developers for Developers….Slide38Slide39