reexperience of social home activities Anton Nijholt 소프트컴퓨팅연구실 황주원 Overview Introduction Browsing sharing visiting inhabiting participating Ambient intelligence technology and environments ID: 207543
Download Presentation The PPT/PDF document "Google home: Experience, support and" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Google home: Experience, support andre-experience of social home activitiesAnton Nijholt
소프트컴퓨팅연구실
황주원Slide2
Overview IntroductionBrowsing, sharing, visiting, inhabiting, participating
Ambient intelligence technology and environments
The meeting paradigm
Social and intelligent home environments: support and looking backThe role of autonomous and semi-autonomous embodied agentsSmart and distributed meeting environmentsGeneral background and introductionAMI: from signal processing to interpretationProgress and research resultsVisualization, virtual reality representation and replayConclusions
1Slide3
Introduction (1)2
Ambient Intelligence (AMI)
Definition
일상 생활 속에 존재하는 모든 사물이 지능화되어, 인간의 눈에 띄지 않으면서 언제 어디서든지 인간이 원하는 활동을 편하고 효율적으로 수행할 수 있도록 지원하는 것Feature다양한 공간 중 어느 곳으로 이동하더라도 끊임없이 사용자가 원하는 서비스가 제공된다는 것Slide4
Introduction (1)3Slide5
Introduction (2)4
Ambient Intelligence (AMI)
Function
시스템의 주변 환경과 상황 정보를 파악할 수 있어야 함다양한 조건 하에서도 동적으로 시스템을 조건에 맞게 설정할 수 있어야 함주변 시스템과 효율적으로 상호작용할 수 있는 규칙을 찾고 생성할 수 있어야 함예외적인 사건에 대해서도 사용자에게 불편함을 주지 않고 스스로 복구할 수 있어야 함In this paperGoalProvide inhabitants or visitors of ambient intelligence environments with support in their activities.
Activities
Interactions between inhabitants and between inhabitants and (semi-) autonomous agentsSlide6
Introduction (2)5Slide7
Introduction (3)6
In
this paperGoogles’ search engineSearch engines, tools for retrieval, searching and summarizingThis is familiar to the users and adapted to their preferences.We can build our own personalized and real-life web environmentIn daily activity, during our work, at home and during times of recreation.Slide8
Browsing, sharing, visiting, inhabiting, participating(1)7
Web environments and web tools
Retrieve pictures
Query results can be categorized according to relevance and user’s interests.Text, audio, pictures, video on the web can be queried.A individual homepageShare their diaries, their photo albums, their music and videoIndividual life on the webShare it with friendsMyLifeBitsVirtual communitiesIn virtual 3D environmentsActiveWorldsYou can build your own home and gave it visited by other members of the community.While these artificial worlds allow the display of personal information through chat, choice of avatars and the design of buildings and roomsSlide9
Browsing, sharing, visiting, inhabiting, participating(2)8
Google Map & Google Earth
They allow interactive access to maps and satellite photos and, although presently only for a limited number of locations, 3D views of parts of cities.
Modeling home environments in 3D virtual realityCameras, microphones and other sensors, and from the informationOn-and off-line searching, browsing and participatingWhat becomes possible if we can do this in real-time?We can observe events taking place in reality in a virtual reality representationAccess meta-informationReal-time generation allows real-time interaction with human and virtual agents in these environmentsSlide10
Ambient intelligence technology and environments9
support activities of inhabitants in Ambient intelligence environments
Cameras, microphones and other sensors can be used to detect and capture such activities
In this sectionSupport to individuals and partiesRemote participationOff-line access to the captured informationThis off-line access should also allow the replay of experiences.Sub sectionThe meeting paradigmSocial and intelligent home environments: support and looking backThe role of autonomous and semi-autonomous embodied agentsSlide11
Ambient intelligence technology and environments10
The meeting paradigm (1)
Why we extend the usual viewpoint on ambient intelligence
- 1
Provide real-time support to activities taking place in a smart environment
Memorized these activities
Manipulate
and replay these activities
- 2 (meeting
problems
)
People
who cannot be present to view what is going on
People to remotely participate
Provide access to captured multi-media information about a previous meeting
People
who were present and want to recall part of a meeting
People
who could not attendSlide12
Ambient intelligence technology and environments11
The meeting paradigm (2)
A first research
In order to be able to provide support, the environment is asked to understand the interactions between its inhabitants and between inhabitants
and the environment
The interaction that gas to be perceived does not only include all aspects of focused interaction, but also aspects of unfocused interaction.
A second research
The real-time monitoring of activities
The on-line access to information about activities
On-line remote participation in activities
Influencing activities in smart environments
A third research
This concerns the off-line access to stored information about activities in smart environments.
Retrieval, summarization, and browsingSlide13
Ambient intelligence technology and environments12
Social and intelligent home environments: support and looking back
Our viewpoint is that there are lots of reasons for wanting to look back on a previous activity
Look at events that involve multi-party interaction for which real-time support is usefulSlide14
Ambient intelligence technology and environments13
The role of autonomous and semi-autonomous embodied agents
3D embodied agents
These agents are real-time
controlled by the
behavior of their human equivalents.
An agent can change from semi-autonomous behavior to human-guided and human-controlled behavior.
Maior
-Domo
A domotic controller represented as avatar
Home lab situation : areal kitchen living room
Prepare a meal, create a shopping list, program the washing machine
The user is wearing a wireless microphone to have her conversation with the embodied agent.Slide15
Smart and distributed meeting environments14
“ What do people do at work?
They go to meetings. How do we deal with meetings? What is it about sitting face-to-face that we need to capture? We need software that makes it possible to hold a meeting with distributed participants ㅡ a meeting with interactivity and feeling, such that, in the future, people will prefer being telepresent.” Bill Gates, 1999.Slide16
Smart and distributed meeting environments15
General background and introduction
The earlier AMI project
M4 project (Multi-Modal Meeting Manager)
This projects are concerned with the design of a demonstration system that enables structuring, browsing and querying of archives of automatically analyzed meetings
The meetings take place in a room equipped with multi-modal sensor.
(microphones, cameras → multi-media meeting minutes)
The result of the M4 project was an off-line meeting browser.
The Recently
AMI project
Multi-modal events
The verbal and nonverbal interaction between participants
Many events take place that are relevant for the interaction
→ communication content and form
(someone enters the room, someone distributes a paper, a person opens of closes the meeting…)
cameras, circular microphone arrays, electronic paper,
laper microphones and camerasSlide17
Smart and distributed meeting environments16
AMI: from signal processing to interpretation
The meeting support application requires
The development of tools
That take into account the meeting context
Bottom-up approach : more general observation on the
ories
of verbal and nonverbal communication
Models
This is needed for the integration of the multi-modal streams in order to be able to interpret events and interactions.
These models include statistical models to integrate asynchronous multiple streams and semantic representation formalisms that allow reasoning and cross-modal reference resolution.
Collected information
Person identification using face recognition
Current speaker recognition using multi-modal information
Speaker trackingSlide18
Smart and distributed meeting environments17
Progress and research results (1)
Review of some more detail the research
Data recording and annotation
Meeting modeling
Audio-video processing
Access to multi-modal meeting data
Real-time supportSlide19
Smart and distributed meeting environments18
Progress and research results (2)
Data recording and annotation (1)
AMI Meeting Corpus consisting of 100 h of multi-modal meeting data
The data allows empirical observations and the training of statistical models
(for speech recognition, for gesture and body pose recognition, the recognition of meeting activities and gaze and turn taking behavior of participants.)
Machine learning techniques
This techniques are based on manually annotated meeting data.
Aim
At developing techniques for automatic recognition of properties that have been annotated explicitly in the training sets.Slide20
Smart and distributed meeting environments19
Progress and research results (3)
Data recording and annotation (2)
The rooms were equipped with microphones, both for close-talking and far-field audio, and with cameras capturing close-ups of the participants and cameras that capture global room views.Slide21
Smart and distributed meeting environments20
Progress and research results (4)
Data recording and annotation (3)
Tools have been developed to annotate the meeting data that has been captured.
Interdependencies of annotated phenomena need to be explored in order to allow us or an automatic extraction procedure to understand meeting activities.Slide22
Smart and distributed meeting environments21
Progress and research results (5)
Meeting modeling
Develop technology
Give real-time support to meeting participants.
These participants can be physically present in the same meeting room.
We can gave remote participants
We can gave a situation where all meeting participants are distributed.
Structure and present meeting information
In such a way that it can be more easily accessed, in an off-line manner, after a meeting, by both participants and others that are interested.
When the methods work in real-time, chairpersons and meeting assistants can use this information about the meeting to improve their performance and the meeting process.Slide23
Smart and distributed meeting environments22
Progress and research results (6)
Audio-video processing (1)
Various recognition algorithms
These have been ported to the AMI meeting domain and evaluated.
Automatic recognition from audio, video, audio & video
Recognize what is said by participants
Recognized what is done by participants (physical actions)
Recognize where each participant is, at each time
Recognize participants’ emotional states
Track what (person, object, or region) each participant is focusing on
Recognize the identity of each participantSlide24
Smart and distributed meeting environments23
Progress and research results (7)
Audio-video processing (2)
Speech recognition
Verbal communication is the backbone of meetings.
Automatic transcription of this communication
- meeting analysis, content analysis, browsing, retrieval and summarization
- speech activity detection, evaluation, keyword spotting and phoneme recognition
Localization and tracking
Detecting and tracking of head, face and hands provides us with information about locations
It is a first step towards identifying people, face recognition, facial expression recognition and emotion recognition
Actions and gestures
Recognized what is done by participants (physical actions)
Recognize where each participant is, at each time
Recognize participants’ emotional states
Track what (person, object, or region) each participant is focusing on
Recognize the identity of each participantSlide25
Smart and distributed meeting environments24
Progress and research results (5)
Access to multi-modal meeting dataSlide26
Smart and distributed meeting environments25
Visualization, virtual reality representation and replaySlide27
Smart and distributed meeting environments26
Visualization, virtual reality representation and replaySlide28
27
Conclusions
Ambient intelligence in the home environment
Home automation is important
But providing real-time support to the inhabitants during their activities is important as well
We have to deal with multi-party interaction
That is, there are verbal and nonverbal interactions between the human inhabitants of the environment.
The environment needs some understanding of such interactions