Alex Edgcomb Department of Computer Science and Engineering University of California Riverside httpwwwexaminercomarticlefallprevention Copyright 2014 Alex Edgcomb UC Riverside 1 of 37 ID: 538555
Download Presentation The PPT/PDF document "Automated camera-based fall detection of..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Automated camera-based fall detection of elderly persons
Alex EdgcombDepartment of Computer Science and EngineeringUniversity of California, Riverside
http://www.examiner.com/article/fall-prevention
Copyright © 2014 Alex Edgcomb, UC Riverside.
1
of 37Slide2
Outline
Elderly person falls and background work
SynchSM and moving-region-based fall detectionOther related work
Copyright © 2014 Alex Edgcomb, UC Riverside.
2
of 37Slide3
Falls in the elderly population need to be
detectedLeading cause of injury-related
hospitalization1 and death234% have fallen in the last year
314% have fallen more than
once3Post-fall long lie correlated with passing-away4
50% who experience a long lie pass-away within 6 months4
1Baker, S.P. and A.H. Harvey. Fall injuries in the elderly. Clinics in geriatric medicine, 1985.2Hoyert, D.L., K.D. Kochanek, and S.L. Murphy. Deaths: final data for 1997. National vital statistics reports, 1999.
3
Lord
S.R., J.A. Ward, P. Williams, and K.J. Anstey. An epidemiological study of falls in older community-dwelling
women.
Australian journal of public health, 1993.
4
Wild
, D., U.S. Nayak, and B. Isaacs. How dangerous are falls in old people at home? British medical journal (Clinical research ed.), 1981.
http://www.examiner.com/article/fall-prevention
http://www.presstv.ir/detail/218170.html
Copyright © 2014 Alex Edgcomb, UC Riverside.
3
of 37Slide4
Falls need to be
automatically detected.False alarm rates must be low.
5
Genworth 2013: Cost of care. https
://www.genworth.com/corporate
/about-genworth/industry-expertise/cost-of-care.html
In-home care
$3,400 – 5,800 / mo.
5
http://www.chcb.org/services-programs/medical-care/elder-care
http://
www.lifelinesys.com/content/
lifeline-products/auto-alert
http://www.mobilehelpnow.com/products.php
Copyright © 2014 Alex Edgcomb, UC Riverside.
4
of 37
Discontinued…
AmberSelect and Alert1
Discontinued b/c
false alarms too high
http://
www.primemedicalalert.com/fall-detection.html
Automated monitoring
Under $100 / mo.Slide5
Reasons for video-based assistive monitoring
Detect
manyevents and trends
Privacy
enhance-able
7,8,9
Body-worn
Pro: Anywhere
Con: Not always
worn
6
http://www.mobilehelpnow.com/products.php
Copyright © 2014 Alex Edgcomb, UC Riverside.
5
of 37
6
Bergmann
, J.H.M. and A.H. McGregor. Body-Worn Sensor Design: What Do Patients and Clinicians Want? Annals of Biomedical Engineering. Volume 39, pgs. 2299-2312, 2011.
7
Beach
, S., R. Schulz, K. Seelman, R. Cooper and E. Teodorski. Trade-Offs and Tipping Points in the Acceptance of Quality of Life Technologies: Results from a Survey of Manual and Power Wheelchair Users.
Intl.
Symposium on Quality of Life Technology, 2011.
8
Beach
, S., R. Schulz, J. Downs, J. Mathews, B Barron and K. Seelman. Disability, Age, and Informational Privacy Attitudes in Quality of Life Technology Applications: Results from a National Web Survey. ACM Transactions on Accessible
Computing,
2009
.
9
Demiris
, G., M.J. Rantz, M.A. Aud, K. D. Marek, H.W. Tyrer and M. Skubic, A.A. Hussam. Older adults’ attitudes towards and perceptions of ‘smart home’ technologies: a pilot study. Medical Informatics and The Internet in Medicine, 2004.Slide6
Approaches to camera-based fall detection
Increasing order of computational complexity
Minimum bounding rectangle (MBR)
Head tracking
3D projection
10
standing
laying
1
0
Auvinet
, E., F. Multon, A. Saint-Arnaud, J. Rousseau, and J.
Meunier. Fall
detection with multiple
cameras:
An
occlusion-resistant method based on 3-d silhouette vertical
distribution. Information
Technology
in
Biomedicine
, IEEE Transactions on 15, no. 2 (2011): 290-300.
Copyright © 2014 Alex Edgcomb, UC Riverside.
6
of 37
Image from paper by Auvinet
20Slide7
MBR-based fall detection
11Hung, D.H. and H. Saito. The Estimation of Heights and Occupied Areas of Humans from Two Orthogonal Views for Fall Detection. IEEJ Trans. EIS 133, no. 1, 2013.
12Miaou, S.-G., P.-H. Sung and C.-Y. Huang. A Customized Human Fall Detection System Using Omni-Camera Images and Personal Information.Proceedings of the 1st Distributed Diagnosis and Home Healthcare Conference, 2006
.1
3Thome, N., S. Miguet and S. Ambellouis. A Real-Time, Multiview Fall Detection System: A LHMM-Based Approach. IEEE Transactions on Circuits and Systems
for Video Technology, Vol. 18, No. 11, November 2008.
Copyright © 2014 Alex Edgcomb, UC Riverside.
Hung
–
Occupied area and height
1
1
Miaou – Height-to-width threshold
1
2
7
of 37
Thome – Height and width probabilistic model
1
3Slide8
Rougier – Head’s vertical velocity
1
4
standing
laying
Auvinet – Person’s height volume
16
Anderson – Combine two silhouettes
1
5
H
ead tracking and 3D projection
Copyright © 2014 Alex Edgcomb, UC Riverside.
1
4
Rougier
, C., J. Meunier, A. St-Arnaud, and J. Rousseau. Monocular 3D head tracking to detect falls of elderly people. In Engineering in Medicine and Biology Society, 2006. EMBS'06. 28th Annual International Conference of the IEEE, pp. 6384-6387. IEEE, 2006
.
1
5
Anderson
, D.,
et al.
Linguistic summarization of video for fall detection using voxel person and fuzzy logic. Computer Vision and Image Understanding
113,
2009.
1
6
Auvinet
, E., F. Multon, A. Saint-Arnaud, J. Rousseau, and J. Meunier. Fall detection with multiple cameras: An occlusion-resistant method based on 3-d silhouette vertical distribution. Information Technology in Biomedicine, IEEE Transactions on 15, no. 2, 2011.
8
of 37Slide9
Outline
Elderly person falls and background workMoving-region and synchSM-based fall detection
Other related work
Copyright © 2014 Alex Edgcomb, UC Riverside.
9
of 37Slide10
Person tracking with in-home video via foregrounding
MBR builder
(adjacent pixel groups merged)
MBR filters
(dampen, smooth, & glitch-removal)
Background model
17,18
(Pixel-level GMM based on color)
Copyright © 2014 Alex Edgcomb, UC Riverside.
Stop learning background if insignificant amount of motion
Learn a second frame with MBR area replaced by background
17
Zivkovic
, Z. Improved adaptive Gaussian mixture model for background subtraction. In Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17th International Conference on, vol. 2, pp. 28-31. IEEE, 2004
.
18
OpenCV
. http://opencv.org/. November 2013.
Current
frame
Foreground
Minimum bounding
rectangle (MBR)
Person tracking may occur on the camera itself
Computer vision person trackers tend to be 10x slower because of 3D projections and additional modeling
10
of 37
GMM = Gaussian mixture modelSlide11
Synchronous state machines: Good fit for fall detection
Copyright © 2014 Alex Edgcomb, UC Riverside.
11 of 37
Normal behavior
Rapid descent
Extended lay
Normal behavior
Fall suspected
Fall detected
Rapid descent
Extended lay
Not laying
hadFall = 0
hadFall = 0
hadFall = 1
Sitting, standing, or laying
Descent velocity
hadFallSlide12
Synchronous state machine (SynchSM) fall detection (1 of 3)
Copyright © 2014 Alex Edgcomb, UC Riverside. 12
of 37
MBR tracker
SynchSMs promote capturing specific, modular behavior.Slide13
SynchSM fall detection (2 of 3)
Copyright © 2014 Alex Edgcomb, UC Riverside. 13
of 37Slide14
SynchSM fall detection (3 of 3)
Copyright © 2014 Alex Edgcomb, UC Riverside.
14 of 37
* Camera may only contribute single-camera fall score if a person is observed by that camera.Slide15
SynchSM fall detection vs. state-of-the-art
Copyright © 2014 Alex Edgcomb, UC Riverside.
15 of 37
# of cameras
SynchSM
HungAuvinet
RougierAnderson
Miaou
Thome
1
0.960
-
-
0.955
-
0.900
0.820
2
0.990
0.958
-
-
1.000
-
0.980
3
0.998
-
0.806
-
-
-
-
4
1.000
-
0.997
-
-
-
-
5
1.000
-
0.999
-
-
-
-
6
1.000
-
1.000
-
-
-
-
7
-
-
1.000
-
-
-
-
8
-
-
1.000
-
-
-
-
Sensitivity
A dash (-) means unreported or not applicable, such as Hung’s algorithm that uses exactly two cameras.
# of cameras
SynchSM
Hung
Auvinet
Rougier
Anderson
Miaou
Thome
1
0.995
-
-
0.964
-
0.860
0.980
2
1.000
1.000
-
-
0.938
-
1.000
3
1.000
-
1.000
-
-
-
-
4
0.995
-
0.998
-
-
-
-
5
0.993
-
1.000
-
-
-
-
6
1.000
-
1.000
-
-
-
-
7
-
-
1.000
-
-
-
-
8
-
-
1.000
-
-
-
-
Specificity
22 recordings from University of Montreal data set
;
each recording has multiple labels
Trained on 1 recording by selecting smallest threshold for sit-lay that got perfect accuracy
Tested with all combinations of remaining 21 videos
Did not use OK-to-lay SMs.
MBR
MBR
MBR
MBR
Head
3D proj.
3D proj.Slide16
Fall behavior coverage
Fall
behaviorSynchSMHung
Auvinet
RougierAnderson
MiaouThome
Suspected fall eventY
Y
Y
Person orientation
Y
Y
Y
Y
Y
Y
Y
Fall sense
Y
Y
Y
Y
Y
Copyright © 2014 Alex Edgcomb, UC Riverside.
16
of 37
MBR
MBR
MBR
MBR
Head
3D proj.
3D proj.
Did not consider sudden downward movement
Did not give time for person to get upSlide17
Trade-off: Accuracy and efficiency (1 of 2)
Copyright © 2014 Alex Edgcomb, UC Riverside.
17 of 37
Higher is better. Closest to top-right is best.
Combined accuracy score = sensitivity
*
specificitySlide18
Trade-off: Accuracy and efficiency
(2
of 2)
Copyright © 2014 Alex Edgcomb, UC Riverside.
18 of 37
Higher is better. Closest to top-right is best.
Combined accuracy score = sensitivity
*
specificitySlide19
SynchSM fall detection on other data sets
Ran synchSM fall detection on 55 of my own recordings (26 falls, 29 non-falls) using 1 and 2 cameras.Perfect accuracyRan synchSM fall detection on 22.5 hours of normal activity videos using 1 and 2 cameras.
Perfect accuracyCopyright © 2014 Alex Edgcomb, UC Riverside.
19 of 37Slide20
With more resources, can synchSMs do better?Can head tracking improve fall detection?
Copyright © 2014 Alex Edgcomb, UC Riverside.
20 of 37Slide21
Fall detection accuracy: 2D/3D head tracking vs MBR tracking
87 video recordings, 1 min each1969 non-confounding recordings (35 fall, 34 non-fall)
18 confounding recordings (5 fall, 13 non-fall)Automated MBR: height, width, and topManual 2D head tracking by clicking on headManual 3D head tracking by estimating head height from groundSame synchSMs. Suspected fall used feature.
Perfect accuracy with non-confounding scenarios
Copyright © 2014 Alex Edgcomb, UC Riverside.
21 of 37
19Edgcomb, A. and F. Vahid. Video-based fall detection dataset with 2D and 3D head tracking, and moving-region tracking. http://www.cs.ucr.edu/~aedgcomb/3D_2D_head_an_MBR_videos.html, June 2014.
Head vertical position:
342 pixels
Head height:
5.5 feetSlide22
Confounding recordings:Head tracking vs MBR tracking (1 of 2)
Copyright © 2014 Alex Edgcomb, UC Riverside.
22 of 37
Confounding recordings
3D head
2D headMBR top
MBR heightMBR widthCrouch with box
Y
Kneel and move
chair
Y
Sit quickly
Y
Y
Sit then
toss up item
Y
Y
Sit then hands to side
Y
Y
Y
Hands up, down, then lay
Y
Y
Y
Y
Hands up, down, then sit
1
Y
Y
Y
Y
Y
Hands
up, down, then sit 2
Y
Y
Y
Y
Y
Sit then hands up/down
Y
Y
Y
Y
Y
Lay then toss up item
Y
Y
Y
Y
Y
Hands to side then sit
Y
Y
Y
Y
Y
Stand then toss up item
Y
Y
Y
Y
Y
Set cushion on couch
Y
Y
Y
Y
Y
Non-falls
Confused person orientation synchSM
Head tracking could tell that
head not near groundSlide23
Confounding recordings:Head tracking vs MBR tracking
(2 of 2)Copyright © 2014 Alex Edgcomb, UC Riverside.
23 of 37
Confounding recordings
3D head
2D headMBR top
MBR heightMBR widthFall w/ vacuum 1
Y
Y
Y
Y
Y
Fall w/ vacuum 2
Put book in shelf
Y
Y
Y
Y
Y
Look under couch
Y
Y
Y
Y
Y
Take picture off wall
Y
Y
Y
Y
Y
Falls
3D head
2D
head
MBR top
MBR height
MBR width
Sensitivity
0.80
0.80
0.80
0.80
0.80
Specificity
1.00
0.85
0.54
0.54
0.54
Summary
Head tracking allowed rule that head had to be near the ground.
MBR suitable for a variety of scenarios.
If confounding scenarios likely, then head tracking may be justified.
Confused person orientation synchSMSlide24
Outline
Elderly person falls and background workSynchSM and moving-region-based fall detection
Other related work
Copyright © 2014 Alex Edgcomb, UC Riverside.
24
of 37Slide25
Assistive monitoring for the elderly
Copyright © 2014 Alex Edgcomb, UC Riverside.
Assistive monitoring analyzes data from cameras and sensors for events and trends of interest.
25
of 37Slide26
Commercial in-home assistive monitoring
GrandCare
Many sensor-based anomaly detectionand if-then user programmability
QuietCare (Intel and GE)
Motion sensor-based
anomaly detection
BeCloseMany sensor-basedanomaly detection
Copyright © 2014 Alex Edgcomb, UC Riverside.
http://beclose.com
/
http://www.grandcare.com/sensors/
http://www.careinnovations.com
/
26
of 37Slide27
Master’s work – Monitoring and Notification Flow Language (MNFL)20,21
Copyright © 2014 Alex Edgcomb, UC Riverside.
20
Edgcomb
, A., and F. Vahid. Feature extractors for integration of cameras and sensors during end-user programming of assistive monitoring systems. In Proceedings of the 2nd Conference on Wireless Health, p. 13. ACM, 2011
.
21
Edgcomb
, A., and F. Vahid. MNFL: the monitoring and notification flow language for assistive monitoring. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium, pp. 191-200. ACM, 2012.
Spatial
programming
more intuitive than
temporal. Under 8 mins to solve goal.
No compilation. Blocks
are always
executing, so users
get instant
feedback.
27
of 37
EasyNotify exampleSlide28
Estimating daily energy expenditure from video for assistive monitoring22
Actor
Fidelity1
r = 0.996
2r
= 1.0003r
= 0.9834r = 0.997
Combined
r
= 0.997
(Ideal is 1.0)
Average accuracy =
90.9%
(about same as body-worn device)
Copyright © 2014 Alex Edgcomb, UC Riverside.
22
Edgcomb
, A., and F. Vahid. Estimating Daily Energy Expenditure from Video for Assistive Monitoring, IEEE International Conference on Healthcare Informatics (ICHI), 2013
. (to appear)
Energy expenditure levels on Monday
We considered 12 features
Motion in video not correlated with energy expenditure (r = -0.01, p = 0.53)
Horizontal acceleration had highest correlation (r = 0.80,p < 0.01)
Power regression had best fit (R
2
-value = 0.76) compared to linear, logarithmic, and exponential regressions.
Feature F
28
of 37
Data set available:
http://www.cs.ucr.edu/~
aedgcomb/
videoBasedEnergyEstimate.html
Fidelity = correlation(video-based Calories, BodyBugg)
Accuracy = 1 - (|expected - observed|/expected)Slide29
Privacy perception and fall detection accuracy with privacy-enhanced video
23
Copyright © 2014 Alex Edgcomb, UC Riverside.
23
Edgcomb, A., and F. Vahid. Privacy Perception and Fall Detection Accuracy for In-Home Video Assistive Monitoring with Privacy Enhancements, ACM SIGHIT (Special Interest Group on Health Informatics) Record, 2012.
Common privacy enhancements not providing sufficient privacy
Privacy critical for adoption but makes events harder to detect
Did a fall occur? If so,
at about what second in the video?
Does this style provide sufficient privacy for grandpa? Yes / No.
29
of 37
376 participantsSlide30
Falls have a characteristic
shape that
is nearly identical for raw and privacy-enhanced
video
Copyright © 2014 Alex Edgcomb, UC Riverside.
30
of 37Slide31
Privacy-enhanced fall detection24 (1
of 2)
Observed shape
Characteristic
fall shape
Similarity0.84
Dynamic time warping
Non-fall
Non-fall
Fall
Observed shape
0.46
0.88
Binary tree
classification
25
Copyright © 2014 Alex Edgcomb, UC Riverside.
24
Edgcomb
, A. and F. Vahid. Automated Fall Detection on Privacy-Enhanced Video. IEEE Engineering in Medicine and Biology Society, 2012
.
25
Mueen
, A., E. Keogh and N. Young. Logical-shapelets: An Expressive Primitive for Time Series Classification. Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2011
.
DTW established time series technique
Script to produce this image provided by Professor Keogh, 2012.
31
of 37Slide32
Privacy-enhanced fall detection
(2 of 2)
Privacy enhancementAverage sensitivityAverage specificity
Raw
0.91
0.92Blur1.00
0.67Silhouette0.91
0.75
Oval
0.91
0.92
Box
0.82
0.92
Copyright © 2014 Alex Edgcomb, UC Riverside.
-This method does not consider the time a person spends on the ground post-fall.
+More accurate fall detection than human observers.
Binary tree classifier trained on raw video only
Evaluated using leave-one-out method
23 videos, 1 min each
Each video labeled fall or not-fall
32
of 37Slide33
Automated in-home assistive monitoring with privacy-enhanced video
26
Copyright © 2014 Alex Edgcomb, UC Riverside.26Edgcomb, A. and F. Vahid. Automated In-Home Assistive Monitoring with Privacy-Enhanced
Video, IEEE International Conference on Healthcare
Informatics (ICHI), 2013. (to appear)
Already discussed
Energy
trends
Fall detection
33
of 37
Arisen in morning
Person
enters main living
area
Abnormally inactive
Person home but inactive for extended period
In room too long
Enter to left
Exit from left
In region too longSlide34
Energy estimation
fid./acc.
Fall detection sens./spec.In room too long sens./
spec.
Arisen in morning sens. /
spec.In region too long sens. / spec.
Abnormally inactive during day sens./spec.Raw0.997 /90.9%
0.91 /
0.92
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
Blur
0.994 /
80.5%
1.00 /
0.67
1.0 / 1.0
1.0 / 1.0
0.5
/ 1.0
1.0 / 1.0
Silhouette
0.998 /
85.0%
0.91 /
0.75
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
Oval
0.997 /
85.6%
0.91 /
0.92
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
Box
1.000 /
84.3%
0.82 /
0.92
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
1.0 / 1.0
Most goals were achieved equally well
even with privacy enhancements
Copyright © 2014 Alex Edgcomb, UC Riverside.
34
of 37
Data set available:
http://www.cs.ucr.edu
/~
aedgcomb/MNFLevents.html
Trained on raw
video only
MNFL goals trained on different person than tested
MNFL goalsSlide35
The background model tended to learn the blue
much more than the actorCopyright © 2014 Alex Edgcomb, UC Riverside.
35 of 37
Current frame
Background model
Foreground and MBRSlide36
Accurate and efficient algorithms that adapt to privacy-enhanced video
27
Copyright © 2014 Alex Edgcomb, UC Riverside.
27
Edgcomb, A. and F. Vahid. Accurate and Efficient Algorithms that Adapt to Privacy-Enhanced Video for Improved Assistive Monitoring, ACM Transactions on Management Information Systems (TMIS): Special Issue on Informatics for Smart Health and Wellbeing, 2013.
Specific-color hunter
Edge-void filler
Fall detection
Energy estimation
36
of 37Slide37
Contributions
MBR and synchSM-based fall detection is more accurate and efficient than all previous work.MBR and head tracking are equally accurate,
except in very specific cases.Although monitoring goal accuracy degrades with privacy-enhanced video, adaptive algorithms can compensate without loosing computational efficiency.The common privacy enhancements of silhouette and blur provide insufficient privacy, whereas a bounding oval or box were sufficient.Copyright © 2014 Alex Edgcomb, UC Riverside.
37
of 37Slide38
Summary of graduate research
Monitoring and notification flow language for assistive monitoringEdgcomb, A., and F. Vahid. Feature extractors for integration of cameras and sensors during end-user programming of assistive monitoring systems. In Proceedings of the 2nd Conference on Wireless Health, p. 13. ACM, 2011
. (2 pages)Edgcomb, A., and F. Vahid. MNFL: the monitoring and notification flow language for assistive monitoring. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium, pp. 191-200. ACM, 2012.Estimating daily energy expenditure from video for assistive monitoring
Edgcomb, A., and F. Vahid. Estimating Daily Energy Expenditure from Video for Assistive Monitoring, IEEE International Conference on Healthcare Informatics (ICHI), 2013. (to appear)
Participant privacy perceptions and fall detection accuracy with privacy enhancementsEdgcomb, A., and
F. Vahid. Privacy Perception and Fall Detection Accuracy for In-Home Video Assistive Monitoring with Privacy Enhancements, ACM SIGHIT (Special Interest Group on Health Informatics) Record, 2012.
Automated fall detection on videoEdgcomb, A. and F. Vahid. Automated Fall Detection on Privacy-Enhanced Video. IEEE Engineering in Medicine and Biology Society, 2012. (4 pages)Edgcomb, A. and F. Vahid. Accurate and Efficient Video-based Fall Detection using Moving-Region and State
Machines. (To be submitted)
Automated in-home assistive monitoring with privacy-enhanced video
Edgcomb
, A. and F. Vahid. Automated In-Home Assistive Monitoring with Privacy-Enhanced Video, IEEE International Conference on Healthcare Informatics (ICHI), 2013. (to appear)
Edgcomb
, A. and F. Vahid. Accurate and Efficient Algorithms that Adapt to Privacy-Enhanced Video for Improved Assistive Monitoring, ACM Transactions on Management Information Systems (TMIS): Special Issue on Informatics for Smart Health and Wellbeing, 2013
.
Efficacy of digitally-enhanced education
Edgcomb
, A. and F. Vahid. Interactive Web Activities for Online STEM Learning Materials, American Society for Engineering Education Pacific Southwest Section Conference, 2013
.Edgcomb, A. and F. Vahid. Effectiveness of Online Textbooks vs. Interactive Web-Native Content, Proceedings of ASEE Annual Conference, 2014. (to appear)
Copyright © 2014 Alex Edgcomb, UC Riverside.
38