SurroundSense - PowerPoint Presentation

Slide1

SurroundSense:Mobile Phone Localizationvia Ambience Fingerprinting

Martin

Azizyan

Duke University

Ionut

Constandache

Duke University

Romit

Roy Choudhury Duke UniversitySlide2

AbstractMobile computing applications center around user’s locationTerm- Physical Location

(

coordinates Latitude and Longitude)

Term- Logical Location

(

like Starbucks or McDonalds)

Lots of Research available for physical

location

Few attempts recognizing logical

locationSlide3

AbstractAmbient sound, light, and color from phone’s camera and

microphone

A

ccelerometers for user-motion

Adjacent

stores can be separated

logically

They

propose

SurroundSense

, a mobile phone based system

explores ambience fingerprinting

51

different stores

- average

accuracy of 87% when all sensing modalities are

employedSlide4

IntroductionStarbucks -coffee machines and microwaves

Restaurants

-forks

and spoons

clinking

Target – red colors

Panera Breads- yello

w colors

Floors

with

carpets

, ceramic

tiles, or wooden

strips

Bars -dim

yellow lights

BlockBuster

-

bright white

lightSlide5

IntroductionWal-Mart - walking up and down aisles

Barnes

and Noble

-relaxed

stroll with long

pauses

R

estaurants - short

queuing

followed by

a long duration of

sitting

May not

be unique based on any one attribute, the

combination of

all

exhibits diversitySlide6

SurroundSense ArchitectureSlide7

SurroundSense ArchitectureMobile phone user visits

an

unknown store

.

The

phone senses the

ambience

V

alues forwarded

to the fingerprinting

factory

Types

of

sensor data

identified

P

hone’s

(

GSM-based) physical

coordinates

Geographical database

Fingerprint database and Fingerprint matching

The matching module

-best match for test fingerprintSlide8

System DesignFingerprinting SoundFingerprinting Motion using AccelerometersFingerprinting Color/Light using Cameras

Fingerprinting Wi-Fi

Fingerprint MatchingSlide9

Fingerprinting SoundRecorded ambient sound for one

Time

domain, a

simple fingerprinting scheme based

on

signal amplitude

Acoustic

fingerprints

- computed

the

pair-wise distances

.

Use

sound only as a

filter

The

output

is

fed to the accelerometer

filterSlide10

Sound fingerprints from 3 adjacent storesSlide11

Fingerprinting Motion using AccelerometersHuman movements in a location

Restaurants

-stationary

for long durations

G

rocery

store

- more

mobile

Accelerometer readings -

stationary

vs

in

motion

Vector

machines (SVM),

a popular

data classification

tool

User

movement is prone to

fluctuation

Clothing

store

-

browse long

time

or purchase in haste

A

ccelerometers

as

a filtering mechanism

tooSlide12

Sample Accelerometers TracesSlide13

Fingerprinting Color/Light using CamerasThe wall and floor

colors contribute to theme

Use automatically-taken

phone

pictures

Only floor-facing

pictures are

used

Color/light extraction

Why picture of the floor?

Colors

of carpets

, tiles

, marble, and wooden

floors

HSL - hue-saturation-lightness

Clusters

of color

color-light fingerprintSlide14

Color/light fingerprint in the HSL spaceSlide15

Fingerprinting Wi-FiWiFi fingerprinting no good for

logical places

WiFi

based fingerprinting

-fifth

sensor

MAC

addresses of

visible APs

MAC

addresses

recorded every

5

seconds

Computing

the

fraction of times each unique MAC address

was seen

over all

recordings

A

tuple of fractions

forms

the

WiFi

fingerprint of that

placeSlide16

Fingerprint MatchingSurroundSense uses 4 filtering/matching The (WiFi

, sound, and accelerometer) filters are applied first

Candidate

set

fed

to the color/light-based matching

scheme

Use

the color/light based matching scheme

last

The

final

output is an ordered list of candidates – the top

ranked candidate

is declared to be the location of the

phoneSlide17

ProtoType ImplementationClient and ServerPopulating the Fingerprint Database

Special Note:

SurroundSense

was implemented on Nokia N95

using

Python

platform

.

The

server

-MATLAB

,

Python

code,

data mining toolsSlide18

Client and ServerSensor runs on threads and execute

API

calls

The

accelerometer

samples - 4

readings per second

.

The

audio sampling rate is 8 kHz.

Pictures

are taken every

5

seconds

A

meta file

-stores date

,

time

,

GSM,

camera

mode

The server

- several modules.

A

Data

Manager

formats raw data appropriately.

The

formatted data

sent to

Fingerprinting

Factory

A MATLAB

/ Python

based Filtering/Matching Module

-computes

the top-ranked match.Slide19

Populating the Fingerprint DatabaseHow did they build a fingerprint database?

46

business locations

5

locations in

India

Students

visited 51

stores

Stores visited

multiple times

Design

location

labeling

games. The

person with a best match may win a prize.

More people

play

larger

fingerprint

databaseSlide20

EvaluationPartially Controlled ExperimentationPerformance Pre-Cluster Accuracy

Per-Shop Accuracy

Per-User Accuracy

Per-Sensor AccuracySlide21

Partially Controlled ExperimentationNot performed with a real user base Mobile

phones

in our hand (and not in our pockets

)

Phones took pictures

for color and light fingerprinting.

In uncontrolled environments

, phones

in pocket

New wearable mobile

phones

Wrist

watches and

N

ecklacesSlide22

Mimicking Customer BehaviorGroups of 2 people Went

to different stores

-time-separated

Fingerprinted

every store in

cluster

Behave

like normal

customers

Purchase

coffee

and food

Mimic

the movement of another

customers

Atypical behavior -picking

up pre-ordered food

shopping

very quickly

. Slide23

Performance: Per-Cluster AccuracyEvaluate 4 modes1.

WiFi

-only

2

.

Sound, Accelerometer, Light and Color

3

.

Sound, Accelerometer

4

.

SurroundSense

(SS)

combined all

modes

of ambience

fingerprinting

SurroundSense

average

accuracy

of 87%

All the sensors

90%...Slide24

Performance: Per-Shop Accuracy47% of the shops can be localized perfectly using SurroundSense.

WiFi

displays

bimodal

behavior –

it’s

either

high

accuracy, or

seriously suffers

Clearly, the combination of multi-modal

fingerprinting is bestSlide25

Performance: Per-User AccuracyUser assigned to a random set of stores

Report

the average

accuracy

SurroundSense

users

achieve between 73% and 75%

accuracy

The accuracy grows

to an average of 83% or more for 80% of the

users

The median accuracy is around 88%, while 10% users experience 96% accuracy or

moreSlide26

Performance: Per-Sensor Accuracyhand-picked 6 examples -merits and demerits of each

sensor

Whenever

the accelerometer

is

used-accuracy

is always 100

%

Only

the camera,

-100

%

accuracy in

this

location

Only

color gives average accuracy of 91%. When

sound is

added,

-66%

If the correct

location is filtered out, the final match

incorrectSlide27

Limitations and Future WorkEnergy ConsiderationsEnergy efficient localization

and

sensing

Simple

sensing mechanisms

– when outdoors

Variation in GSM signal strengths

T

emperature

sensing

An

accelerometer trace requires

time

Faster

methods of

localization without

compromising

accuracySlide28

ConclusionLogical location, vs physical coordinates The

main idea

-ambient

sound, light, color, RF,

luser

movement

Fingerprint identifies user’s location

SurroundSense

not a stand-alone technique- use with GSM location

SurroundSense

step

towards

indoor localization

Further

research

-

better

energy management

SurroundSense

a viable solution of the

futureSlide29

Questions/Comments?