Tools for Online Technical Collaboration - PowerPoint Presentation

Slide1

Tools for Online Technical Collaboration

Stephen M. WattUniversity of Western Ontario

TRICS, University of Western

Onatrio

,

10 September 2014

Preliminary version of

a talk

to be given

at

West University of Timisoara, Romania, 22

September

2014Slide2

Previous TRICS!Slide3
Slide4

MenuAppetizersCollaboration

Technical Content vs PicturesMainsDigital InkMathematical Handwriting

DesertsPrevious SoftwarePresent GenerationSlide5

CollaborationSlide6

Collaborative SoftwareSlide7
Slide8

Lots of Collaborative SoftwareSlide9

Common FeaturesSlide showsWhiteboardingVoice chatVideo chatImage captureSlide10
Slide11

Technical CollaborationSlide12

Technical CollaborationMissing: MathematicsDiagrams, graphs

Geometric figuresReferencesDocument markupScientific software connections (Maple, Mathematica, GeoGebra, R,…)Slide13

Technical Collaboration

Isn’t a shared whiteboard, with the ability to save images enough????Slide14

Technical Collaboration

No!Slide15

The Treachery of Images(La trahison des images)Slide16

Einstein’s BlackboardEinstein to receive honorary doctorate at Oxford, May 1931.Lecture at Rhodes House.

Board retrieved and preserved by Edmund (“Ted”) Bowen.Nice to look at, but content is trapped.Slide17
Slide18

Digital InkLocation, time information, sometimes also pressure and angles. Capture online pen strokes, not images.Suitable for

Recognition algorithmsSemantic grouping AnnotationManipulation

: search, transformation, archival.Problem: Multiple vendor-specific formats.Slide19
Slide20

Pen-Based MathInput for CAS and document processing.2D editing.

Computer-assisted collaboration.Slide21

Pen-Based MathDifferent than natural language recognition:2-D layout is a combination of writing and drawing.

Many similar few-stroke characters.Many alphabets, used idiosyncratically.Many symbols, each person uses a subset.No fixed dictionary for disambiguation.Slide22

Character RecognitionA story about a UI proposalA story about three statisticians

Concentrate on character recognitionSeveral projects ignore this problemSlide23

Usual Character Reco. Methods

Smooth and re-sample data THENMatch against N models by sequence alignment

ORIdentify “features”, such asCoordinate values of sample points, number of loops, cusps, writing direction at selected points,

etc Use a classification method, such asNearest neighbour, Subspace projection, Cluster analysis, Support Vector Machine

THEN

Rank choices by consulting dictionarySlide24

DifficultiesHaving many similar characters (e.g. for math) means comparison against all possible symbol models is slow.Determining features from points

Requires many ad hoc parameters.Replaces measured points with interpolationsIt is not clear how many points to keep, and most methods depend on number of pointsDevice dependentWhat to do since there is no dictionary?

New ideas are needed!Slide25

What the Computer SeesSlide26

What the Computer SeesSlide27

Orthogonal Series Representation

Main idea: Represent traces as curves, not discrete points and coordinate curves as truncated orthogonal series.

Advantages:Compact – few coefficients neededGeometric – the truncation order is a property of the character set– gives a natural metric on the space of characters

Algebraic – properties of curves can be computed algebraically (instead of numerically using heuristic parameters) Device independent – resolution of the device is not importantSlide28

ProblemsWant fast response – how to work while trace is being captured.

Low RMS does not mean similar shape.Slide29

Problem 1. On-Line InkThe main problem:

In handwriting recognition, the human and the computer take turns thinking and sitting idle.We ask:Can we do useful work while the user is writing and thereby get the answer faster

after the user stops writing?The answer is “Yes”! Slide30

Problem 1. On-Line Series Coefficients

Use Legendre polynomials

as basis on the interval , with weight function

.Collect numerical values for

on

=

arc length.

is not known until the pen is lifted

.

As the sample points are collected

, numerically integrate the moments

After last point, compute series coefficients for

with domain and range scaled to

This uses a simple linear transformation of the moments

.

 Slide31

Problem 1. On-Line Series Coefficients

Transform moments

of

on

Normalize range of

:

 Slide32

Problem 1. On-Line Series Coefficients

Approach works for any inner product with linear weight function.This is the Hausdorff moment problem (1921), shown to be unstable by

Talenti (1987).It is just fine, however, for the dimensions we need.Slide33

Problem 2. Shape vs Variation

The corners are not in the right places.Work in a jet space to force coords & derivatives close.

Use a Legendre-Sobolev inner product

1

st

jet space

set

for

.

Choose

experimentally to maximize

reco

rate.

Can be also done on-line

.

[

Golubitsky

+ SMW 2008, 2009

]

 Slide34

Legendre-

Sobolev Basis

Use Gram-Schmit on

w.r.t <.,.> to compute LS polynomials.

 Slide35

Life in an Inner Product SpaceWith the Legendre-Sobolev

inner product we haveLow dimensional rep for curves (10 + 10 + 1)Compact rep of samples ~ 160 bits [G+W 2009] >99% linear separability => convexity of classes

A useful notion of distance between curves that is very fast to compute Slide36

Distance Between CurvesElastic matching:Approximate the variation between curves

by some fn of distances between sample points.May be coordinate curvesor curves in a jet space.Sequence alignmentInterpolation (“resampling”)Why not just calculate the area?

This is very fast in ortho. series representation.Slide37

Distance Between Curves

 Slide38

Linear SeparabilitySlide39

Linear SeparabilitySlide40

Linear Separability

Can separate classes with

SVM planes.Each class is then (mostly) within its own convex polyhedral cell.Can classify either by

SVM majority voting + run-off elections (96%)Distance to convex hull of k nearest neighbours (97.5%). On-line computation.

 Slide41

RecognitionSome classification methods compute the distance between the input curve and models.

E.g. Elastic matching with DTW takes time up to quadratic in the number of sample points and linear in the number of models.Many tricks and heuristics to improve on this.

E.g. Limit amount of dynamic time warping, pre-classify based on features, ...We can do substantially better.Slide42

Comparison of Candidate to ModelsUse Euclidean distance in the coefficient space.

Just as accurate as elastic matching.Much less expensive.Linear in

d, the degree of the approximation.< 3 d machine instructions (30ns)

vs several thousand!Can trace through SVM-induced cells incrementally. Normed space for characters gives other advantages. Slide43

Choosing between Alternatives

Red class or blue class?Slide44

Choosing between Alternatives

The nearest

samples are blue.

 Slide45

The Joy of Convexity

Can

compute distance of a sample to this

line

Distance to convex

hull of

nearest

neighbors

in class

gives best recognition

[

Golubitsky+SMW

2009,2010]

Convexity

Linear

homotopies

stay within

a

class

 Slide46

Choosing between Alternatives

The nearest convex hull

of neighbors is red. Slide47

TrainingUsing CHKNN allows training with relatively few samples. (Dozens vs Thousands per class)Slide48

Error Rates as Fn of Distance

SVM Convex Hull

Error rate as fn of distance gives confidence measure for classifiers [MKM – Golubitsky + SMW 2009]Slide49

Combining with Statistical InfoEmpirical confidence on classifiers allows geometric recognition of isolated symbols to be combined with statistical methods.

Domain-specific n-gram information:Research mathematics – 20,000 articles from arXiv[MKM -- So+SMW 2005]

2nd year engineering math – most popular textbooks[DAS -- SMW 2008]Inverse problem –

identifying area via n-gram freq! [DML -- SMW 2008]Slide50

Deciding with Confidence Measure

Symbol Recognizer: X  Class1 with Conf

x1Symbol X

in an Expression EContext-Based Predictor:

X



Class

2

with Conf

x

2

X



Class

i



x

i

= max(

x

1

,

x

2

)Slide51

Baseline Estimation

Figure out baseline from the characters,rather than the other way around, which is more ususal.

We can locate some important features by identifying special points.

51

We refer to a point such as this, that determines the height of a metric line, as a

determining point

.Slide52

Baseline EstimationJuxtaposition ambiguity

Handwriting neatening

52

P9 Pq pq p9Slide53

Determining PointsWe focus on European alphabets.We consider 6 types of determining points.

53Slide54

Average SymbolThe average symbol of a set of known samples for a class can be computed as the average point in the functional space,

54Slide55

Deriving from a Reference Symbol

55

Average

Symbol

Sample-1-Initial

Sample-1-Derived

Sample-2-Initial

Sample-2-Derived

Arc-length guess

Arc-length guess

Optimization

OptimizationSlide56

Using HomotopySome samples are far away from the reference symbol.

We use a homotopy between the reference symbol and the target sample in a multi-step method.

56

Average Target

Average Step-1 Step-2 Target Slide57

Number of Steps Required

as fn of Distance to Average Symbol

57

The overall success rate is 99.63%. Each distance interval in the dense area (x≤0.59) contains 2800 samples while each interval in the sparse area (x>0.59) contains 1500 samples except the last one contains 1370 samples.Slide58

SNC for FeaturesSlide59

Sensible Critical PointsFunctional

approx uses non-local informationPuts critical points where they should be.Univ. polynomial root finding.

 Slide60

RepresentationsSlide61

SNC ProblemsWant small perturbations wrt LS norm.Transformation between LS basis and monomial basis ill-conditioned.

Want to compute resultants, etc, without transforming to monomial basis.Can use degree-grading to push some arguments through. How far can we take this?Slide62

Recognition SummaryDatabase of samples

set of LS pointsCharacter to recognize Integrate moments as being writtenLin. trans. to obtain one point in LS spaceClassify by distance to convex hull of

-NN. Slide63

Prior Generations of Software2000 Cross Pad:Slide64

Prior Generations of Software2002 Pocket PC:Slide65

Prior Generations of Software2002-2008 Tablet: Slide66

Prior Generations of Software2008-2013 Java Application: Slide67

InkChat (Java Version)Skype and

GTalk add-on to the Java application.Slide68

Problems

Requires installation:Big hassle for someone to use only once in a while or on all their machines.Limited portability:Users expect versions on Android, iOS, Windows, Mac OSX, Linux, etc… Incompatible software basesFlakey, moving APIs

Need to support multiple devices.Nowadays a single user will want to work across many devices.Slide69

SolutionUse browser infrastructure.Slide70

SolutionUse browser infrastructure.

JavaScript is not a great language for large projects, but…..It is ubiquitous: Telephones, tablets, laptops, …Libraries for many UI elementsOur new recognition algorithms are fast enough 

Rapid development:Prototype developed in 3 months by 3 students.Slide71

Current Generation

Desktop

TelephoneSlide72

Current GenerationTabletSlide73

Simple Interface with device-adapted menusSlide74

Simple Interface with device-adapted menusSlide75

Ink ControlsSlide76

Collaboration:Multiple Users Connected to Same URISlide77

Collaboration:Different Viewports from Different DevicesSlide78

Collaboration:

Pointers for DiscussionsSlide79

Collaboration:Document AnnotationSlide80

Collaboration:Google Hangout EmbeddingSlide81
Slide82

Cloud IntegrationSave or load files to cloud storageDropBoxOthers possible

Previous work to store user profilesSave cloud of ground-truth labelled symbols(corrected/accepted)Future work to store user-defined brushesSlide83

Architectural DirectionSlide84

Architectural DirectionSlide85

Architectural DirectionSlide86

Architectural DirectionSlide87

Architectural DirectionSlide88

Application Web SiteSlide89

Bruce CharJoseph ChoiMichael FriesenOleg Golubitsky

Rui HuVadim MazalovShirley Miao

Maplesoft

Microsoft

MITACS

NSERC

Thanks

Jeliasko

Polihronov

Maya Ramamurthy

Elena

Smirnova

Clare So

Stephen Solis

Coby Viner

James Wake