04
4 404 Trill Logo Not Found<br>
07
CEDR Complex
Event
Detection and
Response A? 7<br>
08
Select (projection, transform)
Where (filtration)
Join
Anti-semijoin/Outer Join
Union
Aggregation
Pivot/Unpivot (Progressive) Tumbling
(Progressive) Hopping
Sliding
Session
Timeline extension
Timeline truncation
“Point-at-end” Query Lifespan CEDR 8 Regular Expressions and Augmented Finite Automata<br>
09
new SensorReading { time = 7:00am, temperature = 70, id = 0 }, new SensorReading { time = 7:20am, temperature = 68, id = 1 }, new SensorReading { time = 7:40am, temperature = 72, id = 2 }, new SensorReading { time = 8:00am, temperature = 76, id = 3 }, new SensorReading { time = 8:20am, temperature = 80, id = 4 }, new SensorReading { time = 8:40am, temperature = 84, id = 5 } Beginning of Time End of Time ID: 0 ID: 1 ID: 2 ID: 3 ID: 4 ID: 5 Average: 75 Average: 75 Average: 75 9<br>
10
new SensorReading { time = 7:00am, temperature = 70, id = 0 }, new SensorReading { time = 7:20am, temperature = 68, id = 1 }, new SensorReading { time = 7:40am, temperature = 72, id = 2 }, new SensorReading { time = 8:00am, temperature = 76, id = 3 }, new SensorReading { time = 8:20am, temperature = 80, id = 4 }, new SensorReading { time = 8:40am, temperature = 84, id = 5 } Beginning of Time End of Time ID: 0 ID: 1 ID: 2 ID: 3 ID: 4 ID: 5 7:00a 7:20a 7:40a 8:00a 8:20a 8:40a 9:00a Average: null Average: 68 10<br>
11
new SensorReading { time = 7:00am, temperature = 70, id = 0 }, new SensorReading { time = 7:20am, temperature = 68, id = 1 }, new SensorReading { time = 7:40am, temperature = 72, id = 2 }, new SensorReading { time = 8:00am, temperature = 76, id = 3 }, new SensorReading { time = 8:20am, temperature = 80, id = 4 }, new SensorReading { time = 8:40am, temperature = 84, id = 5 } Beginning of Time End of Time ID: 0 ID: 1 ID: 2 ID: 3 ID: 4 ID: 5 7:00a 7:20a 7:40a 8:00a 8:20a 8:40a 9:00a Average: 70 Average: 80 11<br>
12
new SensorReading { time = 7:00am, temperature = 70, id = 0 }, new SensorReading { time = 7:20am, temperature = 68, id = 1 }, new SensorReading { time = 7:40am, temperature = 72, id = 2 }, new SensorReading { time = 8:00am, temperature = 76, id = 3 }, new SensorReading { time = 8:20am, temperature = 80, id = 4 }, new SensorReading { time = 8:40am, temperature = 84, id = 5 } Beginning of Time End of Time ID: 0 ID: 1 ID: 2 ID: 3 ID: 4 ID: 5 7:00a 7:20a 7:40a 8:00a 8:20a 8:40a 9:00a 12<br>
13
new SensorReading { time = 7:00am, temperature = 70, id = 0 }, new SensorReading { time = 7:20am, temperature = 68, id = 1 }, new SensorReading { time = 7:40am, temperature = 72, id = 2 }, new SensorReading { time = 8:00am, temperature = 76, id = 3 }, new SensorReading { time = 8:20am, temperature = 80, id = 4 }, new SensorReading { time = 8:40am, temperature = 84, id = 5 } Beginning of Time End of Time ID: 0 ID: 1 ID: 2 ID: 3 ID: 4 ID: 5 7:00a 7:20a 7:40a 8:00a 8:20a 8:40a 9:00a 13<br>
14
IStreamable<Empty, Payload> stream = ... stream.TumblingWindowLifetime(TimeSpan.FromMinutes(60)) stream.HoppingWindowLifetime( TimeSpan.FromMinutes(60), TimeSpan.FromMinutes(20)) stream.ProgressiveQuantizeLifetime(...) stream.AlterEventLifetime( start => newStartTime(start),
(start, end) => newDuration(start, end)) 14<br>
15
var average = stream
.TumblingWindowLifetime(TimeSpan.FromMinutes(60))
.Select(event => event.temperature)
.Average() 15<br>
16
Left: Right: Beginning of Time End of Time Join: 16<br>
17
Left: Right: Beginning of Time End of Time Join: LASJ: 17<br>
18
Left: Right: Beginning of Time End of Time Output: ? ? 18<br>
19
19 SELECT
I1.TollId,
I1.EntryTime,
I2.ExitTime,
I1.LicensePlate,
DATEDIFF(minute,I1.EntryTime,I2.ExitTime) AS DurationInMinutes
FROM
Input1 I1 TIMESTAMP BY EntryTime
JOIN
Input2 I2 TIMESTAMP BY ExitTime
ON
DATEDIFF(minute,I1,I2) BETWEEN 0 AND 15 SAQL (Stream Analytics)<br>
20
20 SELECT *
FROM Orders o, Shipments s
WHERE
o.id = s.orderId
AND
o.ordertime BETWEEN s.shiptime - INTERVAL '4' HOUR AND s.shiptime Flink SQL<br>
21
21 orangeStream
.join(greenStream)
.where(<KeySelector>)
.equalTo(<KeySelector>) .window(EventTimeSessionWindows.withGap(Time.milliseconds(1)))
.apply (new JoinFunction<Integer, Integer, String> (){
@Override public String join(Integer first, Integer second) {
return first + "," + second;
}
}); Flink Java<br>
22
Left: Right: Beginning of Time End of Time Output: 22<br>
23
Left: Right: Beginning of Time End of Time Output: 23<br>
24
IStreamable<Empty, LeftData> left = ... IStreamable<Empty, RightData> right = ... var joinResult =
left.AlterEventDuration(10).Join(
right.AlterEventDuration(10),
leftEvent => ...,
rightEvent => ...,
(leftEvent, rightEvent) => ...); 24<br>
26
IStreamable<Empty, Payload> stream = ... Beginning of Time End of Time 26 Order
Of
Arrival<br>
27
observable.ToStreamable(...) Beginning of Time End of Time Order
Of
Arrival Throw error?
Drop entirely?
Adjust to current time? 27<br>
28
observable.ToStreamable(DisorderPolicy.Adjust(Timespan.FromMinutes(1).Ticks)…) Beginning of Time End of Time Order
Of
Arrival Adjust to current time 28 Buffer<br>
29
partitionedObservable.ToStreamable(...) Beginning of Time End of Time Order
Of
Arrival Partition 1 Partition 2 29 IStreamable<Key, Payload> stream = ...<br>
30
30 new T { field1 = 0, field4 = “A”, … } new T { field1 = 1, field4 = “B”, … } new T { field1 = 2, field4 = “C”, … } new T { field1 = 3, field4 = “D”, … } new T { field1 = 4, field4 = “E”, … } new T { field1 = 5, field4 = “F”, … } new T { field1 = 6, field4 = “G”, … } new T { field1 = 7, field4 = “H”, … } new T { field1 = 8, field4 = “I”, … } Any type “T”<br>
31
31 Sync Time Other Time Grouping Key Deleted Bits Data Entries, Co-located in an Array new T { field1 = 0, field4 = “A”, … } new T { field1 = 1, field4 = “B”, … } new T { field1 = 2, field4 = “C”, … } new T { field1 = 3, field4 = “D”, … } new T { field1 = 4, field4 = “E”, … } new T { field1 = 5, field4 = “F”, … } new T { field1 = 6, field4 = “G”, … } new T { field1 = 7, field4 = “H”, … } new T { field1 = 8, field4 = “I”, … }<br>
32
32 Sync Time Other Time Grouping Key Deleted Bits Field Column 1 Field Column 2 Field Column 3 Field Column 4 Field Column 5<br>
33
33 Sync Time Other Time Grouping Key Deleted Bits Field Column 1 Field Column 2 Field Column 3 Char Array 4 Field Column 5<br>
35
35 Batch Size: 80k<br>
36
36 Batch Size: 100<br>
37
37 Batch Size: 100 Cache specific to 100-row batches<br>
39
39 IStreamable<Empty, Payload> stream IStreamable<Payload> stream CEDR API In-order/Out-of-order processing<br>
40
40 404 Trill Logo Not Found 404 FASTER Logo Not Found https://github.com/Microsoft/FASTER<br>
41
41 404 Trill Logo Not Found 404 A.M.B.R.O.S.I.A Logo Not Found https://github.com/microsoft/Ambrosia<br>
45
Obligatory “Rate today ’s session” slide Session page on conference website O’Reilly Events App 45 https://github.com/microsoft/Trill
https://github.com/microsoft/TrillSamples<br>
