Srinivas Narayana Princeton University Network Management is Complex Computer networks getting bigger Commodity components high failure rates Outages are costly 300Khour 200 minutesoutage ID: 642750
Download Presentation The PPT/PDF document "Compiling Network Path Queries" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Compiling Network Path Queries
Srinivas Narayana
Princeton UniversitySlide2
Network Management is ComplexComputer networks getting biggerCommodity components: high failure ratesOutages are costly!~ $300K/hour, 200 minutes/outageManagement tools are usefulProgrammatic tools can really help!2“The cost of downtime.” Andrew Lerner, 2014“Downtime, outages and failures: Understanding their true costs.” Evolven, 2012Slide3
Management = Measure + Control3
Network Operators
Measure
Control
Software-Defined Networking (SDN)Slide4
Software-Defined Networking4SDN controllerForwarding
Control logic
RulesSlide5
Measuring is a Hard ProblemA few standard tools:Ping, traceroute, SNMP, NetFlow, tcpdumpAn operator must “join” multiple data streams:Forwarding: controller policy, topology updatesTraffic: packet samples, counters5Slide6
Example: Where’s the Packet Loss?ABSuspect: Faulty network device(s) along the way. 6100 pkts25
pkts
Slide7
ABIdea: “Follow” the path of packets through the network.Example: Where’s the Packet Loss?
Fine-grained (AB) counters
+
Forwarding
Understand Complex Policies
7
100
pkts
25
pkts
Program Complex
RulesSlide8
Lots of Measurement Use-CasesTraffic matrixUneven load balancingDDoS source identificationPort-level traffic matrixCongested link diagnosisSlice isolationLoop detectionMiddlebox traversal orderIncorrect NAT rewriteFirewall evasion...8Resource management Policy enforcementProblem diagnosisComplex join procedureInaccurate resultsHigh overheadsSlide9
Solution ApproachDeclarative Query SpecificationForwarding independenceCo-Measurement independenceHardware independenceQuery-Driven MeasurementAccurate answersPay exactly for what you queryCommodity hardware9Path Query LanguageQuery Run-Time SystemSlide10
Solution Approach101. Path Query Language2. Query Run-Time SystemSDN controller
Payloads
Statistics
3. Optimizations
Query Expressions
Statistics
ForwardingSlide11
Solution Approach2. Query Run-Time System3. OptimizationsExpressive measurementAccurate measurementson commodity hardwareEfficient measurement1. Path Query LanguageSlide12
ABInstead: get A B packet counts each step along paths where A B traffic flowsExample: Where’s the Packet Loss?
12
100
pkts
25
pkts
50
10
0
50
25
25
25
25
50
0
(
srcip
=a &
dstip
=b) ^
repeat
(
groupby
(true, [switch]), MAX_HOPS)Slide13
I. Path Query Language13Slide14
Goal: Declarative Measurement SpecPacket loss localizationUneven load balancingTraffic matrixSlice isolationDDoS source identificationPort-level traffic matrixCongested link diagnosisLoop detectionMiddlebox traversal orderIncorrect NAT rewriteFirewall evasion...14Q: Common Primitives?Slide15
Common Goals?Relate a packet across interfaces it flows throughSpecify multiple related trajectoriesExtract different outputs: packets, counters, ...Specify where to collect measurement15How to design general primitives that are efficiently implementedin the network?Slide16
Common Goals?Relate a packet across interfaces it flows through... in the data plane16Static analysis of policyisn’t enough Slide17
Key Primitive: Packet PathPacket paths: Tests on packets at a single location in networkSame packet must satisfy multiple tests Regular expressions!17GeneralInefficientNarrowEfficientSingle location packet testPacket paths over regular expressionsTests on sets of packetsSlide18
Key Primitive: Packet PathPacket paths: Tests on packets at a single location in networkSame packet must satisfy multiple tests Regular expressions!Multiple related trajectories:SQL-like constructs for aggregationSpecify measurements precisely:Upstream and downstream captureCapture packets, counters, packet samples18Slide19
Expressing Packet PathsRegular expressions: natural to state paths on graphs19
S1
S2
S3
S4
S5
S6
S1 ^ .* ^ S4Slide20
Expressing Packet PathsRegular expressions: natural to state paths on graphs20
S1
S2
S3
S4
S5
S6
S1 ^ .* ^ S4
Boolean packet predicate
pred
::= true | false
| header=value
| location=value
|
pred
&
pred
|
pred
|
pred
| ~
pred
| ingress()
| egress()Slide21
Expressing Packet PathsRegular expressions: natural to state paths on graphs21S1Input packetOutput packetForwardingatom ::= in_atom(pred) | out_atom
(
pred)
| in_out_atom
(pred
,
pred)Slide22
Expressing Packet PathsRegular expressions: natural to state paths on graphs22
S1
S2
S3
S4
S5
S6
i
n_atom
(
sw
=S1) ^
in_atom
(true)* ^
in_atom
(
sw
=S4)
S1 ^ .* ^ S4Slide23
Query Languagepath ::= atom| path ^ path| path | path| path*| path & path| ~path23
p1
p2
“hop”Slide24
24An Example0 or more repetitionsCapture packets evading a firewall in the networkingress()^(~switch=FW)*^egress()ingressegress
i
ngress
i
ngress
egress
egressSlide25
25An Exampleingressegress
i
ngress
i
ngress
egress
egress
Capture packets evading a firewall in the network
ingress
()
^
(~switch=FW
)*
^
egress()Slide26
Query Language: AggregationSwitch-level traffic matrix:26E1E2...I1250100...I212095...............Slide27
Query Language: AggregationSwitch-level traffic matrix:in_atom(ingress())^in_atom(true)*^out_atom(egress())27Count all packets, going from any ingress to any egress.Flow#pkts*1000Slide28
Query Language: AggregationSwitch-level traffic matrix:in_group(ingress(), [switch])^in_atom(true)*^out_group(egress(), [switch])28Group counts by packet’s ingress and egress switch! Traffic matrix!Flow#pktssw=I1, sw=E1250sw=I1, sw=E2100
...
...Slide29
Query Languagepath ::= atom| path ^ path| path | path| path*| path & path| ~path29atom ::= | in_atom(pred) | out_atom(pred) | in_out_atom(pred, pred) | in_group( pred, [fields]) |
out_group(
pred,
[fields])
| in_out_group(
pred,
[fields], pred,
[fields])Slide30
Where to capture matching packets?30UpstreamDownstreamQueried Path
Packet flow
{path}.up
()
{path}.down
()
For a given query: packets may be different
!
Slide31
How to process matching packets?31{path}.set_bucket(bucket)count_bucket()(Get switch counters)packet_bucket()(Send to controller)sampling_bucket()(Get sFlow packet samples)Slide32
More Query Examples32Compiling path queries, to appear in NSDI 2016Slide33
More Query Examples33Compiling path queries, to appear in NSDI 2016Slide34
Path Query Language: SummaryPredicates: single point testsPaths: regular expressions on predicatesAggregation: groupby constructLocation: downstream and upstream measurementResult format: packets, counts, samples34Slide35
II. The Run-Time System35Slide36
Solution Approach36Query expressionsStatistics2. Query Run-Time SystemSDN controller
Payloads
Statistics
3. Optimizations
1. Path Query
L
anguageSlide37
Goal: Query Network MeasurementAccurate answerPay exactly for what you queryCommodity hardware37Slide38
Commodity HW: Match-Action Tables3838match1 action1match2 action2...
Wildcard bit pattern
(ternary matching)
Forward/Drop/ModifySlide39
Commodity HW: Multi-Stage Match-Act39match1 action1match2 action2...match1 action1match2 action2...
...Slide40
Goal: Query Network MeasurementAccurate answerPay exactly for what you queryCommodity hardware40Consider downstream measurement...Slide41
Goal: Query Network MeasurementAccurate answerPay for what you queryCommodity hardware41How to observe a packet’s path accurately in the data plane with low overhead?Slide42
Approach 1: Join Traffic & Forwarding42Traffic datasete.g., NetFlow, SNMPForwarding updatese.g., OF/routing protocol updatesX
Dynamic
Timestamps not aligned!
Packets may be dropped downstream
Deriving
traffic demands for
operational
IP
networks.
Feldmann
et al., 2001
Sampled
NetFlowSlide43
Approach 1: Join Traffic & Forwarding43Traffic datasete.g., NetFlow, SNMPForwarding updatese.g., OF/routing protocol updatesXDynamicTimestamps not aligned!
??
Ambiguity between identical packets downstream
Packet
traceback
for software-defined networks
. Zhang et al., 2015Slide44
Approach 1: Join Traffic & Forwarding44Traffic datasete.g., NetFlow, SNMPForwarding updatese.g., OF/routing protocol updatesXDynamicTimestamps not aligned!
??
Ambiguity between identical packets downstream
Trajectory sampling for direct traffic observation. Duffield et al., 2001
Packet rewriting compounds the ambiguity!Slide45
Approach 2: Collect at Every Hop45
Using
packet histories to troubleshoot networks.
Handigol
et al., 2014
Hash-based
IP
traceback
.
Snoeren
et al.,
2001
Packet-level telemetry in
large data-center networks. Zhu et al.,
2015Slide46
Approach 2: Collect at Every Hop46
Too expensive to collect
up front!
Using
packet histories to troubleshoot networks.
Handigol
et al., 2014
Hash-based
IP
traceback
.
Snoeren
et al.,
2001
Packet-level telemetry in
large data-center networks. Zhu et al.,
2015Slide47
Approach 2: Collect at Every Hop47
Too expensive to collect
up front!
Sampling to reduce overhead may mis
s the packets you care about…
Trajectory
sampling for direct traffic observation. Duffield et al.,
2001Slide48
Approach 3: Write Path into Packet48
sw
: S1,
s
rcip
: ___,
d
stip
: ___,
...
sw
: S2,
s
rcip
: ___,
d
stip
: ___;
s
w
: S1, ...;
sw
: S3, ...;
s
w
: S2, ...;
sw
: S1, ...;
Too much info on packet!
:-(
Switches have very accurate info on prior packet path
IP record route,
RFC 791.
Postel
, 1981
Tracing
packet trajectory in
data-center
networks.
Tammana
et al., 2015
Match-action HW can’t match regexes
:-(Slide49
Approach 3: Write Path into Packet49
sw
: S1,
s
rcip
: ___,
d
stip
: ___,
...
sw
: S2,
s
rcip
: ___,
d
stip
: ___;
s
w
: S1, ...;
sw
: S3, ...;
s
w
: S2, ...;
sw
: S1, ...;
As such, too much info on packet!
Switches have very accurate info on prior packet path
[8] IP record route
[9]
Tracing
packet trajectory in
data-center
networks.
Tammana
et al., 2015
How to reduce the amount of path information on packets?Slide50
Reducing Path Info on PacketsObservation 1: Queries already tell us what’s needed!Only record path info needed by queriesObservation 2: Queries are regular expressionsProcessing: Finite automaton (DFA)Distinguish only DFA states on the network!50Slide51
Reducing Path Info on PacketsObservation 1: Queries already tell us what’s needed!Only record path state needed by queriesObservation 2: Queries are regular expressionsRegular expressions Finite automaton (DFA)Distinguish only paths corresponding to DFA states51Record only DFA state on packets (1-2 bytes)Use existing “tag” fields (e.g., VLAN)Slide52
Downstream Query Compilation52QueriesForwardingData Plane RulesDownstream Query CompilerRegex Matching on Packet Trajectories!Slide53
Downstream Query Compilation53QueriesForwardingData Plane Rules(1) Get Query DFA(2) Get Transitioning & Accepting Policies
(3) Compose into Unified
P
olicySlide54
Downstream Query Compilation (1/3)p = (switch=S1 & srcip=10.0.0.1) ^ (switch=S2 & dstip=10.0.0.3)p.set_bucket(count_bucket())54Q0Q1Q2switch=S1 & srcip=10.0.0.1switch=S2 & dstip
=10.0.0.3
S1
S2Slide55
Downstream Query Compilation (2/3)55Q0Q1Q2switch=S1 & srcip=10.0.0.1switch=S2 & dstip=10.0.0.3DFA Transition
DFA Accept
s
tate=Q0 & switch=S1 &
srcip
=10.0.0.1
stateQ1
s
tate=Q1 & switch=S2 &
dstip
=10.0.0.3
stateQ2
s
tate=Q1 & switch=S2 &
dstip
=10.0.0.3
count
Generate “match-action-able” rulesSlide56
Downstream Query Compilation (3/3)56All acting on the same data plane packets!Use policy composition operators and compilerDFA-TransitioningForwardingDFA-Accepting
>>
+
)
(
Composing software-defined networks. Monsanto et al.,
2013Slide57
Downstream Query Compilation (3/3)57DFA-TransitioningForwardingDFA-Accepting>>+)(state=Q0 & switch=S1 & srcip=10.0.0.1 stateQ1state=Q1 & switch=S2 & dstip=10.0.0.3 stateQ2 dstip=10.0.0.1 fwd
(1)
dstip
=10.0.0.2
fwd
(2)
dstip=10.0.0.3
fwd
(3)
...
>>
state=Q0 &
switch=S1 &
srcip
=10.0.0.1 &
dstip
=10.0.0.2
state
Q1,
fwd
(2)
Composing software-defined networks. Monsanto et al.,
20
13Slide58
Downstream Query Compilation (3/3)58(DFA-Ingress-Transitioning >> Forwarding >> DFA-Egress-Transitioning) +(DFA-Ingress-Accepting) +(DFA-Ingress-Transitioning >> Forwarding >> DFA-Egress-Accepting)Slide59
Compiling the Full LanguageDFA construction with overlapping predicatesUpstream queriesGroupby atoms59Generate an ‘orthogonal basis’ for predicatesFormulate reachability test with Filter outputTraverse AST proactively, rewriting to “basic” atomsSlide60
Query Run-Time System: SummaryTranslation of Queries to Network MeasurementsAccurate measurementPackets carry trajectoriesEfficient encoding through DFA statesPay exactly for what you queryOnly collect packets matching queriesUse commodity hardwarePath regex matching on match-action tables!60Slide61
III. Optimizations61Slide62
Solution Approach62Query expressionsStatistics2. Query Run-Time SystemSDN controller
Payloads
Statistics
3. Optimizations
1. Path Query
L
anguageSlide63
Goal: Make Run-Time EfficientMetrics:Rule spaceQuery compile timePacket state spaceStanford network on a mix of queries (~550 statistics)Unoptimized: didn’t compile in 2 hoursFully optimized:Query compile time: ~ 5 secondsRule space: ~ 650 rules (TCAM capacity 2-4K)Packet state space: state fits in VLAN header63Fit in switch rule memory?Debugging “interactive”?Fit on typical “tag” headers?Slide64
Optimizations: Summary64Optimization# Rules?Time?# States?Separate query & forwarding actions into separate stagesOptimize conditional policy compilationIntegrate tagging and capture policiesPre-partition predicates by flow spaceCache predicate overlap decisionsDecompose query predicates into multiple stagesDetect predicate overlaps with Forwarding Decision DiagramsSlide65
Optimizations: Summary65Optimization# Rules?Time?# States?Separate query & forwarding actions into separate stagesOptimize conditional policy compilationIntegrate tagging and capture policiesPre-partition predicates by flow spaceCache predicate overlap decisionsDecompose query predicates into multiple stagesDetect predicate overlaps with FDDs
Cross-Product ExplosionSlide66
Cross-Product Explosion (1/2)66DFA-TransitioningForwardingDFA-Accepting>>+)(state=Q0 & srcip=10.0.0.2 stateQ1state=Q1 & srcip=10.0.0.3 stateQ2state=Q2 & port=4 stateQ4state=Q3 & srcmac=01:* stateQ5
state=Q4
& srcip
=10.0.0.1 stateQ3
state=Q5
& tpdstport=80
stateQ5
state=Q6
&
srcip
=10.0.0.3
state
Q6
state=Q7
&
dstip
=10.0.0.4
state
Q2
...
>>
dstip
=10.0.0.1
fwd
(1)
dstip
=10.0.0.2
fwd
(2)
dstip
=10.0.0.3
fwd
(3)
dstip
=10.0.0.4
fwd
(4)
dstip
=10.0.0.5
fwd
(5)
dstip
=10.0.0.6
fwd
(6)
dstip
=10.0.0.7
fwd
(7)
dstip
=10.0.0.8
fwd
(8)
...
state=Q0 &
srcip
=10.0.0.2 &
dstip
=10.0.0.1
state
Q1,
fwd
(1)Slide67
Cross-Product Explosion (2/2)Packets can satisfy both p1 and p101Also p1 and p102; p1 and p103; ...Query DFA edges (and rules) for each combination!67p1: srcip=10.0.0.1p2: srcip=10.0.0.2...p100: srcip=10.0.0.100p101: dstip=192.168.0.101p102: dstip=192.168.0.102...p200: dstip=192.168.0.200Slide68
Taming Cross-Product ExplosionKey Problem: Coerce multiple actions on overlapping sets of packets68Slide69
Taming Cross-Product ExplosionKey Problem: Coerce multiple actions on overlapping sets of packets69match1 action1match2 action2...Idea: Can we leverage
multiple passes on each packe
t in hardware?
Multi-stage match-action tables can already do this!Slide70
Taming Cross-Product ExplosionKey Idea: Leverage multiple passes for each packetOverlapping actions on different tables!70match1 action1match2 action2...match1 action1match2 action2...
...
Rule space O(M+N), not O(M*N)Slide71
Taming Huge Policies71(DFA-Ingress-Transitioning >> Forwarding >> DFA-Egress-Transitioning) +(DFA-Ingress-Accepting) +(DFA-Ingress-Transitioning >> Forwarding >> DFA-Egress-Accepting)(DFA-Ingress-Transitioning + DFA-Ingress-Accepting) >>Forwarding>>(DFA-Egress-Transitioning + DFA-Egress-Accepting)Slide72
Taming Huge Policies72in_transition+ in_acceptforwardingout_transition+ out
_accept
(DFA-Ingress-Transitioning
+
DFA-Ingress-Accepting)
>>
Forwarding
>>
(DFA-Egress-Transitioning
+
DFA-Egress-Accepting)
O(M*N)
O(M+N) Slide73
Taming Overlapping Query Predicates73p1: srcip=10.0.0.1p2: srcip=10.0.0.2...p100: srcip=10.0.0.100p101: dstip=192.168.0.101p102: dstip=192.168.0.102...p200: dstip=192.168.0.200in_transition+ in_accept
s
rcip
=10.0.0.1 state1q1
srcip
=10.0.0.2 state1q2
...
dstip
=192.168.0.101
state2q1
dstip
=192.168.0.102
state2q2
...
...
Running many parallel Query DFAs!
O(M*N)
O(M+N) Slide74
Taming Overlapping Query PredicatesApproach: “pack” queries into tables to minimize cost while respecting per-stage rule capacities and #stages74Total # stages# Rules in stage jSlide75
Taming Overlapping Query PredicatesCost function: upper bound on actual # rulesMove “dissimilar” predicates into different table stages75Concurrent NetCore: From policies to pipelines. Schlesinger et al., 2014Compiling packet programs to reconfigurable swithces. Jose et al., 2015Slide76
ImplementationPrototypePyretic SDN controllerNetKAT (Ocaml) compilerInstall rules on OpenVSwitchCurrently single-threadedIntel Xeon E3, 3.70Ghz 32GBImplementation publicly available onlinehttp://frenetic-lang.org/pyretic/76Composing software-defined networks. Monsanto et al., 2013A fast compiler for NetKAT. Smolka et al., 2015OpenVSwitch.orgSlide77
Benefit of OptimizationsStanford campus network topologySeveral queries:Traffic matrix, DDoS detection, per-hop packet loss, firewall evasion, slice isolation, congested linkMetrics and Stanford results (all queries together):Compile time: > 2 hours 5 seconds# Rules: ~ 650# State bytes: 2 bytes77Slide78
Benefit of Optimizations (Stanford)78Cumulative OptimizationTime (s)# Rules# State BitsNone> 7900DNFDNFSeparate query & forwarding actions into separate stages> 4920DNFDNFOptimize conditional policy compilation> 4080DNFDNFIntegrate tagging and capture policies2991259610Pre-partition predicates by flow space56.19184610Cache predicate overlap decisions
35.13
1846
10Decompose query predicates into m
ultiple stages5.467
260
16Slide79
Scalability TrendsFive synthetic ISP (Waxman) topologies at various network sizesAt each network size, run mix of queries from beforeAveraged metrics across queries & topologies79Slide80
I. Query Compile Time80Response time in man-computer conversational transactions. Miller, 1968Interactive problem solving (15s)Slide81
II. Rule Count81Switch TCAM capacity: 2K-4K rulesSlide82
III. Packet State Bits82VLANMPLSSlide83
SummaryDeclarative path query languageRegular expressions, groupingCapture locations, capture actionsCompositional run-time systemQuery-DFA packet stateKey optimizations for a practical systemAddressing cross-product explosionPaper and more info at http://www.cs.princeton.edu/~narayana/pathqueries 83Slide84
84Queries? Slide85
85