Snort Virtual Network Function with DPI Service - PowerPoint Presentation

Slide1

Snort Virtual Network Function with DPI Service

Asher Gruber | January 2017

This work was carried out under the supervision of

Prof.

Anat

Bremler

-Bar

and

Mr.

Yotam

HarholSlide2

Agenda

IntroductionDPI as a Service paper

Project Goals

Background

Deep Packet InspectionSnort OverviewService ChainingNetwork Service Header (NSH)ImplementationNSH SupportDPI Service EnhancementSnort EnhancementExperimental ResultsConclusions

Snort VNF with DPI Service

2Slide3

Introduction

Snort VNF with DPI Service

3Slide4

DPI as a Service

ProblemDeep Packet Inspection (DPI) is a widespread functionality among middlebox applications

In

the common architecture

, each packet is inspected from scratch by multiple middleboxes on its path until reaching its final destinationFor numerous middleboxes the IDS task is the most time-consuming and can take most of the processing timeSnort VNF with DPI Service

4Slide5

DPI as a Service (Cont.)

SolutionExtract DPI out of the middleboxesProvide DPI as a service to the

middleboxes

Service inspects the

packets once, against the rules of all the middleboxesService reports the resultsMiddleboxes consume the results, while avoiding the highly expensive pattern search phase, and focus on performing more complex processing based on to the pattern match resultsSnort VNF with DPI Service

5Slide6

DPI as a Service (Cont.)

Solution AdvantagesPerformance – each packet is scanned onceShared network services create room for innovation

Advanced DPI functionality

Consolidated DPI allows to focus more on functionality

SecurityFocus on securing a single DPI implementationDeploy more instances across the network to mitigate attacksReduced cost of middleboxesSnort VNF with DPI Service

6Slide7

DPI as a Service (Cont.)

The paper outlines a framework for deploying the service and provide a reference implementation on a simulation environment

The DPI as a Service framework

implementation supports the deployment of

multiple DPI services across the network which are all controlled by a centralized DPI ControllerThe controller is responsible for managing the overall DPI process such as middlebox registration, pattern set management, and DPI service initializationSnort VNF with DPI Service

7Slide8

Project Goals

Integrate the DPI as a Service framework with the complex "real world" Snort NIDSIntegrate the DPI Service with Snort using the Network Service Header (NSH) protocol

The integrations will allow to evaluate if the suggested framework can operate in a more realistic environment setup

Snort VNF with DPI Service

8Slide9

Background

Snort VNF with DPI Service

9Slide10

Deep Packet Inspection (DPI)

Inspection of packet payloads to identify predefined sets of patternsPatternsStrings matchingRegular expressions matching

String matching is a core component in most DPI engines

Used for pre-filtering

Constitutes most of the work performed by the engineNetwork Intrusion Detection Systems (NIDS) perform DPI to detect malicious content in packet payloads routed through the networkSnort VNF with DPI Service10Slide11

Snort overview

Open source NIDSWidely adopted by the security communityUsed by numerous enterprises

Modes

Sniffer

Packet LoggerNIDSSnort VNF with DPI Service11Slide12

Packet Processing Flow

Snort VNF with DPI Service

12

Network packets processed by Snort follow a similar flowSlide13

Packet Acquisition

Performed via Data AcQuisition library (DAQ)Snort supports packet capturing

Network interface

Input file

Snort VNF with DPI Service13Slide14

Packet Decoding

Packet are decoded according to network protocol stackSnort VNF with DPI Service

14

DecodeEthPkt

(Ethernet)

DecodeIP

(IPv4)

DecodeUDP

(UDP)

DecodeTCP

(TCP)

DecodeICMP

(ICMP)

PacketSlide15

Packet Preprocessing

Preprocessors perform variety of operationPacket checks and alertingPacket data modification

Normalization

Reassembly

DecompressionMore …Snort VNF with DPI Service15Slide16

Detection

Detection engine is focused around identifying predefined sets of attack patternsSnort rules allow to define NIDS policies

DPI options

Content – match content in packet payload

PCRE – search packet payload for RegEx patternSnort VNF with DPI Service16Slide17

Detection (Cont.)

Rule EngineCommon Snort setup includes vast number of rulesThe rule engine performance is extremely important

Rule are grouped by protocol, direction and port

Groups are stored in special data structures

Each rule group has a dedicated MPSE instanceMulti-Pattern Search EnginePattern matching on real-time network trafficInterface for pattern matching algorithmsWrapper of the Snort implementation of the Aho-Corasick (AC) algorithm

Snort VNF with DPI Service

17Slide18

Detection (Cont.)

Aho-Corasick (AC)String matching algorithm which is commonly used by NIDSMatches multiple strings simultaneously

Constructs a DFA from the pattern

set (defined in rules)

Accepting state = pattern matchO(n) search time; n = payload sizeImplementation is critical, since it has a significant impact on the overall performance of SnortSnort VNF with DPI Service

18Slide19

Snort Output

Packets are matched during detection and queuedActions (log, alert) are fetched from the event queueActions are filtered (rate, threshold) and triggered

Snort VNF with DPI Service

19Slide20

Snort Configuration

Snort configuration are defined in snort.confRule operations (e.g. path to rule file)Plug-ins activation

Much more…

Distribution contains sensible defaults

Usually customized for specific environmentSnort VNF with DPI Service20Slide21

Service Chaining

Modern way for deploying and delivering composite servicesLogical group of Service Functions (SF) (Firewall, DPI, etc.) a packet needs to go through

Elastic, simple and modern service deployment model by treating SF as resources which can be scheduled and

consumed

Linked based off policySnort VNF with DPI Service21

NAT

FW

DPI

FW

LB

Web

Classifier

Classifier

Classifier

WebSlide22

Service Chaining (Cont.)

Services are assembled using Software Defined Network (SDN)Service Path is required for implementationPackets are "steered" to the next SF in the path

With metadata (the killer app)

Using Network overlay (

VxLAN, GRE, MPLS)Possible today, but suffers from tight coupling to the transport layerSnort VNF with DPI Service22

NAT

FW

DPI

FW

LB

Web

Classifier

Classifier

Classifier

WebSlide23

Network Service Header (NSH)

Data-plan protocol which defines a service planeIETF adopted protocol (NSH

)

Two main components:

path info and metadataTopology IndependenceDe-coupling of service topology and the actual network topologyService forwarding is within the service planeEnables Service ChainingHas path ID needed to realize service pathProvides ability to monitor and troubleshoot service chain end-to-endTransport Agnostic – appropriate transport to encapsulate traffic

NSH-aware control plane is required (e.g. OpenDaylight)

Snort VNF with DPI Service

23Slide24

Snort VNF with DPI Service

24

SDN Controller platform

1

4

5

7

SFF

SFF

Service Classifier

SF

1

6

Classification Policy

Packet

3

VXLAN: D-IP

NSH: SPI ID 10, SI: 2

Packet

Packet

2

SF

2

SF

3

SF

4

VXLAN: D-IP

NSH: SPI ID 10, SI: 1

Packet

NSH: SPI ID 10, SI: 2

Packet

Metadata

Packet

1

NSH: SPI ID 10, SI: 1

MetadataSlide25

Implementation

Snort VNF with DPI Service

25Slide26

Implementation

DPI Service was enhanced to support the NSH protocol and can now pass the pattern match results with the inspected packetSnort was enhanced in order to apply its rules without the need to re-scan the packets from scratch, while leveraging the match results reported by the DPI Service

Snort was furthermore enhanced in order to allows it's registration to the DPI controller

Snort VNF with DPI Service

26Slide27

NSH Support

DPI Service match results are reported using NSHNSH supports metadata exchange along the service pathThe metadata is used to transfer the pattern match results

Snort VNF with DPI Service

27Slide28

NSH Support (Cont.)

We use VxLAN-gpe as the encapsulation protocolVxLAN is encapsulated as specified in the RFCUDP (port = 4789)

IP

Ethernet

Snort VNF with DPI Service28Slide29

DPI Service Enhancements

Original Implementation used dedicated packets to report pattern match resultsDPI Service was extended to report pattern match results within the inspected packet using the NSH protocolReporting result using dedicated packets is still supported

Packets without pattern matches are forwarded as received

Snort VNF with DPI Service

29Slide30

DPI Service (Cont.)

Snort VNF with DPI Service

30

DPI Service

DPI Controller

1

Packet

2

Packet

3

Initialize

Packet Scanned

Results Aggregated

IP

UDP

VxLAN

Payload

Eth

NSH

Network Services Header

NSH base header

Service path (24 bit) / Index

Optional Metadata

Eth

IP

4

Port 4790

Packet

5Slide31

Snort Enhancements

Snort configurations were added to support the DPI Service functionality (e.g. On\Off, settings, etc.)Snort was extended to communicate with the DPI ControllerRegistration of Snort instance

Registration of instance rule patterns

Snort was modified to leverage the DPI Service pattern match results

NSH Support (decode)VxLAN Support (decode)Bypass the Snort DPI (decode, detect)More…Snort VNF with DPI Service31Slide32

Snort DPI Service Configuration

DPI Service functionality can be controlled via the Snort configuration file (i.e. snort.conf)Parsing logic was added to the Snort initialization phase

Snort VNF with DPI Service

32Slide33

Snort Controller Communication

Snort registration to the DPI Controller was addedJSON messageMiddlebox’s ID (i.e. Snort ID)Rule patterns required for DPI

Allows Controller to register Snort to DPI Service

Allows DPI Service to report matches to Snort

Message is sent according to snort.config Snort VNF with DPI Service33Slide34

Snort Controller Communication (Cont.)

Message is constructed during Snort initialization

Snort VNF with DPI Service

34

prmUdpRTNX

prmIpRTNX

prmIcmpRTNX

prmTcpRTNX

prmSrcPort

[]

prmDstPort

[]

prmGeneric

PORT_RULE_MAP

. . .

. . .

. . .

. . .

. . .

SnortConf

88, 8080

9000

53

PORT_GROUP (Content)

MPSE

MPSE

MPSESlide35

Snort Controller Communication (Cont.)

Rule patterns are fetched from the AC DFA of the various rule groupsThe patterns are taken from the DFA accepting states

Fetching

rules via the DFA is essential

We want the DPI Service to search only for patterns that are searched by the DFAWe want to bypass the AC DFA execution by Snort and use the rule match resultsWe later use the same code base to associate rules to their DFA accepting stateSnort VNF with DPI Service

35Slide36

Snort – DPI Service Integration

The goal is to prevent Snort from re-scanning the packets using the MPSE (AC DFA) which has a significate impact on the overall performanceEnabling Snort to use the DPI Service pattern match results required multiple medications to the code base

Snort Initialization

Snort Packet Processing

DecodingDetectionSnort still needs to have the ability to perform other DPI operations using the MPSE and AC DFA, since finding a matched pattern using the AC DFA does not guarantee a rule matchSnort VNF with DPI Service36Slide37

Snort Initialization

To take advantage of the DPI Service results, while skipping the AC DFA search, a mapping between a Snort rule and its associated AC DFA accepting state is requiredThe mapping will allow Snort to use the accepting state in order to perform additional DPI rule operations which are required for every matched rule that is reported by the DPI Service

The creation of the mapping is performed during the Snort initialization phase while constructing the registration massaged, since it involves traversing the same data structures

Snort VNF with DPI Service

37Slide38

Snort Initialization (Cont.)

The registration message is built while visiting the AC DFA accepting states associated to each of the rule groups (one AC DFA per rule group)DFA accepting states represents a pattern of at least one or more rules

Obtaining an accepting state provides access to the associated rules and patterns

During the registration message construction, whenever a rule-pattern pair is added to the message the rule-accepting state pair is added to the mapping

Every rule group has a dedicated AC DFA, therefore a rule-accepting state map will be created per rule group (or rule group AC DFA)Snort VNF with DPI Service38Slide39

Snort Initialization (Cont.)

Snort VNF with DPI Service

39

The Rule Group => (rid => acc. state) mapping is essential, since it is possible that the DPI Service will report a rule match within a packet that does not meet the packets rule group

Having a rule-accepting state map which is shared amount all rule groups will potentially cause false positive rule match alerts

Rule Group DFA

DFA

DFA

DFA

DFA

rid

acc. state

56

C1

71

C7

34

C9

rid

acc. state

9

C

3

12

C

5

86

C

14Slide40

Snort Packet Processing - Decoding

Traditionally when a packet is captured by Snort it is decoded according to it’s network protocol stackWe have extended Snort to support the NSH protocolSupporting NSH required supporting VxLAN (UDP, port = 4789)

The support for the protocols was added to the UDP decoding stack

Once

the NSH Base Header is decoded we extract the pattern match results from the metadataSnort VNF with DPI Service40Slide41

Snort VNF with DPI Service

41

DecodeEthPkt

(Ethernet)

DecodeIP

(IPv4)

DecodeUDP

(UDP)

DecodeTCP

(TCP)

DecodeICMP

(ICMP)

PacketSlide42

Snort VNF with DPI Service

42

DecodeEthPkt

(Ethernet)

DecodeIP

(IPv4)

DecodeUDP

(UDP)

DecodeTCP

(TCP)

DecodeICMP

(ICMP)

Packet

DecodeVxLAN

(

VxLAN

)

DecodeNSH

(NSH)Slide43

Snort Packet Processing – Decoding (Cont.)

The decoded match reports are added to the packet structureThe list of match reports is used to bypass the AC DFA search during the packet detection phaseWhen the decoding of the NSH is completed we continue to decode the original inner packet which follows the standard Snort decoding stack

Snort VNF with DPI Service

43Slide44

Snort Packet Processing – Detection

No packet re-scanIf DPI Service active go to alternative funcBypass AC DFA

Once func completed packet returns to standard processing

Snort VNF with DPI Service

44Slide45

Snort Packet Processing – Detection (Cont.)

Snort VNF with DPI Service

45Slide46

Snort Packet Processing – Detection (Cont.)

The Match operation is also called in the standard Snort analysis functionality whenever an accepting state is reached in the AC DFA searchBy calling the Match operation we assure that all the DPI content options which were not included in the DPI Service search are met before the rule is considered as matched

Once the Match operation concludes that all the options of a given accepting state are met it registrars a match event to the queue.

The

analysis functionality ends when all the rule match results are evaluated. Then the packet continues through the standard Snort packet processing flow to the output phaseSnort VNF with DPI Service46Slide47

Experimental Results

Snort VNF with DPI Service

47Slide48

Experimental Results

We analyze the integration of Snort with the DPI Service by repeating a subset of the original paper experimentsWe compare the results, to those of the original paperRepeating the original experiments will allow us to evaluated if the promising results presented in the paper, can be reproduced in a more realistic environment setup

Snort VNF with DPI Service

48Slide49

Experimental Results - Environment

MachineIntel Xeon E3-1270 v3 CPUQuad-Core, quad-core, each core having two hardware threads

Cache: 32 KB L

1

(per core), 256 KB L2(per core), 8 MB L3 (shared)Linux Ubuntu 14.04 LTS (Trusty)SetupInput traffic - 148MB HTTP trace crawled from most popular websitesRulesSnort - 3498 original Snort rulesDPI ServiceRules sent by Snort to the ControllerExact match patterns of length 4 or moreExperiments were conducted using the DPI Service and Snort

Snort VNF with DPI Service

49Slide50

Pipelined Middlebox Scenario

Snort VNF with DPI Service

50

Without a DPI Service

With a DPI Service

Latency traditional: 32.73

µs/p

Latency DPI Service: 27.57 µs/p

At least 38% faster

16% improvementSlide51

Pipelined Middlebox Scenario (Cont.)

Snort VNF with DPI Service

51

Comparing our results to the original paper

38

% faster Throughput

86

% faster Throughput

16%

Latency improvement

36% Latency

improvementSlide52

Virtual DPI Performance

Original paper - middlebox DPI app is much faster then the virtual DPI instancesWith Snort – middlebox is the bottleneck from both latency and throughput in relation to the DPI Service

Snort VNF with DPI Service

52Slide53

Analysis of Reporting Results with NSH

Snort VNF with DPI Service

53

Average NSH size is 189 bytes

Median NSH size is 156 bytesSlide54

Conclusions

We have integrated the original DPI as a Service framework with SnortWe have extended the DPI Service to support NSHThe two enhancements allow to re-evaluate the framework by performing a subset of the original experiments

The experiments show the original promising results can be reproduced in a more realistic environment setup

Snort VNF with DPI Service

54Slide55

Snort VNF with DPI Service

55Slide56

Analysis of Reporting Results with NSH

Snort VNF with DPI Service

56

Average NSH size is 189 bytes

Median NSH size is 156 bytes