infrastructure Ira Weiny John Fleck OFADevWorkshop SA interaction difficulties SA MAD formats RMPP libibumad quirkiness Application shortcuts Hard coded PR data I ID: 497935
Download Presentation The PPT/PDF document "Scalable name and address resolution" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Scalable name and address resolution infrastructure-- Ira Weiny/John Fleck
#OFADevWorkshopSlide2
SA interaction difficultiesSA MAD formats, RMPP, libibumad “quirkiness”Application shortcutsHard coded PR dataIgnoring parts of queried PR dataOnly work on a limited set of clusters or cluster typesDirect access to libibumad and the SA are vectors for security breaches
March 30 – April 2, 2014 #OFADevWorkshop
2Slide3
Current stack is not scalableNodes access the same SA services multiple times from ibacm, kernel, libibumad…PR queriesNotice/multicast registrationsName resolution through standard DNS requires an ARP from IP to GIDMarch 30 – April 2, 2014 #OFADevWorkshop
3Slide4
ibacm name resolutionRelies on IPoIB (DNS, ARP, etc)Names map to <GID, Pkey> “end point”User’s often don’t care about the partition they are running on.“cross” partition names can’t be resolvedLocal apps need knowledge of a common partition prior to name resolution.Some work done in this area via ibacm_hosts.data
Current name resolution requires source “end point” to be
specified
March 30 – April 2, 2014 #OFADevWorkshop
4Slide5
ibacm as a SA proxyibacm provides a good starting point for addressing some of these concerns…March 30 – April 2, 2014 #OFADevWorkshop5Slide6
GoalsProvide controlled and consistent access to user space name and PR resolution services (AKA SA access)SA access controlAccuracyEase of usePortabilityEnable all consumers to access ibssaProvide caching and other ibacm services to kernel usersMarch 30 – April 2, 2014 #OFADevWorkshop
6Slide7
ibacm enhancementsApplications query ibacm as a local “SA proxy”All SA interactions done through ibacmAdditional name services providedibacm can control access to SA and libibumadibacm is backed by “providers”ibssa
Current features as default
Enhancements for name services are planned
March 30 – April 2, 2014 #OFADevWorkshop
7Slide8
ibacm enhancementsName resolution services“DNS” for direct name resolutionName to PR (or GID, <GID, PR>, IP, <IP, PR>)ibacm provides service to the kernelUses netlinkLeverages the same infrastructure for all users
March 30 – April 2, 2014 #OFADevWorkshop
8Slide9
Arch9March 30 – April 2, 2014 #OFADevWorkshopSlide10
Architecture (non-eye chart)March 30 – April 2, 2014 #OFADevWorkshop10Slide11
Implementation plansSeparate out ibacm into “core” and default providerCore handlesProvider loading and assignment to ports/End pointsSteering client requests to correct providerPort/device EventsNetlink requests and eventsAdministration like config file parsing, log file, etcDefault provider handlesSame functionality as current resolve functions
March 30 – April 2, 2014 #OFADevWorkshop
11Slide12
Initial data flow detailsMarch 30 – April 2, 2014 #OFADevWorkshop12Slide13
Provider API’sPrototype code being workedCollaborating with OFI WG and rdmacmsubmission to the list imminent“prov” branch in ibacm’s git treeThe API will evolve, collaborating with ibssaMain API calls will includePath Record resolutionName to GID mapping helper
March 30 – April 2, 2014 #OFADevWorkshop
13Slide14
Expand *_getaddrinfoUse ibacm first to resolve a Name prior to calling getaddrinfo (DNS)Call can provide Path Record hints through the normal “hints” parameterFor example Service ID, Pkey etc.Need both librdmacm and ibacm changesOnly single local end point can be supported nowFuture local end point resolution can be determined by GID returned from provider name -> GID map
March 30 – April 2, 2014 #OFADevWorkshop
14Slide15
Kernel ibacm accessSRP, IPoIB, and rdma_cm kernel modules use ib_sa to query for Path RecordsExtend ibacm PR resolution/caching to kernel modulesUse netlink messages to communicate with ibacmExpand existing RDMA netlink interface
Currently connecting with
ib_sa
using
ibacm messages
Exploring the use of
ib_mad
and using MAD
formatted messages
March 30 – April 2, 2014 #OFADevWorkshop
15Slide16
OverviewMarch 30 – April 2, 2014 #OFADevWorkshop16Slide17
A little more detail…March 30 – April 2, 2014 #OFADevWorkshop17Slide18
Current Netlink / ibacm Message FormatMarch 30 – April 2, 2014 #OFADevWorkshop18Slide19
Kernel still uses SA when ibacm not availableMarch 30 – April 2, 2014 #OFADevWorkshop19Slide20
Future workSA event registration and reportingNoticeMulticastIP to GID mappingIPoIB netlink to ibacm?arpd extentions?March 30 – April 2, 2014 #OFADevWorkshop
20Slide21
Thank youHal RosenstockKaike WanSean HeftyMarch 30 – April 2, 2014 #OFADevWorkshop21Slide22
Thank YouSlide23
Current SA interactionsApplicationsDirect SALibibumadUD QPLibrdmacmIbacmDns/arpKernelDirect SA access onlyMarch 30 – April 2, 2014 #OFADevWorkshop
23Slide24
Name service requirementsGeneric interface to request remote node by name through “DNS like” resolutionMapping provided by providers based on cluster configuration, node configuration, and/or provider/SA communication.March 30 – April 2, 2014 #OFADevWorkshop24Slide25
librdmacm exampleNew librdmacm example app$ resolve_name -husage: resolve_name <name> [-h] [-s <service id>] Specify a service ID in PR 'hints'March 30 – April 2, 2014 #OFADevWorkshop
25Slide26
librdmacm example$ resolve_name priv03ai_family 0ai_route : 0x1ff15a0Path information service_id: 0x0 dgid: fe80::11:7500:79:1815
sgid
: fe80::11:7500:79:1763
dlid
: 2
slid: 1
…
$
resolve_name
google.com
ai_family
2
dest
(null)
173.194.33.133
March 30 – April 2, 2014 #OFADevWorkshop
26