The madness beyond the gate Steve Loughran stevelhortonworkscom steveloughran 2015 Page 2 Me Before Kerberos Page 3 Me After Kerberos Page 4 HP Lovecraft Kerberos Evil lurking in New England ID: 380867
Download Presentation The PPT/PDF document "Hadoop and Kerberos:" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Hadoop and Kerberos: The madness beyond the gate
Steve Loughran
stevel@hortonworks.com
@steveloughran
2015Slide2
Page 2
Me: Before KerberosSlide3
Page 3
Me: After KerberosSlide4
Page 4HP LovecraftKerberos
Evil
lurking in New England
MIT Project Athena
Ancient, inhuman deities
Kerberos Domain Controller
Manuscripts to drive the reader
insane
IETF RFC 4120
Entities never spoken
of aloud
UserGroupInformation
Doomed explorers of darkness
YouSlide5
Leave now if you want to retain your life ofnaïve innocencePage 5Slide6
Page 6Slide7
export HADOOP_USER="root"Page 7Slide8
Modern Hadoop clusters are locked downthrough KerberosPage 8Slide9
Discover Kerberosbefore Kerberosdiscovers youPage 9Slide10
Kerberos:the gateway to hellPage 10Slide11
This is not a metaphorArt: Andrés
Álvarez
IglesiasSlide12
K
P
Kerberos
is the gateway
Page
12
Authentication
Service
Ticket
Granting Service
Principal
user@REALM
user/
hostname@REALM
(P, TGS, n
1
)
{
K
P.TGS
, n
1
}
KP
,
{ticket
(P,TGS)}
KTGS
Ticket(P, TGS) =
(
TGS, P,
t
start
,
t
end
, K
PT)
K
P
{
K
P.S
, n
2
}
KP
,
{ticket
(P,S)}
KS
{
auth
(P)
}
KP.TGS
,{
ticket
(P,TGS)
}
KTGS,
S,n
2
K
TGS
Kerberos Domain Controller
Client
auth
(P)
KP.TGS
= {
P, time)}
KP.TGSSlide13
Every service is a principalalice@REALMbob@REALMoozie
/
ooziehost@REALM
namenode
/nn1@REALM
hdfs
/_HOST@REALM
hdfs
/r04s12@REALM
hdfs
/r04s13@REALM
yarn
/_HOST@REALM
yarn/r04s12@REALM
HTTP
/_HOST@REALM
Page
13
short names
:
alice
bob
oozie
namenode
hdfs
yarn
HTTPSlide14
Page 14
Entering the darknessSlide15Slide16
Page 16Slide17
HDFS Bootstrap: Kerberos LoginPage 17
shared
keytab
in /
etc
/
hadoo
p
log in to
kerberos
datanode
/_HOST@REAL
M
tickets for TGS
namenode
/
nn@REALMSlide18
HDFS Bootstrap: DNs register with NN
Page
18
shared
keytab
in /
etc
/
hadoo
p
DN registration
Ticket
for
namenode
/
nn@REALM
ExportedBlockKeys
Request
t
icket
for
namenode
/
nn@REALM
namenode
/
nn@REALM
datanode
/_HOST@REAL
MSlide19
Hadoop TokensIssued and tracked by individual services(HDFS, WebHDFS, Timeline Server, YARN RM, …)Grant some form of access:Block tokens, Delegation TokensCan be passed on to other processesRenewable via service APIs (RPC, HTTP)Revocable in
server via service APIs
Page
19
read: O'Malley 2009,
Hadoop
Security
ArchitectureSlide20
HDFS IO: Block Tokens
Page
20
alice@REAL
M
Obtain
t
icket
for
namenode
/
nn@REALM
BlockToken
BlockToken
BlockToken
:
userId
, (
BlockPoolId
,
BlockId
),
keyId
,
expiryDate
, access-modes
namenode
/
nn@REAL
M
open("file")
Slide21
service/host@REALM
Delegation Tokens delegate access
Page
21
alice@REAL
M
BlockToken
HDFS Delegation Token
BlockToken
HDFS Delegation Token
HDFS Delegation Token
namenode
/
nn@
REAL
M
Token
Obtain
t
icket
for
namenode
/
nn@REALM
Request delegation token
Slide22
Launch Context
YARN app launch
Page
22
alice@REAL
M
HDFS Delegation Token
HDFS
resourcemanager
/
rm@REAL
M
nodemanager
/_HOST@REAL
M
alice
Launch Context
AM/RM
HDFS
AM/RM
HDFS
HDFS
HDFS
AM/RM
namenode
/
nn@
REALM
Obtain ticket for
resourcemanager
/
rm@REALM
Request delegation token
AM/RM
Token
Obtain
tickvet
for
namenode
/
nn@REALM
AM/RM'
AM/RM'
AM/RM'
Refresh AM/RM
Slide23
That which must not be named: UGIif(!UserGroupInformation.isSecurityEnabled()) { stayInALifeOfNaiveInnocence();} else { sufferTheEnternalPainOfKerberos
();
}
UserGroupInformation.checkTGTAndReloginFromKeytab
()
;
UserGroupInformation.getLoginUser
(
) // principal logged in as
UserGroupInformation.getCurrentUser
()
// principal acting as
Page
23Slide24
UGI.doAs()UserGroupInformation bob = UserGroupInformation.createProxyUser("bob",
UserGroupInformation.getLoginUser
());
FileSystem
userFS
=
bob.doAs
(
new
PrivilegedExceptionAction
<
FileSystem
>() {
public
FileSystem
run() throws Exception {
return
FileSystem.get
(
FileSystem.getDefaultUri
(),
conf
);
}
});
Page
24Slide25
Hadoop RPC@KerberosInfo(serverPrincipal = "my.kerberos.principal")public interface MyRpc extends VersionedProtocol { … }public class MyRpcPolicyProvider extends
PolicyProvider
{
public Service[]
getServices
() {
return new Service[] {
new Service("
my.protocol.acl
",
MyRpc.class
)
};
}
}
public class
MyRpcSecurityInfo
extends
SecurityInfo
{ …
}
META-INF/services/
org.apache.hadoop.security.SecurityInfo
org.example.rpc.MyRpcSecurityInfo
Page
25Slide26
IPC Server: get the current user identityMessages.KillResponse killContainer(Messages.KillRequest request) {UserGroupInformation callerUGI;try { callerUGI
=
UserGroupInformation.getCurrentUser
();
} catch (
IOException
ie
) {
LOG.info
("Error getting UGI ",
ie
);
AuditLogger.logFailure
("UNKNOWN", "Error getting UGI");
throw
RPCUtil.getRemoteException
(
ie
);
}
…
Page
26Slide27
IPC Server: AuthorizeString user = callerUGI.getShortUserName();if (!checkAccess(callerUGI, MODIFY)) { AuditLog.unauthorized(user, KILL_CONTAINER_REQUEST, "User doesn't have permissions to " + MODIFY); throw RPCUtil.getRemoteException( new
AccessControlException
(
+
user + " lacks access "
+
MODIFY_APP.name
(
)))
;
}
AuditLog.authorized
(user, KILL_CONTAINER_REQUEST)
Page
27Slide28
SASL: RFC4422Page 28Slide29
REST: SPNEGO (+ Delegation tokens) Page 29
Jersey +
java.net
httpclient
?
“
if lucky it'll
work”
HADOOP-11825:
Move timeline client
Jersey+Kerberos+UGI
support into a public
implementation Slide30
TestingPage 30Slide31
Error messages to fear
Art: Andrés
Álvarez
Iglesias
Failure
unspecified at GSS-API level (Checksum failed
)
No
valid credentials provided (Failed to find any Kerberos
tgt
)
Server not found in Kerberos database
Clock skew too great
Principal not found
No valid credentials provided (Illegal key size
)Slide32
Topics Avoided Not CoveredZookeeperJAASTrying to use HTTPS in a YARN applicationTrying to use Full REST in a YARN applicationSystem properties to debug Kerberos & SPNEGOGroup managementHADOOP_PROXY_USER
Page
32Slide33
gitbook.com/@steveloughran
Questions?
Art: Andrés
Álvarez
IglesiasSlide34
ZookeeperSASL to negotiate security: System.setProperty("zookeeper.sasl.client", "true"
)
;
Permissions are not transitive down the tree
Page
34
List<ACL>
perms
= new
ArrayList
<>
();
if (
UserGroupInformation.isSecurityEnabled
()) {
perms(
new ACL(
ZooDefs.Perms.ALL
,
ZooDefs.Ids.AUTH_IDS
));
perms.add
(new ACL(
ZooDefs.Perms.READ
,ZooDefs.Ids.ANYONE_ID_UNSAFE
));
} else {
perms
.add
(new ACL(
ZooDefs.Perms.ALL
,
ZooDefs.Ids.ANYONE_ID_UNSAFE
));
}
zk.createPath
(path
,
null, perms
,
CreateMode.PERSISTENT
);Slide35
System Properties for debugging-Dsun.security.krb5.debug=true-Dsun.security.spnego.debug=trueexport HADOOP_JAAS_DEBUG=truePage 35Slide36
ServicesRPC authentication via annotations & metadata in JARYARN Web UIs: rely on RM proxy for authenticationAuthentication != AuthorizationAdd audit logs on service endpointsYARN services: come up with a token refresh strategy:keytab everywhere; keytab in AM; update from client
Page
36Slide37
JAASJava Authentication and Authorization ServiceCore Kerberos classes and types (Principal)Text files to configureDifferent for different JVMsNeed to double escape \ for windows pathsUGI handles setting up a JAAS context & logging inPage
37Slide38
GlossarySimple Authentication and Security Layer (SASL)GSSAPI Generic Security Service Application Program Interface (RFC-2743+ others)JAAS: Java Authentication and Authorization ServiceSimple and Protected GSSAPI Negotiation Mechanism (SPNEGO)Page 38