book in sync with apachecon talk, october 2015

Author: steveloughran

sections/biblography.md

@@ -13,7 +13,7 @@
 1. [JAAS Configuration (Java 8)](http://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html)
 1. For OS/X users, the GUI ticket viewer is `/System/Library/CoreServices/Ticket\ Viewer.app`
 1. [Colouris01], Colouris, Dollimore & Kindberg, 2001, *Distributed System Concepts and Design*,
- 
+1. [JAva 8 GSS API](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/jgss-features.html)
 
 ### Kerberos, Active Directory and Apache Hadoop
 

sections/checklists.md

@@ -58,6 +58,8 @@
 
 [ ] Container Credentials are retrieved in AM and containers.
 
+[ ] Delegation tokens revoked during (managed) teardown.
+
 ## YARN Web UIs and REST endpoints
 
 [ ] Primary Web server: `AmFilterInitializer` used to redirect requests to the RM Proxy.

sections/errors.md

@@ -19,6 +19,22 @@
 > *[Supernatural Horror in Literature](https://en.wikisource.org/wiki/Supernatural_Horror_in_Literature), HP Lovecraft, 1927.*
 
 
+Security error messages appear to take pride in providing limited information. In particular,
+they are usually some generic `IOException` wrapping a generic security exception. There is some
+text in the message, but it is often `Failure unspecified at GSS-API level`, which means
+"something went wrong".
+
+Generally a stack trace with UGI in it is a security problem, *though it can be a network problem
+surfacing in the security code*.
+
+The underlying causes of problems are usually the standard ones of distributed systems: networking
+and configuration.
+
+
+In [HADOOP-12426](https://issues.apache.org/jira/browse/HADOOP-12426) I've proposed a CLI entry point
+for health checking this. Volunteers to implement welcome.
+
+
 # OS/JVM Layer; GSS library
 
 Some of these are covered in Oracle's Troubleshooting Kerberos docs.
@@ -27,7 +43,8 @@ This section just highlights some of the common causes, other causes that Oracle
 ## Server not found in Kerberos database (7) 
 
 * DNS is a mess and your machine does not know its own name.
-* Your machine has a hostname, but it's not one there's an entry in the keytab for
+* Your machine has a hostname, but the service principal is a `/_HOST` wildcard and the hostname
+is not one there's an entry in the keytab for.
 
 ## No valid credentials provided (Mechanism level: Illegal key size)]
 
@@ -59,6 +76,7 @@ This comes from the clocks on the machines being too far out of sync.
 
 This can surface if you are doing Hadoop work on some VMs and have been suspending and resuming them;
 they've lost track of when they are. Reboot them.
+
 If it's a physical cluster, make sure that your NTP daemons are pointing at the same NTP server, one that is actually reachable from the Hadoop cluster. And that the timezone settings of all the hosts are consistent.
 
 ## KDC has no support for encryption type
@@ -79,7 +97,7 @@ an error about checksums.
 ## Principal not found
 
 The hostname is wrong (or there is >1 hostname listed with different IP addrs) and so a principal
-of the form `USER/HOST@DOMAIN` is coming back with the wrong host, and the KDC doesn't find it.
+of the form `user/_HOST@REALM` is coming back with the wrong host, and the KDC doesn't find it.
 
 See the comments above about DNS for some more possibilities.
 

sections/hadoop_and_kerberos.md

@@ -11,7 +11,7 @@
   See the License for the specific language governing permissions and
   limitations under the License. See accompanying LICENSE file.
 -->
-  
+
 # Hadoop's support for Kerberos
 
 Hadoop can use Kerberos to authenticate users, and processes running within a
@@ -31,3 +31,13 @@ to interact with a Hadoop cluster and applications running in it *do need to kno
 
 This is what this book attempts to cover.
 
+## Why do they inflict so much pain on us?
+
+Before going in there, here's a recurring question: why? Why Kerberos and not, say some
+SSL-certificate like system? Or OAuth?
+
+Kerberos was written to support centrally managed accounts in a local area network, one in
+which adminstrators manage individual accounts. This is actually much simpler to manage than
+PKI-certificate based systems: look at the effort it takes to revoke a certificate in a browser.
+
+OAuth? 

sections/hadoop_tokens.md

@@ -62,7 +62,8 @@ public class BlockTokenIdentifier extends TokenIdentifier {
   ...
 ```
 
-Alongside the fields covering the block and permissions, that `cache` data contains 
+Alongside the fields covering the block and permissions, that `cache` data contains the token
+identifier
 
 ## Kerberos Tickets vs Hadoop Tokens
 
@@ -107,6 +108,9 @@ Alongside the fields covering the block and permissions, that `cache` data conta
 
  1. The tokens must be renewed before they expire: once expired, a token is worthless.
  1. Token renewers can be implemented as a Hadoop RPC service, or by other means, *including HTTP*.
+ 1. Token renewal may simply be the updating of an expiry time in the server, without pushing
+ out new tokens to the clients. This scales well when there are many processes across
+ the cluster associated with a single application..
 
  For the HDFS Client protocol, the client protocol itself is the token renewer. A client may
  talk to the Namenode using its current token, and request a new one, so refreshing it.
@@ -138,7 +142,7 @@ in some form of storage shared across the failover services.
 The benefit: there's no need to involve the KDC in authenticating requests, yet short-lived access
 can be granted to applications running in the cluster. This explicitly avoid the problem of having
 1000+ containers in a YARN application each trying to talk to the KDC. (Issue: surely tickets
-offer that feature?)
+offer that feature?).
 
 ## Example
 
@@ -178,20 +182,61 @@ offer that feature?)
  is currently considered valid (based on the expiry time and the clock value of the Name Node)
 
 
-## Determining the Kerberos Principal for a service
 
-1. Service name is derived from the URI (see `SecurityUtil.buildDTServiceName`)...different
-services on the same host have different service names
-1. Every service has a protocol (usually defined by the RPC protocol API)
-1. To find a token for a service, client enumerates all `SecurityInfo` instances; these
-return info about the provider. One class `AnnotatedSecurityInfo`, examines the annotations
-on the class to determine these values, including looking in the Hadoop configuration
-to determine the kerberos principal declared for that service (see [IPC](ipc.html) for specifics).
+
+## What does this mean for my application?
+
+If you are writing an application, what does this mean?
+
+You need to worry about tokens in servers if:
+
+1. You want to support secure connections without requiring Kerberos
+authentication at the rate of the maximum life of a kerberos ticket.
+1. You want to allow applications to delegate authority, such
+as to YARN applications, or other services. (Example, filesystem delegation tokens
+provided to a Hive thrift server could be used to access the filesystem
+as that user).
+1. You want a consistent client/server authentication and identification 
+mechanism across secure and insecure clusters. This is exactly what YARN does:
+a token is issued by the YARN Resource Manager to an application instance's
+Application Manager at launch time; this is used in all communications from
+the AM to the RM. Using tokens *always* means there is no separate codepath
+between insecure and secure clusters.
+
+You need to worry about tokens in client applications if you wish
+to interact with Hadoop services. If the client is required to run
+on a kerberos-authenticated account (e.g. kinit or keytab), then
+your main concern is simply making sure the principal is logged in.
+
+If your application wishes to run code in the cluster using the YARN scheduler, you need to
+directly worry about Hadoop tokens. You will need to request delegation tokens
+from every service with which your application will interact, include them in the YARN
+launch information —and propagate them from your Application Master to all
+containers the application launches.
+
+## Design
+
+(from Owen's design document)
+
+Namenode Token
+
+    TokenID = {ownerID, renewerID, issueDate, maxDate, sequenceNumber}
+    TokenAuthenticator = HMAC-SHA1(masterKey, TokenID)
+    Delegation Token = {TokenID, TokenAuthenticator}
+
+The token ID is used in messages from the client to identify the client; service can
+rebuild the `TokenAuthenticator` from it; this is the secret used for DIGEST-MD5 signing
+of requests.
+
+
+Token renewal: caller asks service provider for a token to be renewed. The server updates
+the expiry date in its local table to `min(maxDate, now()+renew_period)`. A non-HA NN
+can use these renewal requests to actually rebuild its token table —provided the master
+key has been persisted.
 
 ## Implementation Details
 
-What is inside a Hadoop Token? Whatever the
-service provider wishes to supply. 
+What is inside a Hadoop Token? Whatever the service provider wishes to supply. 
 
 A token is treated as a byte array to be passed
 in communications, such as when setting up an IPC
@@ -215,7 +260,7 @@ used to represent a token in Java code; it contains
 |-------|------|------|
 | identifier | `ByteBuffer` | the service-specific data within a token |
 | password | `ByteBuffer` | a password
-| tokenKind | `String` | token kind for looking up tokens. Example 
+| tokenKind | `String` | token kind for looking up tokens.  
 
 
 ### `SecretManager`
@@ -229,10 +274,26 @@ This contains a "secret" (generated by the `javax.crypto` libraries), adding ser
 and equality checks. Because of this the keys can be persisted (as HDFS does) or sent
 over a secure channel. Uses crop up in YARN's `ZKRMStateStore`, the MapReduce History server 
 and the YARN Application Timeline Service.
+
+
+ 
 
 ### How tokens are issued
+
+ 
+A first step is determining the Kerberos Principal for a service:
 
-TODO: how a connection bootstraps from Kerberos auth to Tokens
+ 1. Service name is derived from the URI (see `SecurityUtil.buildDTServiceName`)...different
+ services on the same host have different service names
+ 1. Every service has a protocol (usually defined by the RPC protocol API)
+ 1. To find a token for a service, client enumerates all `SecurityInfo` instances; these
+ return info about the provider. One class `AnnotatedSecurityInfo`, examines the annotations
+ on the class to determine these values, including looking in the Hadoop configuration
+ to determine the kerberos principal declared for that service (see [IPC](ipc.html) for specifics).
+
+
+With a Kerberos principal, 
+
 
 ### How tokens are refreshed
 
@@ -255,21 +316,18 @@ the client-side launcher code to collect the tokens needed, and pass them
 to the launch context used to launch the Application Master..
 
 
-## What does this mean for my application?
+### Proxy Users
 
-If you are writing an application, what does this mean?
+Proxy users are a feature which was included in the Hadoop security model for services
+such as Oozie; a service which needs to be able to execute work on behalf of a user 
 
-You need to worry about tokens in servers if
+Because the time at which Oozie would execute future work cannot be determined, delegation
+tokens cannot be used to authenticate requests issued by Oozie on behalf of a user.
+Kerberos keytabs are a possible solution here, but it would require every user submitting
+work to Oozie to have a keytab and to pass it to Oozie.
 
-1. You want to support secure connections without requiring Kerberos
-authentication at the rate of the maximum life of a kerberos ticket.
-1. You want to allow applications to delegate authority, such
-as to YARN applications, or other services. (Example, filesystem delegation tokens
-provided to a Hive thrift server could be used to access the filesystem
-as that user).
-1. You want a consistent client/server authentication and identification 
-mechanism across secure and insecure clusters. This is exactly what YARN does:
-a token is issued by the YARN Resource Manager to an application instance's
-Application Manager at launch time; this is used in all communications from
-the AM to the RM. Using tokens *always* means there is no separate codepath
-between insecure and secure clusters.
+
+
+## Weaknesses
+
+1. Any compromised DN can create block tokens.

sections/ipc.md

@@ -39,6 +39,16 @@ This is "fiddly". It's not impossible, it just involves effort.
 
 In its favour: it's a lot easier than SPNEGO.
 
+### Annotating a service interface
+
+```
+@KerberosInfo(serverPrincipal = "my.kerberos.principal")
+public interface MyRpc extends VersionedProtocol {
+  long versionID = 0x01;
+...
+}
+```
+
 ### `SecurityInfo` subclass
 
 Every exported RPC service will need its own extension of the `SecurityInfo` class, to provide two things:
@@ -48,16 +58,40 @@ Every exported RPC service will need its own extension of the `SecurityInfo` cla
 
 ### `PolicyProvider` subclass
 
-A `PolicyProvider` subclass. This is used to inform the RPC infrastructure of the ACL policy: who may talk to the service. It must be explicitly passed to the RPC server
 
-		rpcService.getServer()
-		  .refreshServiceAcl(serviceConf, new MyRPCPolicyProvider());
+```
+public class MyRpcPolicyProvider extends PolicyProvider {
+
+  public Service[] getServices() {
+    return  new Service[] {
+     new Service("my.protocol.acl", MyRpc.class)
+    };
+  }
+
+}
+
+```
+
+ This is used to inform the RPC infrastructure of the ACL policy: who may talk to the service. It must be explicitly passed to the RPC server
+
+```
+rpcService.getServer() .refreshServiceAcl(serviceConf, new MyRpcPolicyProvider());
+```
+
+In practise, the ACL list is usually configured with a list of groups, rather than a user.
+
+### `SecurityInfo` class 
+
+```
+public class MyRpcSecurityInfo extends SecurityInfo { ... }
+
+```
 
 ### `SecurityInfo` resource file
 
 The resource file `META-INF/services/org.apache.hadoop.security.SecurityInfo` lists all RPC APIs which have a matching SecurityInfo subclass in that JAR.
 
-		org.apache.example.appmaster.rpc.RPCSecurityInfo
+		org.example.rpc.MyRpcSecurityInfo
 
 The RPC framework will read this file and build up the security information for the APIs (server side? Client side? both?)
 
@@ -70,32 +104,37 @@ the server can determine the identity of the principal.
 
 This is something it can ask for when handling the RPC Call:
 
-      UserGroupInformation callerUGI;
-      
-      // #1: get the current user identity
-      try {
-        callerUGI = UserGroupInformation.getCurrentUser();
-      } catch (IOException ie) {
-        LOG.info("Error getting UGI ", ie);
-        AuditLogger.logFailure("UNKNOWN", "Error getting UGI");
-        throw RPCUtil.getRemoteException(ie);
-      }
+```
+UserGroupInformation callerUGI;
+
+// #1: get the current user identity
+try {
+  callerUGI = UserGroupInformation.getCurrentUser();
+} catch (IOException ie) {
+  LOG.info("Error getting UGI ", ie);
+  AuditLogger.logFailure("UNKNOWN", "Error getting UGI");
+  throw RPCUtil.getRemoteException(ie);
+}
+```
 
 The `callerUGI` variable is now set to the identity of the caller. If the caller
 has delegated authority (tickets, tokens) then they still authenticate as
 that principal they were acting as (possibly via a `doAs()` call).
 
-    
-      // #2 verify their permissions
-      if (!checkAccess(callerUGI, ApplicationAccessType.MODIFY)) {
-        AuditLogger.logFailure(callerUGI.getShortUserName(),
-            AuditConstants.KILL_CONTAINER_REQUEST,
-            "User doesn't have permissions to " + ApplicationAccessType.MODIFY
-            AuditConstants.UNAUTHORIZED_USER);
-        throw RPCUtil.getRemoteException(new AccessControlException("User "
-            + callerUGI.getShortUserName() + " cannot perform operation "
-            + ApplicationAccessType.MODIFY_APP.name());
-      }
+
+```
+// #2 verify their permissions
+String user = callerUGI.getShortUserName();
+if (!checkAccess(callerUGI, MODIFY)) {
+  AuditLog.unauthorized(user,
+    KILL_CONTAINER_REQUEST,
+    "User doesn't have permissions to " + MODIFY);
+  throw RPCUtil.getRemoteException(new AccessControlException(
+    + user + " lacks access "
+    + MODIFY_APP.name()));
+}
+AuditLog.authorized(user, KILL_CONTAINER_REQUEST)
+```
 
 In ths example, there's a check to see if the caller can make a request which modifies
 something in the service, if not the calls is rejected.