tokens, HDFS

Author: steveloughran

sections/errors.md

@@ -21,7 +21,8 @@
 
 # OS/JVM Layer; GSS library
 
-Some of these are covered in Oracle's Troubleshooting Kerberos docs. This section just highlights some of the common causes, other causes that Oracle don't mention —and messages they haven't covered.
+Some of these are covered in Oracle's Troubleshooting Kerberos docs.
+This section just highlights some of the common causes, other causes that Oracle don't mention —and messages they haven't covered.
 
 ## Server not found in Kerberos database (7) 
 
@@ -30,7 +31,8 @@ Some of these are covered in Oracle's Troubleshooting Kerberos docs. This sectio
 
 ## No valid credentials provided (Mechanism level: Illegal key size)]
 
-Your JVM doesn't have the extended cryptography package and can't talk to the KDC. Switch to openjdk or go to your JVM supplier (Oracle, IBM) and download the JCE extension package, and install it in the hosts where you want Kerberos to work.
+Your JVM doesn't have the extended cryptography package and can't talk to the KDC.
+Switch to openjdk or go to your JVM supplier (Oracle, IBM) and download the JCE extension package, and install it in the hosts where you want Kerberos to work.
 
 ## No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt
 
@@ -41,14 +43,22 @@ This may appear in a stack trace starting with something like:
 Possible causes:
 
 1. You aren't logged in via `kinit`.
-2. You did specify a keytab but it isn't there or is somehow otherwise invalid
-3. You don't have the Java Cryptography Extensions installed.
+1. You have logged in with `kinit`, but the tickets you were issued with have expired.
+1. You did specify a keytab but it isn't there or is somehow otherwise invalid
+1. You don't have the Java Cryptography Extensions installed.
 
 ## Clock skew too great
 
 		GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null)) . . . Caused by: javax.security.auth.login.LoginException: Clock skew too great
 
-This comes from the clocks on the machines being too far out of sync. This can surface if you are doing Hadoop work on some VMs and have been suspending and resuming them; they've lost track of when they are. Reboot them.
+    GSSException: No valid credentials provided (Mechanism level: Clock skew too great (37) - PROCESS_TGS
+
+    kinit: krb5_get_init_creds: time skew (343) larger than max (300)
+
+This comes from the clocks on the machines being too far out of sync. 
+
+This can surface if you are doing Hadoop work on some VMs and have been suspending and resuming them;
+they've lost track of when they are. Reboot them.
 If it's a physical cluster, make sure that your NTP daemons are pointing at the same NTP server, one that is actually reachable from the Hadoop cluster. And that the timezone settings of all the hosts are consistent.
 
 ## KDC has no support for encryption type
@@ -62,14 +72,14 @@ to prove to the KDC that the caller has the password. If the password is wrong,
 an error about checksums.
 1. Kerberos is very strict about hostnames and DNS; this can somehow trigger the problem.
 [http://stackoverflow.com/questions/12229658/java-spnego-unwanted-spn-canonicalization](http://stackoverflow.com/questions/12229658/java-spnego-unwanted-spn-canonicalization); 
-1. Java 8 behaves differently from Java 6 & 7 here which can cause problems
+1. Java 8 behaves differently from Java 6 and 7 here which can cause problems
 [(HADOOP-11628](https://issues.apache.org/jira/browse/HADOOP-11628).
 
 
 ## Principal not found
 
 The hostname is wrong (or there is >1 hostname listed with different IP addrs) and so a principal
-of the form USER/HOST@DOMAIN is coming back with the wrong host, and the KDC doesn't find it.
+of the form `USER/HOST@DOMAIN` is coming back with the wrong host, and the KDC doesn't find it.
 
 See the comments above about DNS for some more possibilities.
 
@@ -89,7 +99,7 @@ offers, then the client fails. Workaround: don't use those versions of Java.
 
 This has been seen in the HTTP logs of Hadoop REST/Web UIs:
 
-	2015-06-26 13:49:02,239 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
+	WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: AuthenticationToken ignored: org.apache.hadoop.security.authentication.util.SignerException: Invalid signature
 
 This means that the caller did not have the credentials to talk to a Kerberos-secured channel.
 

sections/hadoop_tokens.md

@@ -64,7 +64,7 @@ public class BlockTokenIdentifier extends TokenIdentifier {
 
 Alongside the fields covering the block and permissions, that `cache` data contains 
 
- ## Tickets vs Tokens
+## Kerberos Tickets vs Hadoop Tokens
 
 
 | Token                | Function                   |
@@ -119,13 +119,7 @@ Alongside the fields covering the block and permissions, that `cache` data conta
  a thread in the background.
 
 
- 
- 
- 
- ## Token Propagation in YARN Applications
- 
- 
- 
+
 
  Imagine a user deploying a YARN application in a cluster, one which needs
  access to the user's data stored in HDFS. The user would be required to be authenticated with
@@ -173,4 +167,88 @@ return info about the provider. One class `AnnotatedSecurityInfo`, examines the
 on the class to determine these values, including looking in the Hadoop configuration
 to determine the kerberos principal declared for that service (see [IPC](ipc.html) for specifics).
 
-## Delegation Token internals
+## Implementation Details
+
+What is inside a Hadoop Token? Whatever the
+service provider wishes to supply. 
+
+A token is treated as a byte array to be passed
+in communications, such as when setting up an IPC
+connection, or as a data to include on an HTTP header
+while negotiating with a remote REST endpoint.
+
+The code on the server which issues tokens,
+the `SecretManager` is free to fill its byte arrays with
+structures of its choice. Sometimes serialized java objects
+are used; more recent code, such as that in YARN, serializes
+data as a protobuf structure and provides that in the  byte array
+(example, `NMTokenIdentifier`).
+
+
+### `Token`
+
+The abstract class `org.apache.hadoop.yarn.api.records.Token` is
+used to represent a token in Java code; it contains
+
+| field | type | role |
+|-------|------|------|
+| identifier | `ByteBuffer` | the service-specific data within a token |
+| password | `ByteBuffer` | a password
+| tokenKind | `String` | token kind for looking up tokens. Example 
+
+
+### `SecretManager`
+
+Every server which issues tokens must implement and run
+a `org.apache.hadoop.security.token.SecretManager` subclass.
+ 
+### `DelegationKey`
+
+This contains a "secret" (generated by the `javax.crypto` libraries), adding serialization
+and equality checks. Because of this the keys can be persisted (as HDFS does) or sent
+over a secure channel. Uses crop up in YARN's `ZKRMStateStore`, the MapReduce History server 
+and the YARN Application Timeline Service.
+ 
+### How tokens are issued
+ 
+TODO: how a connection bootstraps from Kerberos auth to Tokens
+
+### How tokens are refreshed
+
+TODO
+
+### How delegation tokens are shared
+
+DTs can be serialized; that is done when issued/renewed.
+
+When making requests over Hadoop RPC, you don't need to include the DT, simply
+include the Hash to indicate that you have it
+
+### Delegation Tokens
+ 
+### Token Propagation in YARN Applications
+ 
+YARN applications depend on delegation tokens to gain access to cluster
+resources and data on behalf of the principal. It is the task of
+the client-side launcher code to collect the tokens needed, and pass them
+to the launch context used to launch the Application Master..
+
+
+## What does this mean for my application?
+
+If you are writing an application, what does this mean?
+
+You need to worry about tokens in servers if
+
+1. You want to support secure connections without requiring Kerberos
+authentication at the rate of the maximum life of a kerberos ticket.
+1. You want to allow applications to delegate authority, such
+as to YARN applications, or other services. (Example, filesystem delegation tokens
+provided to a Hive thrift server could be used to access the filesystem
+as that user).
+1. You want a consistent client/server authentication and identification 
+mechanism across secure and insecure clusters. This is exactly what YARN does:
+a token is issued by the YARN Resource Manager to an application instance's
+Application Manager at launch time; this is used in all communications from
+the AM to the RM. Using tokens *always* means there is no separate codepath
+between insecure and secure clusters.

sections/hdfs.md

@@ -1,4 +1,18 @@
-# HDFS and Kerberos
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+  
+   http://www.apache.org/licenses/LICENSE-2.0
+  
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# HDFS
 
 > It seemed to be a sort of monster, or symbol representing a monster, of a form which only a diseased fancy could conceive. If I say that my somewhat extravagant imagination yielded simultaneous pictures of an octopus, a dragon, and a human caricature, I shall not be unfaithful to the spirit of the thing. A pulpy, tentacled head surmounted a grotesque and scaly body with rudimentary wings; but it was the general outline of the whole which made it most shockingly frightful.
 > *[The Call of Cthulhu](https://en.wikisource.org/wiki/The_Call_of_Cthulhu), HP Lovecraft, 1926.*
@@ -29,19 +43,47 @@ the HDFS team from implementing user-specific priority/throttling of HDFS data a
 and allow multi-tenant Hadoop clusters to prioritise high-SLA applications over lower-priority
 code.
 
-## HDFS Namenode
+## HDFS NameNode
 
-### TODO
 
-1. Namenode reads in a keytab and initializes itself from there (i.e. no need to `kinit`; ticket
+1. NN reads in a keytab and initializes itself from there (i.e. no need to `kinit`; ticket
 renewal handed by `UGI`).
-1. In a secure cluster, Web HDFS requires SPNEGO
-1. If web auth is enabled in a secure cluster, both the DN web UI will requires SPNEGO
-1. In a secure cluster, if webauth is disabled, kerberos/SPNEGO auth may still be needed
-to access the HDFS browser. This is a point of contention: its implicit from the delegation
- to WebHDFS --but a change across Hadoop versions, as before an unauthed user could still browse
- as "dr who". 
+1. Generates a *Secret*
+
+Delegation tokens in the NN are persisted to the edit log, the operations `OP_GET_DELEGATION_TOKEN`
+`OP_RENEW_DELEGATION_TOKEN` and `OP_CANCEL_DELEGATION_TOKEN` covering the actions. This ensures
+that on failover, the tokens are still valid
+
+
+### Block Keys
+
+A `BlockKey` is the secret used to show that the caller has been granted access to a block
+in a DN. 
+
+The NN issues the block key to a client, which then asks a DN for that block, supplying
+the key as proof of authorization.
+
+Block Keys are managed in the `BlockTokenSecretManager`, one in the NN
+and another in every DN to track the block keys to which it has access. 
+It is the DNs which issue block keys as blocks are created; when they heartbeat to the NN
+they include the keys.
+
+### Block Tokens
+
+A `BlockToken` is the token issued for access to a block; it includes 
+
+    (userId, (BlockPoolId, BlockId), keyId, expiryDate, access-modes)
 
+The block key itself isn't included, just the key to the referenced block. The access modes declare
+what access rights the caller has to the data
+
+    public enum AccessMode {
+      READ, WRITE, COPY, REPLACE
+    }
+
+It is the NN which has the permissions/ACLs on each file —DNs don't have access to that data.
+Thus it is the BlockToken which passes this information to the DN, by way of the client.
+Obviously, they need to be tamper-proof.
 
 
 ## DataNodes
@@ -50,24 +92,41 @@ DataNodes do not use Hadoop RPC —they transfer data over HTTP. This delivers b
 though the (historical) use of Jetty introduced other problems. At scale, obscure race conditions
 in Jetty surfaced. Hadoop now uses Netty for its DN block protocol.
 
+### DataNodes and SASL
+
 Pre-2.6, all that could be done to secure the DN was to bring it up on a secure (&lt;1024) port
 and so demonstrate that an OS superuser started the process. Hadoop 2.6 supports SASL
 authenticated HTTP connections, which works *provided all clients are all running Hadoop 2.6+*
 
-
 See [Secure DataNode](http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SecureMode.html#Secure_DataNode)
 
-### TODO
 
 ## HDFS Client interaction
 
-1. Client asks NN for access to a path, identifying via KST or DT.
-1. NN authenticates caller, if access to path is authorized, returns BT to the client.
-1. Client talks to 1+ DNs with the block, using the BT.
-1. DN authenticates BT using shared-secret with NN.
-1. if authenticated, DN compares permissions in BT with operation requested, grants or rejects it.
+1. Client asks NN for access to a path, identifying via Kerberos or delegation token.
+1. NN authenticates caller, if access to path is authorized, returns Block Token to the client.
+1. Client talks to 1+ DNs with the block, using the Block Token.
+1. DN authenticates Block Token using shared-secret with NameNode.
+1. if authenticated, DN compares permissions in Block Token with operation requested, then
+grants or rejects the request.
 
 The client does not have its identity checked by the DNs. That is done by the NN. This means
-that the client can in theory pass a BT on to another process for delegated access to a single
+that the client can in theory pass a Block Token on to another process for delegated access to a single
 file. It has another implication: DNs can't do IO throttling on a per-user basis, as they do
 not know the user requesting data.
+
+### NN/WebHDFS
+
+1. In a secure cluster, Web HDFS requires SPNEGO
+1. After authenticating with a SPNEGO-negotiated mechanism, webhdfs sends an HTTP redirect,
+including the BlockTocken in the redirect
+
+### NN/Web UI
+
+1. In a secure cluster, Web HDFS requires SPNEGO
+1. If web auth is enabled in a secure cluster, the DN web UI will requires SPNEGO
+1. In a secure cluster, if webauth is disabled, kerberos/SPNEGO auth may still be needed
+to access the HDFS browser. This is a point of contention: its implicit from the delegation
+ to WebHDFS --but a change across Hadoop versions, as before an unauthed user could still browse
+ as "dr who". 
+ 

sections/ugi.md

@@ -111,7 +111,7 @@ This returns the logged in user
 
     UserGroupInformation user = UserGroupInformation.getLoginUser();
 
-If there is no current user --that is, the login process hasn't started yet,
+If there is no logged user --that is, the login process hasn't started yet,
 this triggers the login and the starting of the background refresh thread.
 
 This makes it a point where the security kicks in: all configuration resources
@@ -139,6 +139,29 @@ when a service performs an action on the user's behalf
 
 ### `doAs()`
 
+
+This method is at the core of UGI. A call to `doAs()` executes the inner code
+*as the user*. In secure, that means using the Kerberos tickets and Hadoop delegation
+tokens belonging to them.
+
+Example: loading a filesystem as a user
+
+    FileSystem systemFS = FileSystem.get(FileSystem.getDefaultUri(), conf);
+    UserGroupInformation proxyUser = UserGroupInformation.createProxyUser(
+        user, UserGroupInformation.getLoginUser());
+    FileSystem userFS = proxyUser.doAs(new PrivilegedExceptionAction<FileSystem>() {
+      @Override
+      public FileSystem run() throws Exception {
+        return FileSystem.get(systemFS.getUri(), systemFS.getConf());
+      }
+    });
+
+Here the variable userFS contains a client of the Hadoop Filesystem with
+the home directory and access rights of the user `user`. If the user identity
+had come in via an RPC call, they'd
+
+
+
 ## Environment variable-managed UGI Initialization
 
 There are some environment variables which configure UGI.

sections/what_is_kerberos.md

@@ -205,10 +205,12 @@ To look at and work with keytabs, the `ktutil` command line program is the tool
 
 ## Tickets
 
-Kerberos is built around the notion of tickets.
+Kerberos is built around the notion of *tickets*.
 
-A ticket is something which may be passed to a service to indicate that the caller
-has the permissions contained within the ticket —for the duration of the ticket's lifetime.
+A ticket is something which can be passed to a server to identify that the caller
+and to provide a secret key that can be used between the client an the server 
+—for the duration of the ticket's lifetime. It is all that a server needs to
+authenticate a client: there's no need for the server to talk to the KDC.
 
 What's important is that tickets can be passed on: an authenticated principal
 can obtain a ticket to a service, and pass that on to another process in the distributed
@@ -218,7 +220,7 @@ using that ticket. That recipient only has the permissions granted to the ticket
 also provided), and those permissions are only valid for as long as the ticket
 is valid.
 
-Limited-lifetime tickets ensure that even if a ticket is captured by a malicious
+The limited lifetime iftickets ensure that even if a ticket is captured by a malicious
 attacker, they can only make use of the credential for the lifetime of the ticket.
 The ops team doesn't need to worry about lost/stolen tickets, to have a process for
 revoking them, as they expire within a short time period, usually a couple of days.