incomplete JAAS, ZK, diagnostics checklists. A new error message!

steveloughran · steveloughran · commit 89c5c0d6a886 · 2015-12-15T21:10:01.000Z
diff --git a/SUMMARY.md b/SUMMARY.md
@@ -6,16 +6,17 @@
 * [HDFS and Kerberos](sections/hdfs.md)
 * [UGI](sections/ugi.md)
 * [Java and JDK Versions](sections/jdk_versions.md)
+* [JAAS](sections/jaas.md)
+* [Keytabs](sections/keytabs.md)
 * [Hadoop IPC Security](sections/ipc.md)
 * [Web and REST](sections/web_and_rest.md)
 * [YARN and YARN Applications](sections/yarn.md)
 * [Zookeeper](sections/zookeeper.md)
-* [JJAS](sections/jaas.md)
 * [Testing](sections/testing.md)
 * [Low-Level Secrets](sections/secrets.md)
 * [Error Messages to Fear](sections/errors.md)
 * [The Limits of Hadoop Security](sections/the_limits_of_hadoop_security.md)
 * [Checklists](sections/checklists.md)
 * [Glossary](sections/glossary.md)
 * [Bibliography](sections/biblography.md)
-* [Acknowledgements](sections/acknowledgements.md)
+* [Acknowledgements](sections/acknowledgements.md)
diff --git a/sections/checklists.md b/sections/checklists.md
@@ -101,3 +101,17 @@
 
 [ ] Code invoking Jersey Client reacts to 401/403 exception responses when using Authentication Token by deleting creating a new Auth Token and re-issuing request. (this triggers re-authentication)
 
+### Debugging Workflow
+
+[ ] host has an IP address (`ifconfig` / `ipconfig`)
+[ ] host has an FQDN: `hostname -f`
+[ ] FQDN resolves to hostname `nslookup $hostname`
+[ ] hostname responds to pings `ping $hostname`
+[ ] reverse DNS lookup of IPAddr returns hostname
+[ ] clock is in sync with rest of cluster: `date`
+
+[ ] keytab exists
+[ ] keytab is readable by account running service.
+[ ] keytab contains principals in listing `ktlist -kt $keytab`
+[ ] keytab FQDN is in entry of form `shortname/$FQDN`
+
diff --git a/sections/errors.md b/sections/errors.md
@@ -107,6 +107,13 @@ In `krb5.conf`:
     [libdefaults]
       udp_preference_limit = 1
 
+##  `GSSException: No valid credentials provided (Mechanism level: Connection reset)'
+
+We've seen this triggered in Hadoop tests after the MiniKDC through an exception; it's thread
+exited and hence the Kerberos client got a connection error.
+
+When you see this assume network connectivity problems, or something up at the KDC itself.
+
 ## Principal not found
 
 The hostname is wrong (or there is >1 hostname listed with different IP addrs) and so a principal
@@ -124,6 +131,68 @@ This apparently surfaces in [Java 8 after 8u40](http://sourceforge.net/p/spnego/
 if Kerberos server doesn't support the first authentication mechanism which the client
 offers, then the client fails. Workaround: don't use those versions of Java.
 
+This is [now acknowledged by Oracle](http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8080129) and
+has been fixed in 8u60.
+
+
+## `Specified version of key is not available (44)`
+
+```
+Client failed to SASL authenticate: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Specified version of key is not available (44))]
+```
+
+The meaning of this message —or how to fix it— is a mystery to all.
+
+There is [some tentative coverage in Stack Overflow](http://stackoverflow.com/questions/24511812/krbexception-specified-version-of-key-is-not-available-44)
+
+One possibility is that the keys in your keytab have expired. Did you know that can happen? It does.
+One day your cluster works happily. The next your client requests are failing, with this message
+surfacing in the logs.
+
+```
+ klist -kt zk.service.keytab 
+Keytab name: FILE:zk.service.keytab
+KVNO Timestamp         Principal
+---- ----------------- --------------------------------------------------------
+   5 12/16/14 11:46:05 zookeeper/devix.cotham.uk@COTHAM
+   5 12/16/14 11:46:05 zookeeper/devix.cotham.uk@COTHAM
+   5 12/16/14 11:46:05 zookeeper/devix.cotham.uk@COTHAM
+   5 12/16/14 11:46:05 zookeeper/devix.cotham.uk@COTHAM
+```
+
+
+## `javax.security.auth.login.LoginException: No password provided`
+
+When this surfaces in a server log, it means the server couldn't log in as the user. That is,
+there isn't an entry in the supplied keytab for that user.
+ 
+Some of the possible causes
+
+* The wrong keytab was specified
+* There isn't an entry in the keytab for the user
+* The hostname of the machine doesn't match that of a user in the keytab, so a match of `service/host`
+fails.
+
+Ideally, services list the keytab and username at fault here. In a less than ideal world —that is
+the one we live in— things are less helpful
+
+Here, for example, is a Zookeeper trace, saying it is the user `null` that is at fault.
+
+```
+2015-12-15 17:16:23,517 - WARN  [main:SaslServerCallbackHandler@105] - No password found for user: null
+2015-12-15 17:16:23,536 - ERROR [main:ZooKeeperServerMain@63] - Unexpected exception, exiting abnormally
+java.io.IOException: Could not configure server because SASL configuration did not allow the  ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: No password provided
+        at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207)
+        at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)
+        at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:111)
+        at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
+        at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
+        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
+        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
+
+```
+
+
 # Hadoop Web/REST APIs
 
 ## AuthenticationToken ignored
diff --git a/sections/ipc.md b/sections/ipc.md
@@ -142,3 +142,32 @@ something in the service, if not the calls is rejected.
 Note how failures are logged to an audit log; successful operations should be logged too.
 The purpose of the audit log is determine the actions of a principal —both successful
 and unsuccessful.
+
+### Downgrading to unauthed IPC
+
+IPC can be set up on the client to fall back to unauthenticated IPC if it can't negotiate
+a kerberized connection. While convenient, this opens up some security vulnerabilitie -hence
+the feature is generally disabled on secure clusters. It can/should be enabled when needed
+
+```
+-D ipc.client.fallback-to-simple-auth-allowed=true
+```
+
+As an example, this is the option on the command line for DistCp to copy from a secure cluster
+to an insecure cluster, the destination only supporting simple authentication.
+
+```
+hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true hdfs://secure:8020/lovecraft/books hdfs://insecure:8020/lovecraft/books
+```
+
+Although you can set it in a core-site.xml, this is dangerous from a security perpective
+
+```
+<property>
+  <name>ipc.client.fallback-to-simple-auth-allowed</name>
+  <value>true</value> 
+</property>
+```
+
+*warning* it's tempting to turn this on during development, as it makes problems go away. As it is
+not recommended in production: avoid except on the CLI during attempts to debug problems.
diff --git a/sections/jaas.md b/sections/jaas.md
@@ -20,14 +20,43 @@ JAAS is a nightmare from the Enterprise Java Bean era, one which surfaces from t
 
 JAAS provides for a standard configuration file format for specifying a *login context*; how code trying to run in a specific context/role should login and authenticate.
 
-As a single jaas.conf file can have multiple contexts, the same file can be used to configure the server and clients of a service, each with different binding information. Different contexts can have different login/auth mechanisms, including Kerberos and LDAP, so that you can even specify different auth mechanisms for different roles.
+As a single `jaas.conf` file can have multiple contexts, the same file can be used to configure the server and clients of a service, each with different binding information. Different contexts can have different login/auth mechanisms, including Kerberos and LDAP, so that you can even specify different auth mechanisms for different roles.
 
-In Hadoop, the JAAS context is invariably Kerberos when it comes to talking to HDFS, YARN, etc. However, if Zookeeper enters the mix, it may be interacted with differently —and so need a different JAAS context.
+In Hadoop, the JAAS context is invariably Kerberos when it comes to talking to HDFS, YARN, etc.
+However, if Zookeeper enters the mix, it may be interacted with differently —and so need a different JAAS context.
 
 Fun facts about JAAS
 
 1. Nobody ever mentions it, but the file takes backslashed-escapes like a Java string.
-1. It needs escaped backlash directory separators on Windows, such as: `C:\\security\\krb5.conf`. Get that wrong and your code will fail with what will inevitably be an unintuitive message.
-1. Each context must declare the authentication module to use. The kerberos authentication model on IBM JVMs is different from that on Oracle and OpenJDK JVMs. You need to know the target JVM for the context —or create separate contexts for the different JVMs.
-
-Hadoop's UGI class will dynamically create a JAAS context for Hadoop logins, dynamically determining the name of the kerberos module to use. For interacting purely with HDFS and YARN, you may be able to avoid needing to know about or understand JAAS.
+1. It needs escaped backlash directory separators on Windows, such as: `C:\\security\\krb5.conf`.
+   Get that wrong and your code will fail with what will inevitably be an unintuitive message.
+1. Each context must declare the authentication module to use.
+   The kerberos authentication model on IBM JVMs is different from that on Oracle and OpenJDK JVMs.
+   You need to know the target JVM for the context —or create separate contexts for the different JVMs.
+1. The rules about when to use `=` within an entry, and when to complete an entry with a `;` appear to be:
+start with the login module, one key=value line per entry, quote strings, finish with a `;`
+within the same file.
+
+Hadoop's UGI class will dynamically create a JAAS context for Hadoop logins, dynamically determining the name of the kerberos module to use. For interacting purely with HDFS and YARN, you may be able to avoid needing to know about or understand JAAS.
+
+Example of a JAAS file valid for Sun
+
+If you need a basic JAAS cient configuration which 
+```
+Client {
+  com.sun.security.auth.module.Krb5LoginModule required
+  useKeyTab=false
+  useTicketCache=true
+  doNotPrompt=true;
+};
+```
+
+
+# Setting a JAAS Config file for a Java process
+
+```
+-Djava.security.auth.login.config=/path/to/server/jaas/file.conf
+```
+
+In Hadoop applications, this has to be set in whichever environment variable is picked up
+by the command which your are invoking.
diff --git a/sections/keytabs.md b/sections/keytabs.md
@@ -0,0 +1,65 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+  
+   http://www.apache.org/licenses/LICENSE-2.0
+  
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Keytabs
+
+Keytabs are critical for secure Hadoop clusters, as they allow the services to be launched
+without prompts for passwords
+
+
+## Creating a Keytab
+
+If your management tools sets up keytabs for you: use it.
+
+```
+kadmin.local
+
+ktadd -k zk.service.keytab -norandkey zookeeper/devix@COTHAM 
+ktadd -k zk.service.keytab -norandkey zookeeper/devix.cotham.uk@COTHAM
+exit
+```
+
+and of course, make it accessible
+
+```
+chgrp hadoop zk.service.keytab
+chown zookeeper zk.service.keytab
+```
+
+check that the user can login
+
+```
+# sudo -u zookeeper klist -e -kt zk.service.keytab
+# sudo -u zookeeper kinit -kt zk.service.keytab zookeeper/devix.cotham.uk
+# sudo -u zookeeper klist
+```
+
+### Keytab Expiry
+
+Keytabs expire
+
+That is: entries in them have a limited lifespan (default: 1 year)
+
+This is actually a feature —it limits how long a lost/stolen keytab can have access to the system.
+
+At the same time, it's a major inconvenience as (a) the keytabs expire and (b) it's never
+immediately obvious why your cluster has stopped working.
+
+### Keytab security
+
+Keytabs are sensitive items. They need to be treated as having all the access to the data of that principal
+
+### Keytabs and YARN applications
+
+
diff --git a/sections/zookeeper.md b/sections/zookeeper.md
