more error messages, initial terrors.md

Author: steveloughran

README.md

@@ -36,10 +36,17 @@ or the YARN scheduler starting to summon pre-human deities.
 
 ## Implementation notes.
 
-This is a work in progress book designed to built using the [gitbook tool chain](https://github.com/GitbookIO/gitbook),
-the best OSS implementation to date of the book-as-software process proposed in
-[Refactoring the Publishing Process](http://people.apache.org/~stevel/papers/refactoring_publishing.pdf),
-Loughran and Hatcher, 2002.
+1. This is a work in progress book designed to built using the [gitbook tool chain](https://github.com/GitbookIO/gitbook).
 
-It is hosted on [github](https://github.com/steveloughran/kerberos_and_hadoop)
+1. It is hosted on [github](https://github.com/steveloughran/kerberos_and_hadoop).
+
+1. All the content is Apache licensed.
+
+1. This is not a formal support channel for Hadoop + Kerberos problems. If you have a support
+contract with [Hortonworks](http://hortonworks.com/) then issues related to Kerberos may 
+eventually reach the author. Otherwise: try 
+
+      - [Hortonworks Answerhub](https://community.hortonworks.com/answers/index.html)
+      - The users mailing list of Apache Hadoop, the application and you are using on top of it
+      - [Stack Overflow](http://stackoverflow.com/search?q=hadoop+kerberos).
 

SUMMARY.md

@@ -15,6 +15,7 @@
 * [Testing](sections/testing.md)
 * [Low-Level Secrets](sections/secrets.md)
 * [Error Messages to Fear](sections/errors.md)
+* [Tales of Terror](sections/terrors.md)
 * [The Limits of Hadoop Security](sections/the_limits_of_hadoop_security.md)
 * [Checklists](sections/checklists.md)
 * [Glossary](sections/glossary.md)

sections/biblography.md

@@ -27,6 +27,7 @@
 1. For OS/X users, the GUI ticket viewer is `/System/Library/CoreServices/Ticket\ Viewer.app`
 1. [Colouris01], Colouris, Dollimore & Kindberg, 2001, *Distributed System Concepts and Design*,
 1. [Java 8 GSS API](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/jgss-features.html)
+1. [Ubuntu Kerberos Wiki](https://help.ubuntu.com/community/Kerberos)
 
 ### Kerberos, Active Directory and Apache Hadoop
 

sections/errors.md

@@ -48,7 +48,9 @@ The JVM property MUST be set before UGI is initialized.
 The JVM kerberos operations are configured via the `krb5.conf` file specified in the JVM option
 `java.security.krb5.conf` which can be done on the JVM command line, or inside the JVM
 
-		System.setProperty("java.security.krb5.conf", krbfilepath);
+```java
+System.setProperty("java.security.krb5.conf", krbfilepath);
+```
 
 The JVM property MUST be set before UGI is initialized.
 
@@ -62,7 +64,9 @@ Notes
 
 You can turn Kerberos low-level logging on
 
-		-Dsun.security.krb5.debug=true
+```
+-Dsun.security.krb5.debug=true
+```
 
 This doesn't come out via Log4J, or `java.util logging;` it just comes out on the console. Which is somewhat inconvenient —but bear in mind they are logging at a very low level part of the system. And it does at least log.
 If you find yourself down at this level you are in trouble. Bear that in mind.
@@ -72,15 +76,21 @@ If you find yourself down at this level you are in trouble. Bear that in mind.
 
 If you want to debug what is happening in SPNEGO, another system property lets you enable this:
 
-    -Dsun.security.spnego.debug=true
+```
+-Dsun.security.spnego.debug=true
+```
 
-## Client side JAAS debugging
+## Hadoop-side JAAS debugging
 
 Set the env variable `HADOOP_JAAS_DEBUG` to true and UGI will set the "debug" flag on any JAAS
-files it creates
+files it creates.
 
-    export HADOOP_JAAS_DEBUG=true
+You can do this on the client, before issuing a `hadoop`, `hdfs` or `yarn` command,
+and set it in the environment script of a YARN service to turn it on there.
 
+```
+export HADOOP_JAAS_DEBUG=true
+```
 
 On the next Hadoop command, you'll see a trace like
 
@@ -162,7 +172,7 @@ Current consensus is no: you need DNS set up, or at least a consistent and valid
 
 ## Kerberos's defences against replay attacks
 
-from the javadocs of `org.apache.hadoop.ipc.Client.handleSaslConnectionFailure()`:
+From the javadocs of `org.apache.hadoop.ipc.Client.handleSaslConnectionFailure()`:
 
     /**
      * If multiple clients with the same principal try to connect to the same

sections/secrets.md

@@ -0,0 +1,35 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+  
+   http://www.apache.org/licenses/LICENSE-2.0
+  
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+# Tales of Terror
+
+The following are all true stories. We welcome more submissions of these stories, which will
+all be repeated anonymously.
+
+
+## The Zookeeper's Birthday Present
+
+
+A client program could not work with zookeeper: the connections were being broken. But it
+was working for everything else.
+
+The cluster was one year old that day.
+
+It turns out that ZK reacts to an auth failure by logging something in its logs, and breaking
+the client connection —without any notification to the client. Rather than a network problem
+(initial hypothesis), this was discovered to be an HDFS problem.
+
+When a Kerberos keytab is created, the entries in it have a lifespan. The default value is one
+year. This was its first birthday, hence ZK wouldn't trust the client.
+

sections/terrors.md

@@ -144,40 +144,6 @@ What does that mean? A failure lasts for a while, even if it is a transient one.
 
 This returns the *current* user. 
 
-The current user is not always the same as the logged in user; it changes
-when a service performs an action on the user's behalf
-
-### `createProxyUser()`
-
-Proxy users are a feature which was included in the Hadoop security model for services
-such as Oozie; a service which needs to be able to execute work on behalf of a user 
-
-### `doAs()`
-
-
-This method is at the core of UGI. A call to `doAs()` executes the inner code
-*as the user*. In secure, that means using the Kerberos tickets and Hadoop delegation
-tokens belonging to them.
-
-Example: loading a filesystem as a user
-
-```
-
-UserGroupInformation proxy = 
-  UserGroupInformation.createProxyUser(user,
-   UserGroupInformation.getLoginUser());
-
-FileSystem userFS = proxy.doAs(
-  new PrivilegedExceptionAction<FileSystem>() {
-    public FileSystem run() throws Exception {
-      return FileSystem.get(FileSystem.getDefaultUri(), conf);
-    }
-  });
-```
-
-Here the variable `userFS` contains a client of the Hadoop Filesystem with
-the home directory and access rights of the user `user`. If the user identity
-had come in via an RPC call, they'd
 
 
 
@@ -215,3 +181,44 @@ log4j.logger.org.apache.hadoop.security.authentication=DEBUG
 log4j.logger.org.apache.hadoop.security=DEBUG
 ```
 
+
+## Proxy Users
+
+Some applications need to act on behalf of other users. For example: Oozie wants to run scheduled
+jobs as people, YARN services
+
+
+The current user is not always the same as the logged in user; it changes
+when a service performs an action on the user's behalf
+
+### `createProxyUser()`
+
+Proxy users are a feature which was included in the Hadoop security model for services
+such as Oozie; a service which needs to be able to execute work on behalf of a user 
+
+### `doAs()`
+
+
+This method is at the core of UGI. A call to `doAs()` executes the inner code
+*as the user*. In secure, that means using the Kerberos tickets and Hadoop delegation
+tokens belonging to them.
+
+Example: loading a filesystem as a user
+
+```
+
+UserGroupInformation proxy = 
+  UserGroupInformation.createProxyUser(user,
+   UserGroupInformation.getLoginUser());
+
+FileSystem userFS = proxy.doAs(
+  new PrivilegedExceptionAction<FileSystem>() {
+    public FileSystem run() throws Exception {
+      return FileSystem.get(FileSystem.getDefaultUri(), conf);
+    }
+  });
+```
+
+Here the variable `userFS` contains a client of the Hadoop Filesystem with
+the home directory and access rights of the user `user`. If the user identity
+had come in via an RPC call, they'd

sections/ugi.md

@@ -12,7 +12,7 @@
   limitations under the License. See accompanying LICENSE file.
 -->
 
-# SPNEGO
+# Web, REST and SPNEGO
 
 SPNEGO is the acronym of the protocol by which HTTP clients can authenticate with a web site using Kerberos. This allows the client to identify and authenticate itself to a web site or a web service.
 SPNEGO is supported by
@@ -29,6 +29,8 @@ Unlike, say Hadoop IPC, where the entire authentication code has been implemente
 The sole source of information is the JDK source, and anything which IDE decompilers
 can add if you end up stepping in to vendor-specific classes.
 
+There is [one readme file](https://github.com/ddopson/openjdk-test/blob/master/sun/net/www/protocol/http/spnegoReadme) hidden in the test documentation.
+
 
 ## Configuring Firefox to use SPNEGO