go to language-tagged code snippets, asf header everywhere

steveloughran · steveloughran · commit 62d1f5eefc07 · 2016-01-07T17:33:08.000Z
diff --git a/sections/acknowledgements.md b/sections/acknowledgements.md
@@ -1,3 +1,16 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+  
+   http://www.apache.org/licenses/LICENSE-2.0
+  
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
 # Acknowledgements
 
 * Everyone who has struggled to secure Hadoop deserves to be recognised, their sacrifice acknowledged.
diff --git a/sections/biblography.md b/sections/biblography.md
@@ -1,3 +1,16 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+  
+   http://www.apache.org/licenses/LICENSE-2.0
+  
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
 # Bibliography
 
 1. IETF [RFC 4120](https://www.ietf.org/rfc/rfc4120.txt)
diff --git a/sections/checklists.md b/sections/checklists.md
@@ -110,6 +110,7 @@
 [ ] reverse DNS lookup of IPAddr returns hostname
 [ ] clock is in sync with rest of cluster: `date`
 
+[ ] JVM has Java Crypto Extensions
 [ ] keytab exists
 [ ] keytab is readable by account running service.
 [ ] keytab contains principals in listing `ktlist -kt $keytab`
diff --git a/sections/glossary.md b/sections/glossary.md
@@ -1,3 +1,16 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+  
+   http://www.apache.org/licenses/LICENSE-2.0
+  
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
 # Glossary
 
  
diff --git a/sections/hadoop_and_kerberos.md b/sections/hadoop_and_kerberos.md
@@ -39,5 +39,3 @@ SSL-certificate like system? Or OAuth?
 Kerberos was written to support centrally managed accounts in a local area network, one in
 which adminstrators manage individual accounts. This is actually much simpler to manage than
 PKI-certificate based systems: look at the effort it takes to revoke a certificate in a browser.
-
-OAuth? 
diff --git a/sections/hadoop_tokens.md b/sections/hadoop_tokens.md
@@ -330,3 +330,6 @@ work to Oozie to have a keytab and to pass it to Oozie.
 ## Weaknesses
 
 1. Any compromised DN can create block tokens.
+1. Possession of the tokens is sufficent to impersonate a user. This means it is critical
+to transport tokens over the network in an encrypted form. Typically, this is done
+by SASL-encrypting the Hadoop IPC channel.
diff --git a/sections/ipc.md b/sections/ipc.md
@@ -41,7 +41,7 @@ In its favour: it's a lot easier than SPNEGO.
 
 ### Annotating a service interface
 
-```
+```java
 @KerberosInfo(serverPrincipal = "my.kerberos.principal")
 public interface MyRpc extends VersionedProtocol {
   long versionID = 0x01;
@@ -59,7 +59,7 @@ Every exported RPC service will need its own extension of the `SecurityInfo` cla
 ### `PolicyProvider` subclass
 
 
-```
+```java
 public class MyRpcPolicyProvider extends PolicyProvider {
 
   public Service[] getServices() {
@@ -69,12 +69,11 @@ public class MyRpcPolicyProvider extends PolicyProvider {
   }
 
 }
-
 ```
 
  This is used to inform the RPC infrastructure of the ACL policy: who may talk to the service. It must be explicitly passed to the RPC server
 
-```
+```java
 rpcService.getServer() .refreshServiceAcl(serviceConf, new MyRpcPolicyProvider());
 ```
 
@@ -104,7 +103,7 @@ the server can determine the identity of the principal.
 
 This is something it can ask for when handling the RPC Call:
 
-```
+```java
 UserGroupInformation callerUGI;
 
 // #1: get the current user identity
@@ -116,13 +115,13 @@ try {
   throw RPCUtil.getRemoteException(ie);
 }
 ```
-    
+
 The `callerUGI` variable is now set to the identity of the caller. If the caller
 has delegated authority (tickets, tokens) then they still authenticate as
 that principal they were acting as (possibly via a `doAs()` call).
-    
 
-```
+
+```java
 // #2 verify their permissions
 String user = callerUGI.getShortUserName();
 if (!checkAccess(callerUGI, MODIFY)) {
@@ -162,12 +161,12 @@ hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true hdfs://secure:8
 
 Although you can set it in a core-site.xml, this is dangerous from a security perpective
 
-```
+```xml
 <property>
   <name>ipc.client.fallback-to-simple-auth-allowed</name>
   <value>true</value> 
 </property>
 ```
 
 *warning* it's tempting to turn this on during development, as it makes problems go away. As it is
-not recommended in production: avoid except on the CLI during attempts to debug problems.
+not recommended in production: avoid except on the CLI during attempts to debug problems.
diff --git a/sections/jaas.md b/sections/jaas.md
@@ -39,9 +39,9 @@ within the same file.
 
 Hadoop's UGI class will dynamically create a JAAS context for Hadoop logins, dynamically determining the name of the kerberos module to use. For interacting purely with HDFS and YARN, you may be able to avoid needing to know about or understand JAAS.
 
-Example of a JAAS file valid for Sun
+Example of a JAAS file valid for an Oracle JVM:
+ 
 
-If you need a basic JAAS cient configuration which 
 ```
 Client {
   com.sun.security.auth.module.Krb5LoginModule required
diff --git a/sections/kerberos_the_madness.md b/sections/kerberos_the_madness.md
@@ -4,7 +4,7 @@
 
 Authors:
 
-S.A.Loughran
+S.A. Loughran
 
 
 ----
diff --git a/sections/keytabs.md b/sections/keytabs.md
@@ -22,7 +22,8 @@ without prompts for passwords
 
 If your management tools sets up keytabs for you: use it.
 
-```
+```bash
+
 kadmin.local
 
 ktadd -k zk.service.keytab -norandkey zookeeper/devix@COTHAM 
@@ -32,14 +33,14 @@ exit
 
 and of course, make it accessible
 
-```
+```bash
 chgrp hadoop zk.service.keytab
 chown zookeeper zk.service.keytab
 ```
 
 check that the user can login
 
-```
+```bash
 # sudo -u zookeeper klist -e -kt zk.service.keytab
 # sudo -u zookeeper kinit -kt zk.service.keytab zookeeper/devix.cotham.uk
 # sudo -u zookeeper klist
diff --git a/sections/secrets.md b/sections/secrets.md
@@ -43,7 +43,7 @@ Examples
 The JVM property MUST be set before UGI is initialized.
 
 
-*2. Switch to an alternate `krb5.conf` file.
+*2. Switch to an alternate `krb5.conf` file.*
 
 The JVM kerberos operations are configured via the `krb5.conf` file specified in the JVM option
 `java.security.krb5.conf` which can be done on the JVM command line, or inside the JVM
@@ -191,4 +191,4 @@ into keytabs.
 
 As a result, even if the keytabs are compromised, *they do not grant any access to and
 enterprise-wide kerberos-authenticated services.
- 
+ 
diff --git a/sections/services.md b/sections/services.md
@@ -29,5 +29,3 @@ the user.
 Note that in Hadoop 2.6/2.7 the Filesystem cache creates a new filesystem instance for the given user, even
 if there is an FS client for that specific filesystem already in the cache. (VERIFY)
 
-## Ser
-Some services have the ability 
diff --git a/sections/testing.md b/sections/testing.md
@@ -96,7 +96,7 @@ futile attempts to work with DNS/rDNS.
 ### Core Hadoop
 
 
-```
+```xml
 <property>
   <name>hadoop.security.authentication</name>
   <value>kerberos</value>
@@ -110,7 +110,7 @@ futile attempts to work with DNS/rDNS.
 
 ### HBase
 
-```
+```xml
 <property>
   <name>hbase.security.authentication</name>
   <value>kerberos</value>
@@ -158,4 +158,4 @@ access rights.
 1. YARN applications: verify that REST/Web requests against the real app URL (which can be
 determined from the YARN application record), are redirected to the RM proxy (i.e. that
 all GET calls result in a 30x redirect). If this does not take place, it means that the RM-hosted
-SPNEGO authentication layer can be bypassed.
+SPNEGO authentication layer can be bypassed.
diff --git a/sections/the_limits_of_hadoop_security.md b/sections/the_limits_of_hadoop_security.md
@@ -91,5 +91,3 @@ Mitigation strategies
 3. Never have your programs ask for more rights than they need, to data, to database tables (and in HBase and Accumulo: columns)
 4. Log data in a form which can be used for audit logs. (Issue: what is our story here? Logging to local/remote filesystems isn't it, not if malware could overwrite the logs)
 
-----
- 
diff --git a/sections/ugi.md b/sections/ugi.md
@@ -20,7 +20,7 @@
 
 If there is one class guaranteed to strike fear into anyone with experience in Hadoop+Kerberos code it is `UserGroupInformation`, abbreviated to "UGI"
 
-Nobody says `UserGroupInformation` out loud; it is the *him which must not be named* of the stack
+Nobody says `UserGroupInformation` out loud; it is the *that which must not be named* of the stack
 
 ## What does UGI do?
 
@@ -55,7 +55,7 @@ reset the information).
  As a result: init before you go near the filesystem, with the principal you want.
 * It has to do some low-level reflection-based access to Java-version-specific Kerberos internal classes.
 This can break across Java versions, and JVM implementations. Specifically Java 8 has classes that Java 6 doesn't; the IBM JVM is very different.
-* All its exceptions are basic `IOException` instancss, so hard to match on without looking at the text, which is very brittle.
+* All its exceptions are basic `IOException` instances, so hard to match on without looking at the text, which is very brittle.
 * Some invoked operations are relayed without the stack trace (this should now be fixed).
 * Diagnostics could be improved. (this is one of those British understatements, it really means "it would be really nice if you could actually get any hint as to WTF is going inside the class as otherwise you are left with nothing to go on other than some message that a user at a random bit of code wasn't authorized)
 
diff --git a/sections/web_and_rest.md b/sections/web_and_rest.md
@@ -17,7 +17,7 @@
 SPNEGO is the acronym of the protocol by which HTTP clients can authenticate with a web site using Kerberos. This allows the client to identify and authenticate itself to a web site or a web service.
 SPNEGO is supported by
 
-* the standard browsers, to different levels of pain of use
+* The standard browsers, to different levels of pain of use
 * `curl` on the command line
 * `java.net.URL` in Java7+
 
@@ -32,9 +32,12 @@ can add if you end up stepping in to vendor-specific classes.
 
 ## Configuring Firefox to use SPNEGO
 
-Firefox is the easiest browser to set up with SPNEGO support, as it is done in about:config and then persisted
-Here are the settings for a local VM, a VM which has an entry in the /etc/hosts:
+Firefox is the easiest browser to set up with SPNEGO support, as it is done in `about:config `and then persisted
+Here are the settings for a local VM, a VM which has an entry in the `/etc/hosts`:
+
+```
 192.168.1.134 devix.cotham.uk devix
+```
 
 This hostname is then listed in firefox's config as a URL to trust.
 
@@ -127,20 +130,22 @@ TODO:
 ## Identifying and Authenticating callers in Web/REST endpoints
     
 Here's some code from `org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices`
-    
-      @PUT
-      @Path("/jobs/{jobid}/tasks/{taskid}/attempts/{attemptid}/state")
-      @Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
-      @Consumes({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
-      public Response updateJobTaskAttemptState(JobTaskAttemptState targetState,
-          @Context HttpServletRequest hsr, @PathParam("jobid") String jid,
-          @PathParam("taskid") String tid, @PathParam("attemptid") String attId)
-              throws IOException, InterruptedException {
-        init();
-    
-        String remoteUser = hsr.getRemoteUser();
-        UserGroupInformation callerUGI = null;
-        if (remoteUser != null) {
-          callerUGI = UserGroupInformation.createRemoteUser(remoteUser);
-        }
+
+```java
+@PUT
+@Path("/jobs/{jobid}/tasks/{taskid}/attempts/{attemptid}/state")
+@Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
+@Consumes({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
+public Response updateJobTaskAttemptState(JobTaskAttemptState targetState,
+    @Context HttpServletRequest hsr, @PathParam("jobid") String jid,
+    @PathParam("taskid") String tid, @PathParam("attemptid") String attId)
+        throws IOException, InterruptedException {
+  init();
+
+  String remoteUser = hsr.getRemoteUser();
+  UserGroupInformation callerUGI = null;
+  if (remoteUser != null) {
+    callerUGI = UserGroupInformation.createRemoteUser(remoteUser);
+  }
+```
 
diff --git a/sections/zookeeper.md b/sections/zookeeper.md
