feat: Add topology-provider (#28)

* feat: Add topology-provider

* changelog

* fix yaml

* fix yaml part 2

Author: sbernauer

CHANGELOG.md

@@ -0,0 +1,11 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+### Added
+
+- Add topology-provider from https://github.com/stackabletech/hdfs-topology-provider ([#28]).
+
+[#28]: https://github.com/stackabletech/hdfs-utils/pull/28

README.md

@@ -6,9 +6,10 @@ This project contains multiple plugins for Apache Hadoop, which are intended to
 
 1. OpenPolicyAgent (OPA) authorizer: For every action performed in HDFS an OPA instance is asked if the user is allowed to perform the action.
 2. GroupMapper: It can look up group memberships for users from an OPA instance.
-3. Not (yet?) in this repository is a [TopologyProvider](https://github.com/stackabletech/hdfs-topology-provider/).
+3. TopologyProvider: It is used to feed information from Kubernetes into the HDFS rack awareness functionality.
 
 ## Installation
+
 Run `mvn package` and put the resulting `target/hdfs-utils-*.jar` file on your HDFS classpath.
 The easiest way to achieve this is to put it in the directory `/stackable/hadoop/share/hadoop/tools/lib/`.
 The Stackable HDFS already takes care of this, you don't need to do anything in this case.
@@ -650,3 +651,64 @@ and the Hadoop logs will show that the lookup has taken place:
 - Groups for [nn]: [[admin, superuser]]
 ```
 
+## Network Topology Provider
+
+Hadoop supports a concept called [rack awareness](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/RackAwareness.html) (lately also often called topology awareness).
+
+Historically this has been used to distinguish racks, datacenters and the like and often been read from a manually updated topology file or similar solutions.
+
+In Kubernetes, the most commonly used mechanism for topology awareness are labels - mostly labels set on the Kubernetes nodes.
+The most prevalent example for this is the node label [topology.kubernetes.io/zone](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone) which often refers to availability zones in cloud providers or similar things.
+
+The purpose of this tool is to feed information from Kubernetes into the HDFS rack awareness functionality.
+In order to do this, it implements the Hadoop interface `org.apache.hadoop.net.DNSToSwitchMapping` which then allows this tool to be configured on the NameNode via the parameter `net.topology.node.switch.mapping.impl`.
+
+The topology provider watches all HDFS pods deployed by Stackable and Kubernetes nodes and keeps an in memory cache of the current state of these objects.
+From this state store the tool can then calculate rack IDs for nodes that HDFS asks for without needing to talk to the api-server and incurring an extra network round-trip.
+
+Results are cached for a configurable amount of time and served from the cache if present.
+
+In a Kubernetes environment it is likely that the majority of writes will not come from the DataNodes themselves, but rather from other processes such as Spark executors writing data to HDFS. The NameNode passes these on to the topology provider to request the rack ID i.e. it provides the IP addresses of whichever pods are doing the writing. If a datanode resides on the same Kubernetes node as one of these pods, then this datanode is used for label resolution for that pod.
+
+## Configuration
+
+Configuration of the tool happens via environment variables, as shown below:
+
+### TOPOLOGY_LABELS
+
+A semicolon separated list of labels that should be used to build a rack id for a datanode.
+
+A label is specified as `[node|pod]:<labelname>`
+
+Some examples:
+
+|            Definition            |                                             Resolved to                                              |
+|----------------------------------|------------------------------------------------------------------------------------------------------|
+| node:topology.kubernetes.io/zone | The value of the label 'topology.kubernetes.io/zone' on the node to which the pod has been assigned. |
+| pod:app.kubernetes.io/role-group | The value of the label 'app.kubernetes.io/role-group' on the datanode pod.                           |
+
+Multiple levels of labels can be combined (up to MAX_TOPOLOGY_LABELS levels) by separating them with a semicolon:
+
+So for example `node:topology.kubernetes.io/zone;pod:app.kubernetes.io/role-group` would resolve to `/<value of label topology.kubernetes.io/zone on the node>/<value of label app.kubernetes.io/role-group on the pod>`.
+
+### TOPOLOGY_CACHE_EXPIRATION_SECONDS
+
+Default: 5 Minutes
+
+The default time for which rack ids are cached, HDFS can influence caching, as the provider offers methods to invalidate the cache, so this is not a totally _reliable_ value, as there are external factors not under the control of this tool.
+
+### MAX_TOPOLOGY_LEVELS
+
+Default: 2
+
+The maximum number of levels that can be specified to build a rack id from.
+
+While this can be changed, HDFS probably only supports a maximum of two levels, so it is not recommended to change the default for this setting.
+
+## Testing
+
+There are currently no unit tests.
+
+CRDs for spinning up test infrastructure are provided in `test/stack`.
+
+The actual testing for this happens in the integration tests of the HDFS operator.

pom.xml

@@ -75,9 +75,26 @@
     <dependency>
       <groupId>com.fasterxml.jackson.core</groupId>
       <artifactId>jackson-databind</artifactId>
-      <version>2.12.7.1</version>
+      <version>2.17.1</version>
       <scope>provided</scope>
     </dependency>
+    <!-- Needed by topology-provider -->
+    <dependency>
+      <groupId>com.github.ben-manes.caffeine</groupId>
+      <artifactId>caffeine</artifactId>
+      <version>3.1.8</version>
+    </dependency>
+    <dependency>
+      <groupId>io.fabric8</groupId>
+      <artifactId>kubernetes-client</artifactId>
+      <version>6.13.1</version>
+    </dependency>
+    <dependency>
+      <groupId>io.fabric8</groupId>
+      <artifactId>kubernetes-client-api</artifactId>
+      <version>6.13.1</version>
+    </dependency>
+    <!-- End of needed by topology-provider -->
     <dependency>
       <groupId>junit</groupId>
       <artifactId>junit</artifactId>