Working MSK & Conduktor demo. Up to date docs

Author: JohnPreston

.pre-commit-config.yaml

@@ -18,7 +18,7 @@ repos:
     hooks:
       - id: black
   - repo: https://github.com/pycqa/isort
-    rev: 5.11.2
+    rev: 5.11.4
     hooks:
       - id: isort
         args: [ "--profile", "black", "--filter-files" ]

CHANGELOG.rst

@@ -2,7 +2,11 @@
 History
 =======
 
-0.0.1 (2022-04-24)
-------------------
+0.2.0 (2023-01-12)
+===================
 
-* First release on PyPI.
+Functional extension module for ecs-compose-x version 0.23. Allows for
+
+* creation of a new MSK cluster
+* lookup to use an existing MSK cluster
+* link services to the cluster to get network and IAM/Kafka access to the cluster

docs/installation.rst

@@ -12,7 +12,7 @@ To install msk_cluster, run this command in your terminal:
 
 .. code-block:: console
 
-    pip install ecs_composex_msk_cluster
+    pip install ecs-composex-msk-cluster
 
 This is the preferred method to install msk_cluster, as it will always install the most recent stable release.
 

docs/kafka_access.rst

@@ -54,6 +54,19 @@ There are 4 profiles to manage topics:
 * Consumer: allows to read data from the topic(s)
 * ProducerConsumer: allows both read and write access to the topic(s)
 
++-------------------------+-----------------------------+-----------------------------+-----------------------------+
+|           Admin         |           Producer          |           Consumer          |       ProducerConsumer      |
++=========================+=============================+=============================+=============================+
+| kafka-cluster:*Topic*   | kafka-cluster:DescribeTopic | kafka-cluster:DescribeTopic | kafka-cluster:DescribeTopic |
+|                         |                             |                             |                             |
+|                         |                             |                             |                             |
+| kafka-cluster:ReadData  | kafka-cluster:WriteData     | kafka-cluster:ReadData      | kafka-cluster:WriteData     |
+|                         |                             |                             |                             |
+|                         |                             |                             |                             |
+| kafka-cluster:WriteData |                             |                             | kafka-cluster:ReadData      |
++-------------------------+-----------------------------+-----------------------------+-----------------------------+
+
+
 group
 ******
 
@@ -62,13 +75,32 @@ There are 2 profiles to manage consumer groups
 * Admin: grants all Kafka ACLs on the security group, including delete
 * Consumer: grants normal use of the consumer group(s) for the application.
 
++-----------------------------+-----------------------------+
+|            Admin            |          Consumer           |
++=============================+=============================+
+| kafka-cluster:DescribeGroup | kafka-cluster:DescribeGroup |
+|                             |                             |
+| kafka-cluster:AlterGroup    | kafka-cluster:AlterGroup    |
+|                             |                             |
+| kafka-cluster:DeleteGroup   |                             |
++-----------------------------+-----------------------------+
+
 transactional-id
 *****************
 
 There is 1 profile for Kafka transactional IDs.
 
 * Producer: allows all Kafka ACLs related to Transactional IDs
 
++---------------------------------------+
+|               Producer                |
++=======================================+
+| kafka-cluster:AlterTransactionalId    |
+|                                       |
+|                                       |
+| kafka-cluster:DescribeTransactionalId |
++---------------------------------------+
+
 Schema Definition
 ==================
 

docs/syntax.rst

@@ -49,7 +49,17 @@ Lookup
 The Lookup for MSK allows to use an existing MSK Cluster and setup all the appropriate IAM permissions to access said
 cluster.
 
-Additionally, you can set `ClientsSecurityGroup`_ which will allow you to automatically add an existing Security Group
+Lookup.Cluster
+---------------
+
+Tags associated with the MSK cluster. The tags must point to a single cluster. If multiple clusters match with the given
+tags, the lookup will result in error.
+
+
+Lookup.ClientsSecurityGroup
+---------------------------
+
+You can set ``ClientsSecurityGroup`` which will allow you to automatically add an existing Security Group
 which would have access to the MSK cluster in order to provide yourself with access to said cluster, on the network.
 
 For example, the below Lookup will search for a cluster with tag **Name** and value **msk-lookup**. It will also
@@ -66,3 +76,5 @@ look for a security group to use for the services in addition to their new defau
           ClientsSecurityGroup:
             Tags:
               - aws:cloudformation:logical-id: newclusterClientsSecurityGroup
+
+.. _Properties for MSK Cluster: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-cluster.html

docs/usage.rst

@@ -10,6 +10,7 @@ Use the extension
 To use the extension, after installing the package, simply create a YAML file (or your existing compose file) and simply
 add ``x-msk_cluster``, to define new MSK clusters to create/lookup.
 
+For example, deploy MSK Cluster & Conduktor to manage the resources within.
 
 .. literalinclude:: ../use-cases/conduktor.yaml
     :language: yaml
@@ -20,5 +21,4 @@ Then we simply run the following command
 
 .. code-block:: console
 
-
     ecs-compose-x render -d templates -n conduktor-msk-iam -f use-cases/conduktor.yaml

ecs_composex_msk_cluster/msk_cluster.py

@@ -5,22 +5,20 @@
 
 from typing import TYPE_CHECKING
 
-from compose_x_common.aws.msk import MSK_CLUSTER_ARN_RE
-from compose_x_common.compose_x_common import keyisset
-from ecs_composex.common import ComposeXSettings
-from ecs_composex.ingress_settings import lookup_security_group
-from troposphere.msk import Cluster as CfnMskCluster
-
 if TYPE_CHECKING:
     from ecs_composex.mods_manager import XResourceModule
     from ecs_composex.common.settings import ComposeXSettings
     from ecs_composex.mods_manager import ModManager
 
+from compose_x_common.aws.msk import MSK_CLUSTER_ARN_RE
+from compose_x_common.compose_x_common import keyisset
 from ecs_composex.common.logging import LOG
 from ecs_composex.compose.x_resources.network_x_resources import NetworkXResource
+from ecs_composex.ingress_settings import lookup_security_group
 from ecs_composex.resource_settings import link_resource_to_services
 from ecs_composex.vpc.vpc_params import STORAGE_SUBNETS
 from troposphere import GetAtt, Ref, Select, Split
+from troposphere.msk import Cluster as CfnMskCluster
 
 from ecs_composex_msk_cluster.msk_cluster_params import (
     CONTROL_CLOUD_ATTR_MAPPING,