Carabas: Update documentation

Author: Andreas Motl

CHANGES.md

@@ -1,7 +1,7 @@
 # Changelog
 
 ## in progress
-- Started unlocking AWS Kinesis stream sources
+- Carabas: A subsystem to divert workloads to other people’s computers
 
 ## 2024-07-10 v0.0.2
 - Initial working version, supporting MQTT, AMQP, and SQLAlchemy/CrateDB

doc/carabas/dms/index.md

@@ -26,7 +26,7 @@ help.
 ### Install
 Install LorryStream.
 ```shell
-pip install lorrystream
+pip install --upgrade 'lorrystream[carabas]'
 ```
 Acquire IaC driver program.
 ```shell

doc/carabas/kcl/dynamodb.md

@@ -121,15 +121,16 @@ aws kinesis delete-stream --stream-name dynamodb-cdc --enforce-consumer-deletion
 
 Acquire sources and initialize sandbox.
 ```shell
-git clone https://github.com/daq-tools/lorrystream --branch=kinesis
+git clone https://github.com/daq-tools/lorrystream
 cd lorrystream
 python3 -m venv .venv
 source .venv/bin/activate
+pip install --editable='.[carabas]'
 ```
 
 Install dependencies, mainly the [amazon-kclpy] package.
 ```shell
-cd lorrystream/dynamodb_cloud
+cd lorrystream/spike/kcl_dynamodb
 pip install wheel
 pip install --verbose -r requirements.txt
 ```
@@ -141,7 +142,7 @@ virtualenv on the top-level directory. Then, navigate to the playground
 directory, and seed AWS credentials.
 ```shell
 source .venv/bin/activate
-cd lorrystream/dynamodb_cloud
+cd lorrystream/spike/kcl_dynamodb
 export AWS_ACCESS_KEY=...
 export AWS_SECRET_ACCESS_KEY=...
 ```

doc/carabas/kcl/kinesis.md

@@ -34,15 +34,16 @@ permissions to do so.
 
 Acquire sources and initialize sandbox.
 ```shell
-git clone https://github.com/daq-tools/lorrystream --branch=kinesis
+git clone https://github.com/daq-tools/lorrystream
 cd lorrystream
 python3 -m venv .venv
 source .venv/bin/activate
+pip install --editable='.[carabas]'
 ```
 
 Install dependencies, mainly the [amazon-kclpy] package.
 ```shell
-cd lorrystream/kinesis
+cd lorrystream/spike/kcl_kinesis
 pip install wheel
 pip install --verbose -r requirements.txt
 ```
@@ -65,7 +66,7 @@ virtualenv on the top-level directory. Then, navigate to the playground
 directory, and seed AWS credentials.
 ```shell
 source .venv/bin/activate
-cd lorrystream/kinesis
+cd lorrystream/spike/kcl_kinesis
 export AWS_ACCESS_KEY=...
 export AWS_SECRET_ACCESS_KEY=...
 ```

doc/source/kinesis.md

@@ -1,9 +1,19 @@
 # Kinesis Source
 
-## LocalStack Testbed
-The recipe uses the LocalStack AWS environment to run an Amazon Kinesis surrogate.
+This recipe uses the LocalStack AWS environment to run an Amazon Kinesis surrogate.
 The walkthrough follows the [Get started with Kinesis on LocalStack] tutorial.
 
+If you intend to invoke the commands on a real AWS environment, just use `aws`
+instead of `awslocal`.
+
+:::{tip}
+LocalStack is a cloud service emulator that runs in a single container on your
+laptop or in your CI environment. With LocalStack, you can run your AWS
+applications or Lambdas entirely on your local machine without connecting to
+a remote cloud provider.
+:::
+
+## Setup
 Start the LocalStack service using Docker.
 ```shell
 docker run \
@@ -13,17 +23,13 @@ docker run \
   -v /var/run/docker.sock:/var/run/docker.sock \
   localstack/localstack:3.6
 ```
-:::{tip}
-LocalStack is a cloud service emulator that runs in a single container on your
-laptop or in your CI environment. With LocalStack, you can run your AWS
-applications or Lambdas entirely on your local machine without connecting to
-a remote cloud provider.
-:::
 
 Install LorryStream including LocalStack CLI programs.
 ```shell
-pip install lorrystream
+pip install --upgrade 'lorrystream[carabas]'
 ```
+
+## Configure
 Create a Kinesis Data Stream called `testdrive`.
 ```shell
 awslocal kinesis create-stream \
@@ -43,6 +49,7 @@ Display Stream ARN.
 awslocal kinesis describe-stream --stream-name testdrive | jq -r .StreamDescription.StreamARN
 ```
 
+## Usage
 Submit an item to the data stream, using `awslocal`.
 ```shell
 awslocal kinesis put-record \
@@ -64,4 +71,10 @@ export AWS_ENDPOINT_URL="http://localhost:4566"
 python examples/aws/kinesis_subscribe.py testdrive
 ```
 
+:::{todo}
+Demonstrate how to add a processor pipeline element using both either
+AWS Lambda, or a dedicated processor instance.
+:::
+
+
 [Get started with Kinesis on LocalStack]: https://docs.localstack.cloud/user-guide/aws/kinesis/