File tree Expand file tree Collapse file tree 1 file changed +6
-3
lines changed
Expand file tree Collapse file tree 1 file changed +6
-3
lines changed Original file line number Diff line number Diff line change @@ -27,15 +27,18 @@ The included docker-compose.yml contains link:https://github.com/wurstmeister/ka
2727==== Starting Worker Processes
2828
29291. Launch test services - `docker-compose up setup`
30- 2. Compile: `cargo build`
31- 3. Run kafka-delta-ingest against the web_requests example topic and table (customize arguments as desired):
30+ 1. Download and extract kafka: `curl -L https://dlcdn.apache.org/kafka/3.9.0/kafka_2.13-3.9.0.tgz | tar -xz`
31+ 1. Create kafka topic: `./kafka_2.13-3.9.0/bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic web_requests --create --if-not-exists`
32+ 1. Ingest test messages into kafka: `./kafka_2.13-3.9.0/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic web_requests < tests/json/web_requests-100.json`
33+ 1. Compile: `cargo build --features s3`, MacOS: `brew install librdkafka && cargo build --features s3,dynamic-linking`
34+ 1. Run kafka-delta-ingest against the web_requests example topic and table (customize arguments as desired):
3235
3336```bash
3437export AWS_ENDPOINT_URL=http://0.0.0.0:4566
3538export AWS_ACCESS_KEY_ID=test
3639export AWS_SECRET_ACCESS_KEY=test
3740
38- RUST_LOG=debug cargo run ingest web_requests ./tests/data/web_requests \
41+ RUST_LOG=debug cargo run --features s3,dynamic-linking ingest web_requests ./tests/data/web_requests \
3942 --allowed_latency 60 \
4043 --app_id web_requests \
4144 --transform 'date: substr(meta.producer.timestamp, `0`, `10`)' \
You can’t perform that action at this time.
0 commit comments