You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Adding in new environment variable for configuring Kafka Segment Length
* Adding some descriptive text on Kafka Log rotation principles
* Add topics for revocable lane events
* Adding in RSU status Connector to Jikkou
* Adding in RSU Status connector
* Removing Mongo Index Env vars
* Add notification topics
* Update Kafka and Kafka connectors with RTCM
* Adding Connectors and Indexes for Revocable Lanes
* Adding docs and scripts for how to manage data in mongoDB
* Add jpo-ode RSM topics to Jikkou
* rtcm topics
* Remove RawEncodedTimJson from MongoDB and the Deduplicator connectors
* Remove OdeRawEncoded connectors
* Adding additionalIndexes generator and OdeBsmJson index
* Add topics
* cimms priority/preemption event topic
* Add to mongo script
* add processed ssm/srm connectors and mongo collections
* Update release notes for 2025 q3 release
* Update bitnami kafka
* Update release notes to be accurate with PR number
* Add note regarding intention to switch to official kafka image
* Minor PR fixes
* Adding sample output for storage management readme.
---------
Co-authored-by: john-wiens <[email protected]>
Co-authored-by: jacob6838 <[email protected]>
Co-authored-by: jacob6838 <[email protected]>
Co-authored-by: Michael7371 <[email protected]>
Co-authored-by: Ivan Yourshaw <[email protected]>
Copy file name to clipboardExpand all lines: README.md
+5-1Lines changed: 5 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -83,7 +83,7 @@ Where the `COMPOSE_PROFILES` variable in you're `.env` file are as follows:
83
83
84
84
## 3. Kafka
85
85
86
-
The [Bitnami Kafka](https://hub.docker.com/r/bitnami/kafka) is being used as a hybrid controller and broker in the [docker-compose-kafka](docker-compose-kafka.yml) file. To use a different `kafka_init.sh` script, pass in the relative path of the new script by overriding the `KAFKA_INIT_SCRIPT_RELATIVE_PATH` environmental variable. This can help in initializing new topics at startup.
86
+
The [Bitnami Legacy Kafka](https://hub.docker.com/r/bitnamilegacy/kafka) is being used as a hybrid controller and broker in the [docker-compose-kafka](docker-compose-kafka.yml) file. To use a different `kafka_init.sh` script, pass in the relative path of the new script by overriding the `KAFKA_INIT_SCRIPT_RELATIVE_PATH` environmental variable. This can help in initializing new topics at startup. Note: the intention for the future is to switch this to use the official Apache Kafka image since Bitnami is no longer a reliable image provider.
87
87
88
88
An optional `kafka-init`, `schema-registry`, and `kafka-ui` instance can be deployed by configuring the `COMPOSE_PROFILES` as follows:
89
89
@@ -93,6 +93,10 @@ An optional `kafka-init`, `schema-registry`, and `kafka-ui` instance can be depl
93
93
-`kafka_schema_registry` - deploys a `kafka-schema-registry` service that can be used to manage schemas for kafka topics
94
94
-`kafka_ui` - deploys a [web interface](https://github.com/kafbat/kafka-ui) to interact with the kafka cluster
95
95
96
+
### Kafka Topic Log Retention and Rotation
97
+
98
+
During operation, Kafka stores all the messages published to topics in files called logs. These logs are stored on the host system and can quickly become quite large for deployments with real CV volumes of data. A kafka log may be split across multiple files called segments. Each segment is limited in size based upon the KAFKA_LOG_SEGMENT_BYTES environment variable. The number of segments stored is variable but collectively, the total volume of all segments for 1 partition will not exceed the value specified in KAFKA_LOG_RETENTION_BYTES. When Kafka needs to delete data, either because the total KAFKA_LOG_RETENTION_BYTES is exceeded or because data in the oldest segment exceeds the KAFKA_LOG_RETENTION_HOURS an entire segment file will be deleted. Please note, Kafka will never delete the active log segment. So even if the data is far older than the value specified in KAFKA_LOG_RETENTION_HOURS the data may not be deleted if there is not enough data to fill up the segment and cause the creation of a new log segment. Additionally, please note that the values specified in KAFKA_LOG_RETENTION_BYTES and KAFKA_LOG_SEGMENT_BYTES are on a per topic per partition basis. So for a deployment with multiple partitions and topics, the total used storage will likely become quite large even for what may seem like small individual log and segment sizes. When configuring the KAFKA_LOG_SEGMENT_BYTES and KAFKA_LOG_RETENTION_BYTES variables. Make sure that the KAFKA_LOG_RETENTION_BYTES is larger than the KAFKA_LOG_SEGMENT_BYTES. If the value is not larger, Kafka will not be able to generate a new log segment before needing to rotate the logs which will lead to unpredictable behavior.
99
+
96
100
### Configure Topic Creation
97
101
98
102
The Kafka topics created by the `kafka-setup` service are configured in the [kafka-topics-values.yaml](jikkou/kafka-topics-values.yaml) file. The topics in that file are organized by the application, and sorted into "Stream Topics" (those with `cleanup.policy` = `delete`) and "Table Topics" (with `cleanup.policy` = `compact`).
Copy file name to clipboardExpand all lines: docs/Release_notes.md
+18Lines changed: 18 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,23 @@
1
1
# JPO-UTILS Release Notes
2
2
3
+
## Version 2.2.0
4
+
----------------------------------------
5
+
### **Summary**
6
+
In this release, support for the JPO ODE pipeline was expanded and updated to align with the jpo-ode 5.0.2 rework and to incorporate additional J2735 message types alongside CIMMS lane alignment features. New and updated Kafka topics, connectors, and MongoDB indexes were introduced to support RTCM, SDSM, and RSM messages. Configuration was simplified by removing redundant MONGO_INDEX_* environment variables in favor of CONNECT_CREATE, while deprecated topics and collections were removed to reduce technical debt. Further improvements include enhanced index creation via create_indexes.js, updated default Jikkou configurations, and new documentation with helper scripts for managing MongoDB storage.
7
+
8
+
Enhancements in this release:
9
+
-[CDOT PR 28](https://github.com/CDOT-CV/jpo-utils/pull/28): Rsu Status Message Support
The default configuration of the CV-Manager system is very data intensive and stores multiple forms of CV-Data redundantly in the database. While this provides some benefit for compliance and debugging purposes, it is also often a strain on limited cloud and storage budgets. Below are some tips and tricks for reducing the storage footprint of CV-Data
4
+
5
+
## Stop syncing unused collections
6
+
By Default JPO-utils syncs a number of topics created by the ode, geojson-converter, Conflict Monitor and CV-Manager. However, many of these collections are unused by the CV-Manager system and therefore not required to be saved to the database. In particular, data formats such as the OdeSpatJson format which have lots of traffic can quickly become a storage burden on the system.
7
+
8
+
1) To stop a topic from syncing to mongoDB users can modify the `kafka-connectors-values.yaml` file and remove entries. Make sure to remove the entire entry for the given topic. For example, to stop the OdeSpatJson topic from syncing remove the following:
9
+
```
10
+
- topicName: topic.OdeSpatJson
11
+
collectionName: OdeSpatJson
12
+
generateTimestamp: true
13
+
```
14
+
15
+
Please note, that none of the Ode collections are required for running the GeoJson Converter, Conflict Monitor, or CV-Manager systems. The OdeSpatJson and OdeMapJson collections in particular should be deleted first as they generally contains significant data.
16
+
17
+
2) Once a topic is deleted from the `kafka-connectors-values.yaml` file. Users can bring up the remaining connectors by rebuilding the kafka-setup container (see step 3). For existing deployments that already have the connectors created. The removed connectors will need to be removed manually. This can be done by manually deleting connectors using curl.
Alternatively, users may delete all existing topics are recreate them. This may be faster if updating a large number of connectors. The [delete_connectors.sh](../kafka-connect/delete_connectors.sh) script will automatically delete all existing connectors.
22
+
```
23
+
cd kafka-connect
24
+
./delete_connectors.sh
25
+
```
26
+
27
+
3) Once all the unused connectors have been deleted, reapply the connectors scripts to the cluster by rebuilding the kafka-connect-setup container. Make sure kafka_connect, kafka_connect_standalone, or kafka_connect_setup is specified in the COMPOSE_PROFILES variable of your .env file before rebuilding to ensure the setup container is rerun
28
+
```
29
+
docker-compose up --build -d
30
+
```
31
+
32
+
33
+
34
+
35
+
## Change MongoDB Data Retention Time
36
+
By default all collections in the default mongoDB are only retained for a set time window. Depending on deployment requirements, a shorter retention window may be desirable to reduce storage costs and improve database query speeds. Data retention is managed by adjusting TTL's on collections in mongoDB. Below are instructions to adjust the data retention of an existing mongoDB deployment.
37
+
38
+
1) There are currently two environment variables that configure the data retention duration in mongoDB. The `MONGO_DATA_RETENTION_SECONDS` variable configures the number of seconds that data generated within the system should be retained for. Similarly, the `MONGO_ASN_RETENTION_SECONDS` variable configures the number of seconds that raw ASN.1 data should be retained for. By default Mongo Data is retained for 60 days, while ASN.1 data is configured for 24 hours. To change the data retention window, modify the .env file in the jpo-utils repo, or parent module and provide new values for these variables
39
+
```
40
+
MONGO_DATA_RETENTION_SECONDS=604800 # 1 week
41
+
MONGO_ASN_RETENTION_SECONDS=3600 # 1 Hour
42
+
```
43
+
44
+
2) Once the variables are changed, rebuild the docker-setup container and let it run its setup procedure. Make sure that one of the following is specified in the `COMPOSE_PROFILES` variable to ensure the mongo setup container runs mongo, mongo_full, mongo_setup. Then run the following to reboot
45
+
46
+
```
47
+
docker-compose up --build -d
48
+
```
49
+
50
+
3) Once the indexes have been updated, verify that the change worked properly by checking the TTL index for the adjusted collections. This can be done by logging into the database using
0 commit comments