docs: document 2.9 indexes and changes in schema (#112)

* docs: use correct index definitions

* docs: upgrade index creation instructions

* docs: add indexes for 2.9

* doc: document changes in Mongodb schema

* docs: generate Readme

* docs: improve docs

* docs: improve index docs

Author: masontikhonov

README.md

@@ -1491,7 +1491,8 @@ Use [obtain-oidc-id-token](https://github.com/codefresh-io/steps/blob/822afc0a9a
 
 Sometimes, in new releases of Codefresh On-Prem, index requirements change. When this happens, it's mentioned in the [Upgrading section](#upgrading) for the specific release.
 
-> ℹ️ If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
+> [!TIP]
+> If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
 
 ### Index alignment
 
@@ -1500,10 +1501,9 @@ The required index definitions for each release can be found at the following re
 - `2.6` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.6/indexes>
 - `2.7` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.7/indexes>
 - `2.8` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.8/indexes>
+- `2.9` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.9/indexes>
 
-The indexes are stored in JSON files with keys and options specified.
-
-The directory structure is:
+The indexes specifications are stored in JSON files. The directory structure is:
 
 ```console
 indexes
@@ -1514,11 +1514,12 @@ indexes
 **Overview of the index alignment process:**
 
 1. Identify the differences between the indexes in your MongoDB instance and the required index definitions.
-2. Create any missing indexes one by one. (It's important not to create them in bulk.)
+2. Create any missing indexes.
 3. Perform the upgrade of Codefresh On-Prem installation.
 4. Then remove any unnecessary indexes.
 
-> ⚠️ **Note! Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
+> [!IMPORTANT]
+> Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
 >
 > Building indexes during time periods where the target collection is under heavy write load can result in reduced write performance and longer index builds. ([*Source: MongoDB official documentation*](https://www.mongodb.com/docs/manual/core/index-creation/#index-build-impact-on-database-performance))
 >
@@ -1540,19 +1541,32 @@ mongosh "<connection_string>"
 db.getSiblingDB('<db_name>').getCollection('<collection_name>').getIndexes()
 ```
 
-- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones
+- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones.
 
 **Index creation**
 
-> ⚠ **Note! Always create indexes sequentially, one by one. Don't create them in bulk.**
+- To create an indexes, we recommend using the `createIndexes` command ([ref](https://www.mongodb.com/docs/manual/reference/command/createIndexes/)):
 
-- To create an index, use the `createIndex()` method:
+> [!IMPORTANT]
+> We recommend to create indexes in batches of 3 indexes at a time.
+> However, it's highly recommended before creating indexes in production DB to test performance impact on a staging instance with prod-like amount of data.
+>
+> Previous command should be completed before starting the next batch.
 
 ```js
-db.getSiblingDB('<db_name>').getCollection('<collection_name>').createIndex(<keys_object>, <options_object>)
+db.getSiblingDB('<db_name>').runCommand(
+  {
+    createIndexes: '<collection_name>',
+    indexes: [
+        { ... },  // Index definition from the doc above
+        { ... },  // Index definition from the doc above
+        { ... }   // Index definition from the doc above
+    ],
+  }
+)
 ```
 
-After executing the `createIndex()` command, you should see a result indicating that the index was created successfully.
+After executing the command, you should see a result indicating that the indexes were created successfully.
 
 **Index removal**
 
@@ -1566,7 +1580,8 @@ db.getSiblingDB('<db_name>').getCollection('<collection_name>').dropIndex('<inde
 
 If you're hosting MongoDB on [Atlas](https://www.mongodb.com/atlas/database), use the following [Manage Indexes](https://www.mongodb.com/docs/atlas/atlas-ui/indexes/) guide to View, Create or Remove indexes.
 
-> ⚠️ **Important!** In Atlas, for production environments, it is recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v6.0/tutorial/build-indexes-on-replica-sets/))
+> [!IMPORTANT]
+> In Atlas, for production environments, it may be recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v7.0/tutorial/build-indexes-on-replica-sets/))
 
 ## Upgrading
 
@@ -2374,6 +2389,15 @@ For built-in RabbitMQ `bitnami/rabbitmq` subchart, pre-upgrade hook was added to
 
 - `.Values.runner` is removed
 
+#### Changes in MongoDB schema
+
+Changes in indexes: follow [Maintaining MongoDB indexes](#maintaining-mongodb-indexes) guide to meet index requirements *before* the upgrade process.
+
+Changes in collections: following collections can be safely dropped *after* the upgrade to 2.9.x if they exist. These collections are no longer used and should be removed to maintain optimal database performance and prevent the accumulation of obsolete data.
+
+- `read-models.application-tree`
+- `read-models.<entity>-history` — every collection with `~-history` suffix, such as `read-models.applications-history`, `read-models.services-history`, etc.
+
 ## Troubleshooting
 
 ### Error: Failed to validate connection to Docker daemon; caused by Error: certificate has expired

README.md.gotmpl

@@ -1499,7 +1499,8 @@ Use [obtain-oidc-id-token](https://github.com/codefresh-io/steps/blob/822afc0a9a
 
 Sometimes, in new releases of Codefresh On-Prem, index requirements change. When this happens, it's mentioned in the [Upgrading section](#upgrading) for the specific release.
 
-> ℹ️ If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
+> [!TIP]
+> If you're upgrading from version `X` to version `Y`, and index requirements were updated in any of the intermediate versions, you only need to align your indexes with the index requirements of version `Y`. To do that, follow [Index alignment](#index-alignment) instructions.
 
 ### Index alignment
 
@@ -1508,10 +1509,9 @@ The required index definitions for each release can be found at the following re
 - `2.6` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.6/indexes>
 - `2.7` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.7/indexes>
 - `2.8` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.8/indexes>
+- `2.9` <https://github.com/codefresh-io/codefresh-onprem-helm/tree/release-2.9/indexes>
 
-The indexes are stored in JSON files with keys and options specified.
-
-The directory structure is:
+The indexes specifications are stored in JSON files. The directory structure is:
 
 ```console
 indexes
@@ -1522,11 +1522,12 @@ indexes
 **Overview of the index alignment process:**
 
 1. Identify the differences between the indexes in your MongoDB instance and the required index definitions.
-2. Create any missing indexes one by one. (It's important not to create them in bulk.)
+2. Create any missing indexes.
 3. Perform the upgrade of Codefresh On-Prem installation.
 4. Then remove any unnecessary indexes.
 
-> ⚠️ **Note! Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
+> [!IMPORTANT]
+> Any changes to indexes should be performed during a defined maintenance window or during periods of lowest traffic to MongoDB.**
 >
 > Building indexes during time periods where the target collection is under heavy write load can result in reduced write performance and longer index builds. ([*Source: MongoDB official documentation*](https://www.mongodb.com/docs/manual/core/index-creation/#index-build-impact-on-database-performance))
 >
@@ -1549,19 +1550,32 @@ mongosh "<connection_string>"
 db.getSiblingDB('<db_name>').getCollection('<collection_name>').getIndexes()
 ```
 
-- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones
+- Compare your indexes with the required indexes for the target release, and adjust them by creating any missing indexes or removing any unnecessary ones.
 
 **Index creation**
 
-> ⚠ **Note! Always create indexes sequentially, one by one. Don't create them in bulk.**
+- To create an indexes, we recommend using the `createIndexes` command ([ref](https://www.mongodb.com/docs/manual/reference/command/createIndexes/)):
 
-- To create an index, use the `createIndex()` method:
+> [!IMPORTANT]
+> We recommend to create indexes in batches of 3 indexes at a time.
+> However, it's highly recommended before creating indexes in production DB to test performance impact on a staging instance with prod-like amount of data.
+>
+> Previous command should be completed before starting the next batch.
 
 ```js
-db.getSiblingDB('<db_name>').getCollection('<collection_name>').createIndex(<keys_object>, <options_object>)
+db.getSiblingDB('<db_name>').runCommand(
+  {
+    createIndexes: '<collection_name>',
+    indexes: [
+        { ... },  // Index definition from the doc above
+        { ... },  // Index definition from the doc above
+        { ... }   // Index definition from the doc above
+    ],
+  }
+)
 ```
 
-After executing the `createIndex()` command, you should see a result indicating that the index was created successfully.
+After executing the command, you should see a result indicating that the indexes were created successfully.
 
 **Index removal**
 
@@ -1575,7 +1589,8 @@ db.getSiblingDB('<db_name>').getCollection('<collection_name>').dropIndex('<inde
 
 If you're hosting MongoDB on [Atlas](https://www.mongodb.com/atlas/database), use the following [Manage Indexes](https://www.mongodb.com/docs/atlas/atlas-ui/indexes/) guide to View, Create or Remove indexes.
 
-> ⚠️ **Important!** In Atlas, for production environments, it is recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v6.0/tutorial/build-indexes-on-replica-sets/))
+> [!IMPORTANT]
+> In Atlas, for production environments, it may be recommended to use rolling index builds by enabling the "Build index via rolling process" checkbox. ([*MongoDB official documentation*](https://www.mongodb.com/docs/v7.0/tutorial/build-indexes-on-replica-sets/))
 
 ## Upgrading
 
@@ -2384,6 +2399,15 @@ For built-in RabbitMQ `bitnami/rabbitmq` subchart, pre-upgrade hook was added to
 
 - `.Values.runner` is removed
 
+#### Changes in MongoDB schema
+
+Changes in indexes: follow [Maintaining MongoDB indexes](#maintaining-mongodb-indexes) guide to meet index requirements *before* the upgrade process.
+
+Changes in collections: following collections can be safely dropped *after* the upgrade to 2.9.x if they exist. These collections are no longer used and should be removed to maintain optimal database performance and prevent the accumulation of obsolete data.
+
+- `read-models.application-tree`
+- `read-models.<entity>-history` — every collection with `~-history` suffix, such as `read-models.applications-history`, `read-models.services-history`, etc.
+
 
 ## Troubleshooting
 

indexes/codefresh/agenttasks.json

@@ -1,6 +1,6 @@
 [
     {
-        "keys": {
+        "key": {
             "metadata.accountId": 1,
             "metadata.reIdentifier": 1,
             "metadata.shouldExecute": 1,
@@ -10,11 +10,9 @@
         }
     },
     {
-        "keys": {
+        "key": {
             "metadata.expireAt": 1
         },
-        "options": {
-            "expireAfterSeconds": 0
-        }
+        "expireAfterSeconds": 0
     }
-]
+]

indexes/codefresh/feature-store-versioned.json

@@ -1,16 +1,14 @@
 [
     {
-        "keys": {
+        "key": {
             "createdAt": 1
         },
-        "options": {
-            "expireAfterSeconds": 43200
-        }
+        "expireAfterSeconds": 43200
     },
     {
-        "keys": {
+        "key": {
             "_id": -1,
             "LDRedisStoreVersion": 1
         }
     }
-]
+]

indexes/codefresh/users.json

@@ -1,15 +1,15 @@
 [
     {
-        "keys": {
+        "key": {
             "account": 1,
             "_id": 1
         }
     },
     {
-        "keys": {
+        "key": {
             "role": 1,
             "account": 1,
             "_id": 1
         }
     }
-]
+]