Renaming DWH - leftovers (#23558)

* Renamed everything that kind of made sense. Let's see if it still works.

* Cleaning up; added changelog

* Adding details about the data lake renaming

---------

Co-authored-by: Maxwell <[email protected]>

Author: fpetersen-gl

UPGRADING.md

@@ -33,11 +33,44 @@ The logic for generating color shades based on custom-defined color variants (er
 has been slightly adjusted. This change ensures that the exact color specified in the customization settings
 is now used as the primary color for elements like buttons and badges in the UI.
 
+### Renaming "Data Warehouse" to "Data Lake"
+The feature previously known as "Data Warehouse" is now completely renamed to "Data Lake". That includes not only text
+visible to the user, but with version 7.0 also a lot of places being usually invisible to the regular user, such as API
+endpoints, the content of the database, permissions and much more.
+
+#### API Endpoints and permissions
+All endpoints related to data lake have changed their URLs accordingly. As also noted in
+the [REST API endpoint Changes](#rest-api-endpoint-changes), all endpoints previously accessible at
+`/api/plugins/org.graylog.plugins.datawarehouse/data_warehouse/...` are now moved to
+`/api/plugins/org.graylog.plugins.datalake/data_lake/...`. Similarly, all permissions regarding data lake are renamed
+from `data_warehouse...` to `data_lake...`.
+
+#### Database content
+Three entire collections are renamed: 
+- `data_warehouse_archive_config` to `data_lake_archive_config`
+- `data_warehouse_backends` to `data_lake_backends`
+- `data_warehouse_catalog` to `data_lake_catalog`
+
+Also, documents of the following collections are updated to reflect the change in names:
+- `cluster_config` for migrations related to data lake
+- `enterprise_traffic`
+- `scheduler_job_definitions`
+- `scheduler_triggers`, in case a data lake optimization job is already scheduled.
+
+#### Audit logs
+Audit logs having been written before the update are not changed. However, all audit logs from after the update contain
+the term "Data Lake" instead of "Data Warehouse".
+
+#### Metrics
+Just like audit logs, metrics from before the update to version 7.0 are not changed. Starting with version 7.0, the
+names of data lake related metrics change accordingly.
+
+
 ## Configuration File Changes
 
-| Option        | Action     | Description                                    |
-|---------------|------------|------------------------------------------------|
-| `tbd`         | **added**  |                                                |
+| Option | Action    | Description |
+|--------|-----------|-------------|
+| `tbd`  | **added** |             |
 
 ## Default Configuration Changes
 
@@ -55,52 +88,54 @@ is now used as the primary color for elements like buttons and badges in the UI.
 
 ## General REST API Changes
 
-- In Graylog 7.0, an issue was fixed that previously allowed additional unknown JSON properties to be accepted 
-  (and ignored) in API requests on the Graylog leader node. Now that the issue has been fixed, API requests on the 
+- In Graylog 7.0, an issue was fixed that previously allowed additional unknown JSON properties to be accepted
+  (and ignored) in API requests on the Graylog leader node. Now that the issue has been fixed, API requests on the
   leader node will once again only accept JSON payloads that contain explicitly mapped/supported properties.
-- APIs for entity creation now use a parameter `CreateEntityRequest` to keep entity fields separated from sharing 
+- APIs for entity creation now use a parameter `CreateEntityRequest` to keep entity fields separated from sharing
   information. This is a breaking change for all API requests that create entities, such as streams, dashboards, etc.
-  <br> Affected entities: 
-  - Search / Dashboard 
-  - Search Filter 
-  - Report
-  - Event Definition
-  - Stream
-  - Notifications
-  - Sigma rules
-  - Event procedure
-  - Event step
-  - Content Pack installation
+  <br> Affected entities:
+    - Search / Dashboard
+    - Search Filter
+    - Report
+    - Event Definition
+    - Stream
+    - Notifications
+    - Sigma rules
+    - Event procedure
+    - Event step
+    - Content Pack installation
   - Teams
-  
+
   <br> For example, the request payload to create a stream might now look like this:
 
 ```json
 {
-    "entity":{
-        "index_set_id":"65b7ba138cdb8c534a953fef",
-        "description":"An example stream",
-        "title":"My Stream",
-        "remove_matches_from_default_stream":false
-    },
-    "share_request":{
-        "selected_grantee_capabilities":{
-            "grn::::search:684158906442150b2eefb78c":"own"
-        }
+  "entity": {
+    "index_set_id": "65b7ba138cdb8c534a953fef",
+    "description": "An example stream",
+    "title": "My Stream",
+    "remove_matches_from_default_stream": false
+  },
+  "share_request": {
+    "selected_grantee_capabilities": {
+      "grn::::search:684158906442150b2eefb78c": "own"
     }
+  }
 }
 ```
-- Access to the API browser now requires the `api_browser:read` permission. This permission can be granted by assigning 
+
+- Access to the API browser now requires the `api_browser:read` permission. This permission can be granted by assigning
   the new “API Browser Reader” role to a user.
 
 ## REST API Endpoint Changes
 
 The following REST API changes have been made.
 
-| Endpoint                          | Description                                                                                                                                                                                |
-|-----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `GET /system/urlallowlist`        | Renamed from `GET /system/urlwhitelist`. The corresponding REST API permission is renamed to `urlallowlist:read`.                                                                          |
-| `PUT /system/urlallowlist`        | Renamed from `PUT /system/urlwhitelist`                                                                         . The corresponding REST API permission is renamed to `urlallowlist:write` |
-| `POST /system/urlallowlist/check` | Renamed from `POST /system/urlwhitelist/check`                                                                                                                                             |
-| `POST /system/urlallowlist/generate_regex` | Renamed from `POST /system/urlwhitelist/generate_regex`                                                                                                                                    |
-| `GET /<endpoint>`                 | description                                                                                                                                                                                |
+| Endpoint                                                      | Description                                                                                                                                        |
+|---------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
+| `GET /system/urlallowlist`                                    | Renamed from `GET /system/urlwhitelist`. The corresponding REST API permission is renamed to `urlallowlist:read`.                                  |
+| `PUT /system/urlallowlist`                                    | Renamed from `PUT /system/urlwhitelist`. The corresponding REST API permission is renamed to `urlallowlist:write`                                  |
+| `POST /system/urlallowlist/check`                             | Renamed from `POST /system/urlwhitelist/check`                                                                                                     |
+| `POST /system/urlallowlist/generate_regex`                    | Renamed from `POST /system/urlwhitelist/generate_regex`                                                                                            |
+| All `/api/plugins/org.graylog.plugins.datalake/data_lake/...` | Renamed from `/api/plugins/org.graylog.plugins.datawarehouse/data_warehouse/...`. The corresponding permissions are also renamed to `data_lake...` |
+| `GET /<endpoint>`                                             | description                                                                                                                                        |

api-specs/stream-output-filters.yml

@@ -282,7 +282,7 @@ components:
           description: "The destination type"
           enum:
             - "indexer"
-            - "data-warehouse"
+            - "data-lake"
         title:
           type: "string"
           description: "The filter title"

changelog/unreleased/pr-23558.toml

@@ -0,0 +1,5 @@
+type = "c"
+message = "Renamed all remaining occurrences of 'Data Warehouse' to 'Data Lake'."
+
+issues = ["Graylog2/graylog-plugin-enterprise#9752"]
+pulls = ["Graylog2/graylog-plugin-enterprise#1804", "23558", "Graylog2/e2e-tests#1285"]

graylog2-server/src/main/java/org/graylog/plugins/views/search/searchtypes/DataLakeSearchType.java

@@ -23,5 +23,5 @@
  */
 public interface DataLakeSearchType extends SearchType {
 
-    String PREFIX = "data_warehouse_";
+    String PREFIX = "data_lake_";
 }

graylog2-server/src/main/resources/org/graylog2/featureflag/feature-flag.config

@@ -86,7 +86,7 @@ configurable_value_units=on
 remote_reindex_migration=off
 
 # Enable preview of Graylog-managed Data Lake data
-data_warehouse_search=on
+data_lake_search=on
 
 # Enable preview of external Data Lake data (i.e. AWS)
 external_data_lake_search=on

graylog2-server/src/main/resources/prometheus-exporter.yml

@@ -1265,96 +1265,96 @@ metric_mappings:
     wildcard_extract_labels:
       - "type"
 
-  - metric_name: "data_warehouse_s3_client_head_object"
-    match_pattern: "org.graylog.plugins.datawarehouse.s3client.*.HeadObject"
+  - metric_name: "data_lake_s3_client_head_object"
+    match_pattern: "org.graylog.plugins.datalake.s3client.*.HeadObject"
     wildcard_extract_labels:
       - "head_object"
 
-  - metric_name: "data_warehouse_s3_client_get_object"
-    match_pattern: "org.graylog.plugins.datawarehouse.s3client.*.GetObject"
+  - metric_name: "data_lake_s3_client_get_object"
+    match_pattern: "org.graylog.plugins.datalake.s3client.*.GetObject"
     wildcard_extract_labels:
       - "get_object"
 
-  - metric_name: "data_warehouse_s3_client_put_object"
-    match_pattern: "org.graylog.plugins.datawarehouse.s3client.*.PutObject"
+  - metric_name: "data_lake_s3_client_put_object"
+    match_pattern: "org.graylog.plugins.datalake.s3client.*.PutObject"
     wildcard_extract_labels:
       - "put_object"
 
-  - metric_name: "data_warehouse_s3_client_delete_object"
-    match_pattern: "org.graylog.plugins.datawarehouse.s3client.*.DeleteObject"
+  - metric_name: "data_lake_s3_client_delete_object"
+    match_pattern: "org.graylog.plugins.datalake.s3client.*.DeleteObject"
     wildcard_extract_labels:
       - "delete_object"
 
-  - metric_name: "data_warehouse_iceberg_file_io_s3_read"
+  - metric_name: "data_lake_iceberg_file_io_s3_read"
     match_pattern: "org.graylog.iceberg.file.io.s3.read.*"
     wildcard_extract_labels:
       - "read"
 
-  - metric_name: "data_warehouse_iceberg_file_io_s3_write"
+  - metric_name: "data_lake_iceberg_file_io_s3_write"
     match_pattern: "org.graylog.iceberg.file.io.s3.write.*"
     wildcard_extract_labels:
       - "write"
 
-  - metric_name: "data_warehouse_iceberg_file_io_local_write"
+  - metric_name: "data_lake_iceberg_file_io_local_write"
     match_pattern: "org.graylog.iceberg.file.io.local.write"
 
-  - metric_name: "data_warehouse_iceberg_file_io_local_read"
+  - metric_name: "data_lake_iceberg_file_io_local_read"
     match_pattern: "org.graylog.iceberg.file.io.local.read"
 
-  - metric_name: "data_warehouse_iceberg_file_io_gcs_write"
+  - metric_name: "data_lake_iceberg_file_io_gcs_write"
     match_pattern: "org.graylog.iceberg.file.io.gcs.write.*"
     wildcard_extract_labels:
       - "write"
 
-  - metric_name: "data_warehouse_iceberg_file_io_gcs_read"
+  - metric_name: "data_lake_iceberg_file_io_gcs_read"
     match_pattern: "org.graylog.iceberg.file.io.gcs.read.*"
     wildcard_extract_labels:
       - "read"
 
-  - metric_name: "data_warehouse_journal_writer_written_messages"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.DataWarehouseJournalWriter.written-messages"
+  - metric_name: "data_lake_journal_writer_written_messages"
+    match_pattern: "org.graylog.plugins.datalake.output.DataLakeJournalWriter.written-messages"
 
-  - metric_name: "data_warehouse_journal_writer_written_bytes"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.DataWarehouseJournalWriter.written-bytes"
+  - metric_name: "data_lake_journal_writer_written_bytes"
+    match_pattern: "org.graylog.plugins.datalake.output.DataLakeJournalWriter.written-bytes"
 
-  - metric_name: "data_warehouse_journal_writer_written_failed_write_attempts"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.DataWarehouseJournalWriter.failed-write-attempts"
+  - metric_name: "data_lake_journal_writer_written_failed_write_attempts"
+    match_pattern: "org.graylog.plugins.datalake.output.DataLakeJournalWriter.failed-write-attempts"
 
-  - metric_name: "data_warehouse_journal_last_flush_time"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.lastFlushTime"
+  - metric_name: "data_lake_journal_last_flush_time"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.lastFlushTime"
 
-  - metric_name: "data_warehouse_journal_log_end_offset"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.logEndOffset"
+  - metric_name: "data_lake_journal_log_end_offset"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.logEndOffset"
 
-  - metric_name: "data_warehouse_journal_number_of_segments"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.numberOfSegments"
+  - metric_name: "data_lake_journal_number_of_segments"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.numberOfSegments"
 
-  - metric_name: "data_warehouse_journal_read_messages"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.readMessages"
+  - metric_name: "data_lake_journal_read_messages"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.readMessages"
 
-  - metric_name: "data_warehouse_journal_read_time"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.readTime"
+  - metric_name: "data_lake_journal_read_time"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.readTime"
 
-  - metric_name: "data_warehouse_journal_recovery_point"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.recoveryPoint"
+  - metric_name: "data_lake_journal_recovery_point"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.recoveryPoint"
 
-  - metric_name: "data_warehouse_journal_size"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.size"
+  - metric_name: "data_lake_journal_size"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.size"
 
-  - metric_name: "data_warehouse_journal_uncommitted_messages"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.uncommittedMessages"
+  - metric_name: "data_lake_journal_uncommitted_messages"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.uncommittedMessages"
 
-  - metric_name: "data_warehouse_journal_unflushed_messages"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.unflushedMessages"
+  - metric_name: "data_lake_journal_unflushed_messages"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.unflushedMessages"
 
-  - metric_name: "data_warehouse_journal_write_discarded_messages"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.writeDiscardedMessages"
+  - metric_name: "data_lake_journal_write_discarded_messages"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.writeDiscardedMessages"
 
-  - metric_name: "data_warehouse_journal_write_time"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.writeTime"
+  - metric_name: "data_lake_journal_write_time"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.writeTime"
 
-  - metric_name: "data_warehouse_journal_written_messages"
-    match_pattern: "org.graylog.plugins.datawarehouse.output.journal.writtenMessages"
+  - metric_name: "data_lake_journal_written_messages"
+    match_pattern: "org.graylog.plugins.datalake.output.journal.writtenMessages"
 
   - metric_name: "shared_thread_pools_daemon_scheduler_executor"
     match_pattern: "org.graylog2.shared-thread-pools.scheduled-daemon-executor.*"

graylog2-server/src/test/java/org/graylog2/telemetry/rest/TelemetryTestHelper.java

@@ -34,8 +34,8 @@ public class TelemetryTestHelper {
     public static final String LICENSE = "license";
     static final String PLUGIN = "plugin";
     static final String SEARCH_CLUSTER = "search_cluster";
-    public static final String DATA_WAREHOUSE_ACCOUNTED_OUTPUT = "data_warehouse_accounted_output_traffic_last_month";
-    public static final String DATA_WAREHOUSE_NOT_ACCOUNTED_OUTPUT = "data_warehouse_not_accounted_output_traffic_last_month";
+    public static final String DATA_LAKE_ACCOUNTED_OUTPUT = "data_lake_accounted_output_traffic_last_month";
+    public static final String DATA_LAKE_NOT_ACCOUNTED_OUTPUT = "data_lake_not_accounted_output_traffic_last_month";
 
     static final String DATA_NODES = "data_nodes";
 

graylog2-web-interface/src/logic/telemetry/Constants.ts

@@ -188,10 +188,10 @@ export const TELEMETRY_EVENT_TYPE = {
     STREAM_ITEM_DATA_ROUTING_STREAM_INDEXSET_UPDATE_OPENED: 'Stream Item Data Routing Stream IndexSet Update Opened',
     STREAM_ITEM_DATA_ROUTING_INDEXER_FILTER_UPDATE_OPENED: 'Stream Item Data Routing Indexer Filter Update Opened',
     STREAM_ITEM_DATA_ROUTING_INDEXER_FILTER_CREATE_OPENED: 'Stream Item Data Routing Indexer Filter Create Opened',
-    STREAM_ITEM_DATA_ROUTING_DATA_WAREHOUSE_FILTER_CREATE_OPENED:
-      'Stream Item Data Routing Data Warehouse Filter Create Opened',
-    STREAM_ITEM_DATA_ROUTING_DATA_WAREHOUSE_FILTER_UPDATE_OPENED:
-      'Stream Item Data Routing Data Warehouse Filter Update Opened',
+    STREAM_ITEM_DATA_ROUTING_DATA_LAKE_FILTER_CREATE_OPENED:
+      'Stream Item Data Routing Data Lake Filter Create Opened',
+    STREAM_ITEM_DATA_ROUTING_DATA_LAKE_FILTER_UPDATE_OPENED:
+      'Stream Item Data Routing Data Lake Filter Update Opened',
     STREAM_ITEM_DATA_ROUTING_FILTER_DELETE_OPENED: 'Stream Item Data Routing Filter Delete Opened',
     STREAM_ITEM_DATA_ROUTING_INTAKE_OPENED: 'Stream Item Data Routing Intake Opened',
     STREAM_ITEM_DATA_ROUTING_PROCESSING_OPENED: 'Stream Item Data Routing Processing Opened',