refactor: from ElasticSearch to OpenSearch

Author: fstagni

.github/workflows/integration.yml

@@ -31,8 +31,8 @@ jobs:
       matrix:
         # TEST_NAME is a dummy variable used to make it easier to read the web interface
         include:
-          - TEST_NAME: "MariaDB 11.4, elasticsearch:7.9.1"
-            ARGS: MYSQL_VER=mariadb:11.4.3 ES_VER=elasticsearch:7.9.1
+          - TEST_NAME: "MariaDB 11.4"
+            ARGS: MYSQL_VER=mariadb:11.4.3
           - TEST_NAME: "HTTPS"
             ARGS: TEST_HTTPS=Yes
           - TEST_NAME: "Force DEncode"
@@ -72,8 +72,8 @@ jobs:
       run: ./integration_tests.py test-client || touch client-tests-failed
     - name: Pilot tests
       run: ./integration_tests.py test-pilot || touch pilot-tests-failed
-    - name: Elasticsearch logs
-      run: docker logs elasticsearch
+    - name: Opensearch logs
+      run: docker logs opensearch
     - name: Check test status
       run: |
         has_error=0

dashboards/GrafanaDemo/README.md

@@ -6,12 +6,12 @@ A running example of this dashboard can be found on the LHCb grafana organisatio
 - Production Jobs By Final Minor Status (MySQL)
 - User Jobs By Final Minor Status (MySQL)
 - Job CPU Efficiency By Job Type (MySQL)
-- User Jobs By Job Type (ElasticSearch)
+- User Jobs By Job Type (OpenSearch)
 - User Jobs By Final Minor Status (MySQL)
 - Pilots By Status (MySQL)
 
 ## How to import this dashboard into another Grafana installation?
-- Create Grafana data sources for the MySQL database and ElasticSearch cluster (both the DIRAC certification versions).
+- Create Grafana data sources for the MySQL database and OpenSearch cluster (both the DIRAC certification versions).
 - Create a new dashboard and in 'Dashboard settings' under 'JSON model' replace the JSON representation of the dashboard with this one.
 - In the JSON file, replace the datasource uid's with the ones for the data sources in your grafana installation. You can find the UID of a data source:
     - with the [API](https://grafana.com/docs/grafana/latest/developers/http_api/data_source/)

docs/source/AdministratorGuide/ExternalsSupport/index.rst

@@ -38,17 +38,13 @@ generic connection details can be applied in CS location below (the shown values
    }
 
 
-ElasticSearch versions
-----------------------
+OpenSearch versions
+-------------------
 
-ElasticSearch is an optional dependency for DIRAC servers installations. Supported versions:
+OpenSearch is a non-optional dependency for DIRAC servers installations.
+OpenSearch server is not shipped with DIRAC. You are responsible of its administration.
 
-- 7.x up until 7.13
-- OpenDistro and OpenSearch releases "compatible" with the above ElasticSearch versions.
-
-ElasticSearch server is not shipped with DIRAC. You are responsible of its administration.
-
-You can run your ES cluster without authentication, or using User name and password, or using certificates. You may add the following parameters:
+You can run your OpenSearch cluster without authentication, or using User name and password, or using certificates. You may add the following parameters:
 
   - ``User`` (default:``''``)
   - ``Password`` (default:``''``)

docs/source/AdministratorGuide/ServerInstallations/centralizedLogging.rst

@@ -62,7 +62,7 @@ Logstash and ELK configurations
 
 The suggested logstash configuration (``/etc/logstash/conf.d/configname``) can be found in https://gitlab.cern.ch/ai/it-puppet-module-dirac/-/blob/qa/code/templates/logstash.conf.erb (check the `full documentation <https://opensearch.org/docs/latest/clients/logstash/index/>`_)
 
-The ElasticSearch template ``lhcb-dirac-logs_default`` looks like::
+The OpenSearch template ``lhcb-dirac-logs_default`` looks like::
 
   {
     "order": 1,

docs/source/AdministratorGuide/ServerInstallations/tornadoComponentsLogs.rst

@@ -158,7 +158,7 @@ dirac-json.conf (Add all needed components and choose the output you want)::
   [SERVICE] where we describe our json parser (from dirac Json log backend)
   [INPUT] where we describe dirac components log file and the way it will be parsed (json)
   [FILTER] where we apply modifications to parsed data, for example adding a levelname "DEV" whenever logs are not well formatted, typically "print" in code, or adding fields like hostname to know from which host logs are coming, but also more complex treatments like in dirac.lua script (described later)
-  [OUTPUT] where we describe formatted logs destination, here, we have stdout, files on disks and elasticsearch.
+  [OUTPUT] where we describe formatted logs destination, here, we have stdout, files on disks and opensearch.
 
 dirac-parsers.conf::
 
@@ -203,19 +203,19 @@ dirac.lua::
 Testing
 -------
 
-Before throwing logs to ElasticSearch, config can be tested in Standard output by uncommenting::
+Before throwing logs to OpenSearch, config can be tested in Standard output by uncommenting::
 
   [OUTPUT]
       name             stdout
       match            *
 
-...and commenting ElasticSearch outputs.
+...and commenting OpenSearch outputs.
 
 Then by using command::
 
   /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf
 
-NOTE: When all is OK, uncomment ElasticSearch outputs and comment stdout output
+NOTE: When all is OK, uncomment OpenSearch outputs and comment stdout output
 
 Service
 -------

docs/source/AdministratorGuide/Systems/Framework/Monitoring/index.rst

@@ -6,4 +6,4 @@ The Framework/monitoring service
 
 The framework system for the monitoring of services and agents has been removed and will no longer be used.
 
-It has been replaced by an ElasticSearch-based monitoring system. You can read about it in :ref:`Monitoring <monitoring_system>`
+It has been replaced by an OpenSearch-based monitoring system. You can read about it in :ref:`Monitoring <monitoring_system>`

docs/source/AdministratorGuide/Systems/MonitoringSystem/index.rst

@@ -20,19 +20,18 @@ The Monitoring system is used to monitor various components of DIRAC. Currently,
   - PilotSubmission Monitoring: for monitoring the DIRAC pilot submission statistics from SiteDirector agents.
   - DataOperation Monitoring: for monitoring the DIRAC data operation statistics as well as individual failures from interactive use of ``StorageElement``.
 
-It is based on Elasticsearch (OpenSearch) distributed search and analytics NoSQL database.
-If you want to use it, you have to install the Monitoring service, and of course connect to a ElasticSearch instance.
+It is based on OpenSearch distributed search and analytics NoSQL database.
+If you want to use it, you have to install the Monitoring service, and of course connect to a OpenSearch instance.
 
-Install Elasticsearch/OpenSearch
-================================
+Install OpenSearch
+==================
 
-This is not covered here, as installation and administration of ES are not part of DIRAC guide.
-Just a note on the ES versions supported: only ES7+ versions are currently supported, and are later to be replaced by OpenSearch services.
+This is not covered here.
 
 Configure the MonitoringSystem
 ===============================
 
-You can run your Elastic/OpenSearch cluster even without authentication, or using User name and password. You have to add the following parameters:
+You can run your OpenSearch cluster even without authentication, or using User name and password. You have to add the following parameters:
 
   - User
   - Password
@@ -82,7 +81,7 @@ The given periods above are also the default periods in the code.
 Enable the Monitoring System
 ============================
 
-In order to enable the monitoring of all the following types with an ElasticSearch-based backend, you should add the value `Monitoring` to the flag
+In order to enable the monitoring of all the following types with an OpenSearch-based backend, you should add the value `Monitoring` to the flag
 `MonitoringBackends/Default` in the Operations section of the CS.
 If you want to override this flag for a specific type, say, you want to only have Monitoring (and no Accounting) for WMSHistory, you just create a flag `WMSHistory` set to `Monitoring`.
 If, for example, you want both Monitoring and Accounting for WMSHistory (but not for other types), you set `WMSHistory = Accounting, Monitoring`. If no flag is set for `WMSHistory`, the `Default` flag will be used.
@@ -111,7 +110,7 @@ This can be done either via the CS or directly in the web app in the Configurati
 WMSHistory & PilotsHistory Monitoring
 =====================================
 
-The WorkloadManagement/StatesAccountingAgent creates, every 15 minutes, a snapshot with the contents of JobDB and PilotAgentsDB and sends it to an Elasticsearch-based database.
+The WorkloadManagement/StatesAccountingAgent creates, every 15 minutes, a snapshot with the contents of JobDB and PilotAgentsDB and sends it to an OpenSearch-based database.
 This same agent can also report the WMSHistory to the MySQL backend used by the Accounting system (which is in fact the default).
 
 Optionally, you can use an MQ system (like ActiveMQ) for failover, even though the agent already has a simple failover mechanism.
@@ -169,12 +168,12 @@ Accessing the Monitoring information
 
 After you installed and configured the Monitoring system, you can use the Monitoring web application for the types WMSHistory and RMS.
 
-However, every type can directly be monitored in Kibana dashboards that can be imported into your Elasticsearch (or Opensearch) instance. You can find and import these dashboards from DIRAC/dashboards as per the following example.
+However, every type can directly be monitored in Kibana dashboards that can be imported into your Opensearch instance. You can find and import these dashboards from DIRAC/dashboards as per the following example.
 Grafana dashboards are also provided for some of the types.
 
 *Kibana dashboard for WMSHistory*
   A dashboard for WMSHistory monitoring ``WMSDashboard`` is available `here <https://github.com/DIRACGrid/DIRAC/tree/integration/dashboards/WMS>`__ for import as a NDJSON (as support for JSON is being removed in the latest versions of Kibana).
-  The dashboard may not be compatible with older versions of ElasticSearch.
+  The dashboard may not be compatible with older versions of OpenSearch.
   To import it in the Kibana UI, go to Management -> Saved Objects -> Import and import the JSON file.
 
   Note: the JSON file already contains the index patterns needed for the visualizations. You may need to adapt the index patterns to your existing ones.

docs/source/AdministratorGuide/Systems/WorkloadManagement/architecture.rst

@@ -74,12 +74,8 @@ All these agents are necessary for the WMS, and each of them should be installed
 You can duplicate some of these agents as long as you provide the correct configuration.
 A typical example is the SiteDirector, for which you may want to deploy even 1 for each of the sites managed.
 
-Optional agents are:
-
-StatesAccountingAgent or StatesMonitoringAgent
-  Use one or the other.
-  StatesMonitoringAgent is used for producing Monitoring plots through the :ref:`Monitoring System <monitoring_system>`. (so, using ElasticSearch as backend),
-  while StatesAccountingAgent does the same job but using the Accounting system (so, MySQL as backend).
+StatesAccountingAgent.
+  Used for producing Monitoring plots through the :ref:`Monitoring System <monitoring_system>`, or using the Accounting system (so, MySQL as backend).
 
 A very different type of agent is the *JobAgent*, which is run by the pilot jobs and should NOT be run in a server installation.
 

docs/source/DeveloperGuide/AddingNewComponents/DevelopingDatabases/index.rst

@@ -3,7 +3,7 @@ Developing Databases
 ====================
 
 This is a quick guide about developing classes interacting with MySQL databases.
-DIRAC supports also Elasticsearch/OpenSearch NoSQL database, but it is not part of this document.
+DIRAC supports also OpenSearch NoSQL database, but it is not part of this document.
 
 Before starting developing databases, you have to make sure that MySQL is installed, as well as python-mysql,
 as explained in :ref:`editing_code`, and make sure that MySQL service is on.

docs/source/DeveloperGuide/AddingNewComponents/Utilities/gLogger/Backends/index.rst

@@ -53,28 +53,28 @@ Parameters
 +-----------+------------------------------------------------------------------+----------------------+
 
 
-ElasticSearchBackend
+OpenSearch Backend
 --------------------
 
 Description
 ~~~~~~~~~~~
-Used to emit log records in the an ElasticSearch database.
+Used to emit log records in the an OpenSearch database.
 The *Backend* acccepts logs from *Debug* to *Always* level.
 
 Parameters
 ~~~~~~~~~~
 +-----------+------------------------------------------------------------------+----------------------+
 | Option    | Description                                                      | Default value        |
 +===========+==================================================================+======================+
-| Host      | host machine where the ElasticSearch DB is installed             | ''                   |
+| Host      | host machine where the OpenSearch DB is installed                | ''                   |
 +-----------+------------------------------------------------------------------+----------------------+
-| Port      | port where the ElasticSearch DB listen                           | 9203                 |
+| Port      | port where the OpenSearch DB listen                              | 9203                 |
 +-----------+------------------------------------------------------------------+----------------------+
-| User      | username of the ElasticSearch DB   (optional)                    | None                 |
+| User      | username of the OpenSearch DB   (optional)                       | None                 |
 +-----------+------------------------------------------------------------------+----------------------+
-| Password  | password of the ElasticSearch DB   (optional)                    | None                 |
+| Password  | password of the OpenSearch DB   (optional)                       | None                 |
 +-----------+------------------------------------------------------------------+----------------------+
-| Index     | ElasticSearch index                                              | ''                   |
+| Index     | OpenSearch index                                                 | ''                   |
 +-----------+------------------------------------------------------------------+----------------------+
 | BufferSize| maximum size of the buffer before sending                        | 1000                 |
 +-----------+------------------------------------------------------------------+----------------------+