Merge pull request #5965 from rupozzi/gmonitor-branch

[integration] Removing gMonitor

Author: fstagni

docs/source/AdministratorGuide/Configuration/ConfReference/Tips/ServicesPorts/index.rst

@@ -27,8 +27,6 @@ Ordered by System / Services
 +------+----------------------+---------------------+
 | 9158 | *Framework*          | BundleDelivery      |
 +------+----------------------+---------------------+
-| 9142 | *Framework*          | Monitoring          |
-+------+----------------------+---------------------+
 | 9154 | *Framework*          | Notification        |
 +------+----------------------+---------------------+
 | 9157 | *Framework*          | Plotting            |
@@ -74,8 +72,6 @@ Ordered by port number
 +------+----------------------+---------------------+
 | 9136 | *WorkloadManagement* | JobStateUpdate      |
 +------+----------------------+---------------------+
-| 9142 | *Framework*          | Monitoring          |
-+------+----------------------+---------------------+
 | 9143 | *RequestManagement*  | RequestManager      |
 +------+----------------------+---------------------+
 | 9145 | *WorkloadManagement* | WMSAdministrator    |

docs/source/AdministratorGuide/HowTo/SystemAdministratorInterface.rst

@@ -94,8 +94,7 @@ Show setup command allows administrators to know which components, Services and
      'Services': {'Accounting': ['ReportGenerator',
                                  'DataStore'],
                   'Configuration': ['Server'],
-                  'Framework': ['Monitoring',
-                                'BundleDelivery',
+                  'Framework': ['BundleDelivery',
                                 'SecurityLogging',
                                 'Notification',
                                 'UserProfileManager',

docs/source/AdministratorGuide/Systems/Framework/Monitoring/index.rst

@@ -4,15 +4,6 @@
 The Framework/monitoring service
 ================================
 
-The Framework/Monitoring service collects information from all the active DIRAC services and Agents.
-The information are collected in *rrd* files which are keeping the monitoring information.
-This information is available as time dependent plots via the ActivityMonitor web portal application.
-You can access these plots via the "System overview plots" tab in this application. In particular, it shows the load of the services
-in terms of CPU/Memory but also numbers of queries served, numbers of active threads, pending queries, etc.
-These plots are very useful for understanding of your services behavior, for example, of your FileCatalog service.
+The framework system for the monitoring of services and agents has been removed and will no longer be used.
 
-The bookkeeping of the rrd files is kept in an sqlite database usually kept in /opt/dirac/data/monitoring/monitoring.db file.
-There is no cleaning procedure foreseen for the rrd files.
-
-A Monitoring System based on ElasticSearch database as backend is possible,
-please read about it in :ref:`Monitoring <monitoring_system>`.
+It has been replaced by an ElasticSearch-based monitoring system. You can read about it in :ref:`Monitoring <monitoring_system>`

docs/source/DeveloperGuide/Systems/Framework/index.rst

@@ -107,62 +107,6 @@ The MonitoringUtilities module provides the functionality needed to store or del
   if not result[ 'OK' ]:
     print 'Something went wrong'
 
-gMonitor object Internals
-============================
-This part covers the internal workings of the **gMonitor** object.
-**gMonitor** is a singleton object meaning its instance is only created once from the **MonitoringClient** and is used throughout in other systems for monitoring the component details like CPU, MEM, threads, etc.
-
-Client - Server interactions of the **gMonitor** object (basic overview diagram):
-
-.. image:: /_static/Systems/FS/gMonitorInteractions.png
-   :alt: gMonitor Client & Server internal interactions.
-   :align: center
-
-**1) MonitoringClient (inherited by gMonitor singleton object):**
-This class is inherited by the gMonitor object and is used for reporting all types of components
-Activities.
-Basically, from here all the information is sent to the MonitoringHandler(server side) using methods like
-**registerActivity** and **addMark** and all the data is flushed periodicaly by the object **gMonitoringFlusher**
-every 300 seconds.
-
-**2) MonitoringHandler:**
-This class is inherited from the **RequestHandler** class which is the base required for running any kind of services in DIRAC.
-This class is basically used to expose methods inside ServiceInterface which is accessed
-with the help of the **gServiceInterface** (singleton object) and before exposing the methods
-there are some filters applied like not including invalid actions, decoding view descriptions, etc.
-Here the methods called inside **export_** methods will be explained in depth inside the
-**ServiceInterface** part.
-
-**3) ServiceInterface (inherited by the gServiceInterface singleton object):**
-This is the place where all the activity data is received from the **gMonitor** object and from here the other the internal interactions of the
-**Framework/MonitoringHandler** happens with components like **RRDManager** used for reporting data to the **rrdtool** which is based on command line
-used for generating plots, the **ComponentMonitoringDB** which is used to store the basic component information like component type, component name,
-number of queries performed, etc and the **MonitoringCatalog** which is used to store all the activity data inside the sqlite3 database which is lastly
-referred by the **rrdtool** and the Web interface i.e. the **ActivityMonitor** part.
-
-**4) RRDManager (wrapper around the rrdtool):**
-This is class is called by the **ServiceInterface**
-This class is a wrap around the **rrdtool** as it is a command line based tool within this class there are several methods which take in some parameters
-required by the corresponding rrd command and executes it.
-
-**5) MonitoringCatalog (based on sqlite3):**
-Here all the information about the activity of the components is maintained like which rrd file to refer for what components
-then other details like CPU, MEM, Queries, etc. is stored here.
-
-**6) ComponentMonitoringDB (based on MySQL):**
-This database basically stores the basic information of the components not the activity data like **Type, ComponentName, Port, Host, etc**
-and this all is stored under the **compmon_Components** table inside this database.
-The other info like version history of the components like **VersionTimestamp, DIRACVersion, etc.** is stored under **compmon_VersionHistory**
-table of the database.
-
-**7) PlotCache**
-This is basically used to cache the graphs that are generated once so if the user tries to generate them again the entire
-process of graph generation using **RRDManager** is not performed instead a cached version of the graph is returned to reduce the
-response time.
-
-This was just an overview of the components that **gMonitor** object interacts with more detail can be found inside the code of that particular
-component.
-The use cases of **gMonitor** object can be found inside the **DIRAC/Core/DISET/private/Service.py, DIRAC/Core/Base/AgentModule.py, Transformation System, etc.**
 
 Dynamic Component Monitoring
 ============================

docs/source/DeveloperGuide/Systems/Monitoring/index.rst

@@ -43,7 +43,6 @@ It is based on layered architecture and is based on DIRAC architecture:
 
 
 * **Clients**
-   * MonitoringClient is used to interact withe the Monitoring service.
    * Types contains all Monitoring types.
 
 -------------------------------

docs/source/DeveloperGuide/Systems/RequestManagement/index.rst

@@ -247,7 +247,6 @@ extend `RequestValidator` to cope with the new type. The handler should be a fun
 constructor (__init__) and () operator ( __call__)::
 
     """ KillParrot operation handler """
-    from DIRAC import gMonitor
     from DIRAC.RequestManagementSystem.private.OperationHandlerBase import OperationHandlerBase
     import random
 
@@ -264,9 +263,6 @@ constructor (__init__) and () operator ( __call__)::
         """ constructor -- DO NOT CHANGE its arguments list """
         # # AND ALWAYS call BASE class constructor (or it won't work at all)
         OperationHandlerBase.__init__(self, request, csPath )
-        # # put there something more if you need, i.e. gMonitor registration
-        gMonitor.registerActivity( "ParrotsDead", ... )
-        gMonitor.registerActivity( "ParrotsAlive", ... )
 
       def __call__( self ):
         """ this has to be defined and should return S_OK/S_ERROR """
@@ -282,11 +278,9 @@ constructor (__init__) and () operator ( __call__)::
           self.log.error( "Parrot is still alive" )
           self.operation.Error = "It's only sleeping"
           self.operation.Status = "Failed"
-          gMonitor.addMark( "ParrotsAlive" , 1 )
         else:
           self.log.info( "Parrot is stone dead")
           self.operation.Status = "Done"
-          gMonitor.addMark( "ParrotsDead", 1)
         # # return S_OK/S_ERROR (always!!!)
         return S_OK()
 

docs/source/_static/Systems/FS/gMonitorInteractions.png

@@ -171,7 +171,6 @@ console_scripts =
     dirac-admin-users-with-proxy = DIRAC.FrameworkSystem.scripts.dirac_admin_users_with_proxy:main [admin]
     dirac-install-component = DIRAC.FrameworkSystem.scripts.dirac_install_component:main [server]
     dirac-install-tornado-service = DIRAC.FrameworkSystem.scripts.dirac_install_tornado_service:main [server]
-    dirac-monitoring-get-components-status = DIRAC.FrameworkSystem.scripts.dirac_monitoring_get_components_status:main [admin]
     dirac-myproxy-upload = DIRAC.FrameworkSystem.scripts.dirac_myproxy_upload:main
     dirac-populate-component-db = DIRAC.FrameworkSystem.scripts.dirac_populate_component_db:main [admin]
     dirac-proxy-destroy = DIRAC.FrameworkSystem.scripts.dirac_proxy_destroy:main

setup.cfg

@@ -7,7 +7,6 @@
 
 from DIRAC.Core.Base.DB import DB
 from DIRAC import S_OK, S_ERROR, gConfig
-from DIRAC.FrameworkSystem.Client.MonitoringClient import gMonitor
 from DIRAC.Core.Utilities import List, ThreadSafe, Time, DEncode
 from DIRAC.Core.Utilities.Plotting.TypeLoader import TypeLoader
 from DIRAC.Core.Utilities.ThreadPool import ThreadPool
@@ -48,9 +47,6 @@ def __init__(self, name="Accounting/AccountingDB", readOnly=False):
             }
         )
         self.__loadCatalogFromDB()
-        gMonitor.registerActivity("registeradded", "Register added", "Accounting", "entries", gMonitor.OP_ACUM)
-        gMonitor.registerActivity("insertiontime", "Record insertion time", "Accounting", "seconds", gMonitor.OP_MEAN)
-        gMonitor.registerActivity("querytime", "Records query time", "Accounting", "seconds", gMonitor.OP_MEAN)
 
         self.__compactTime = datetime.time(hour=2, minute=random.randint(0, 59), second=random.randint(0, 59))
         lcd = Time.dateTime()
@@ -281,20 +277,6 @@ def registerType(self, name, definitionKeyFields, definitionAccountingFields, bu
         """
         Register a new type
         """
-        gMonitor.registerActivity(
-            "registerwaiting:%s" % name,
-            "Records waiting for insertion for %s" % " ".join(name.split("_")),
-            "Accounting",
-            "records",
-            gMonitor.OP_MEAN,
-        )
-        gMonitor.registerActivity(
-            "registeradded:%s" % name,
-            "Register added for %s" % " ".join(name.split("_")),
-            "Accounting",
-            "entries",
-            gMonitor.OP_ACUM,
-        )
 
         result = self.__loadTablesCreated()
         if not result["OK"]:
@@ -628,16 +610,13 @@ def __insertFromINTable(self, recordTuples):
             result = self._update("DELETE FROM `%s` WHERE id=%s" % (_getTableName("in", typeName), iD))
             if not result["OK"]:
                 self.log.error("Can't delete row from the IN table", result["Message"])
-            gMonitor.addMark("insertiontime", Time.toEpoch() - insertionEpoch)
 
     def insertRecordDirectly(self, typeName, startTime, endTime, valuesList):
         """
         Add an entry to the type contents
         """
         if self.__readOnly:
             return S_ERROR("ReadOnly mode enabled. No modification allowed")
-        gMonitor.addMark("registeradded", 1)
-        gMonitor.addMark("registeradded:%s" % typeName, 1)
         self.log.info(
             "Adding record", "for type %s\n [%s -> %s]" % (typeName, Time.fromEpoch(startTime), Time.fromEpoch(endTime))
         )
@@ -977,7 +956,6 @@ def retrieveBucketedData(
         result = self.__queryType(
             typeName, startTime, endTime, selectFields, condDict, groupFields, orderFields, "bucket", connObj=connObj
         )
-        gMonitor.addMark("querytime", Time.toEpoch() - startQueryEpoch)
         return result
 
     def __queryType(

src/DIRAC/AccountingSystem/DB/AccountingDB.py

@@ -12,7 +12,6 @@
 from DIRAC import S_OK, S_ERROR, rootPath, gConfig, gLogger
 from DIRAC.Core.Utilities.File import mkDir
 from DIRAC.Core.Utilities import Time
-from DIRAC.FrameworkSystem.Client.MonitoringClient import gMonitor
 from DIRAC.AccountingSystem.DB.MultiAccountingDB import MultiAccountingDB
 from DIRAC.Core.Utilities.Plotting import gDataCache
 from DIRAC.AccountingSystem.private.MainReporter import MainReporter
@@ -59,10 +58,6 @@ def initializeHandler(cls, serviceInfo):
             gLogger.fatal("Can't write to %s" % dataPath)
             return S_ERROR("Data location is not writable")
         gDataCache.setGraphsLocation(dataPath)
-        gMonitor.registerActivity("plotsDrawn", "Drawn plot images", "Accounting reports", "plots", gMonitor.OP_SUM)
-        gMonitor.registerActivity(
-            "reportsRequested", "Generated reports", "Accounting reports", "reports", gMonitor.OP_SUM
-        )
         return S_OK()
 
     def __checkPlotRequest(self, reportRequest):
@@ -121,7 +116,6 @@ def export_generatePlot(self, reportRequest):
         if not retVal["OK"]:
             return retVal
         reporter = MainReporter(self.__acDB, self.serviceInfoDict["clientSetup"])
-        gMonitor.addMark("plotsDrawn")
         reportRequest["generatePlot"] = True
         return reporter.generate(reportRequest, self.getRemoteCredentials())
 
@@ -143,7 +137,6 @@ def export_getReport(self, reportRequest):
         if not retVal["OK"]:
             return retVal
         reporter = MainReporter(self.__acDB, self.serviceInfoDict["clientSetup"])
-        gMonitor.addMark("reportsRequested")
         reportRequest["generatePlot"] = False
         return reporter.generate(reportRequest, self.getRemoteCredentials())