docs: add comments and code documentation about token usage

add more comments and code docs

fix Auth to AuthDB

add docs and comments to TokenDB

Rebased to get pre-commit changes using
  * gh pr checkout 5397
  * git rebase 67f5059
  * pre-commit run --all-files
  * black src/DIRAC/ProductionSystem/DB/ProductionDB.py src/DIRAC/TransformationSystem/DB/TransformationDB.py
  * git commit --no-edit -c 75afcca
  * git rebase upstream/integration

Author: TaykYoku

docs/source/AdministratorGuide/UserManagement/index.rst

@@ -9,10 +9,9 @@ This section provides information you need for the user management.
 What are the components involved in that.
 -----------------------------------------
 
-  * Configuration system
-
-   * Registry
-   * VOMS2CSAgent
+  - Configuration system
+  - Registry
+  - VOMS2CSAgent
 
 
 What is the user in DIRAC context?
@@ -42,10 +41,10 @@ Consider the registration process
 
 User management has been provided by the Registry section of the Configuration System. To manage it you can use:
 
-* :ref:`dirac commands <admin_registry_cmd>` to managing Registry
-* configuration manager application in the Web portal (need to :ref:`install WebAppDIRAC extension <installwebappdirac>`)
-* modify local cfg file manually (by default it located in /opt/dirac/etc/dirac.cfg)
-* use the :mod:`~DIRAC.ConfigurationSystem.Agent.VOMS2CSAgent` to fetch VOMS VO users
+  - :ref:`dirac commands <registry_cmd>` to managing Registry
+  - configuration manager application in the Web portal (need to :ref:`install WebAppDIRAC extension <installwebappdirac>`)
+  - modify local cfg file manually (by default it located in /opt/dirac/etc/dirac.cfg)
+  - use the :mod:`~DIRAC.ConfigurationSystem.Agent.VOMS2CSAgent` to fetch VOMS VO users
 
 In a nutshell, how to edit the configuration from the portal. First, it should be noted that to be able to do this,
 you must be an already registered user in a group that has the appropriate permission to edit the configuration("CSAdministrator").

docs/source/UserGuide/GettingStarted/GettingUserIdentity/index.rst

@@ -19,7 +19,7 @@ The user will be prompted for the password used while exporting the certificate
 to be used with the user's private key. Do not forget it !
 
 Registration with DIRAC
--------------------------
+-----------------------
 
 Users are always working in the Grid as members of some User Community. Therefore, every user must be registered
 with the Community DIRAC instance. You should ask the DIRAC administrators to do that, the procedure can
@@ -30,7 +30,7 @@ determines the user rights for various Grid operations. Each DIRAC installation
 group to which the users are attributed when the group is not explicitly specified.
 
 Proxy initialization
------------------------
+--------------------
 
 Users authenticate with DIRAC services, and therefore with the Grid services that DIRAC expose via "proxies",
 which you can regard as a product of personal certificates.
@@ -53,3 +53,20 @@ If another non-default user group is needed, the command becomes::
   $ dirac-proxy-init -g <user_group>
 
 where ``user_group`` is the desired DIRAC group name for which the user is entitled.
+
+Token authorization
+-------------------
+
+Starting with the 8.0 version of DIRAC, it is possible to authorize users through third party Identity Providers (IdP),
+such as EGI Checkin [https://www.egi.eu/services/check-in/] or WLCG IAM (https://indico.cern.ch/event/739896/contributions/3497694/attachments/1905332/3146590/IAM-WLCG-AuthZ-Fermilab-10092019.pdf).
+To do this, you do not need to have a certificate if you use a terminal, the main thing is that you must be registered in one of the supported IdP. The registration process is different for each IdP.
+
+Once your account is created, you will be able to register with DIRAC using the `dirac-login` command that will return tokens that will be used to access the services::
+
+  dirac-login -g <user_group>
+
+But since not all services currently support tokens, you can get a proxy if you use the *--proxy* key::
+
+  dirac-login -g <user_group> --proxy
+
+Note that to get a proxy you must first put it in DIRAC, see "Proxy initialization".

src/DIRAC/Core/Tornado/Server/TornadoREST.py

@@ -65,12 +65,10 @@ def web_users(self, count: int = 0):
     and we are running using executors, the methods you export cannot write
     back directly to the client. Please see inline comments for more details.
 
-    In order to pass information around and keep some states, we use instance attributes.
-    These are initialized in the :py:meth:`.initialize` method.
-
     The handler define the ``post`` and ``get`` verbs. Please refer to :py:meth:`.post` for the details.
     """
 
+    # By default we enable all authorization grants
     USE_AUTHZ_GRANTS = ["SSL", "JWT", "VISITOR"]
     METHOD_PREFIX = "web_"
     LOCATION = "/"
@@ -118,15 +116,15 @@ def _getMethodName(self):
     @gen.coroutine
     def get(self, *args, **kwargs):  # pylint: disable=arguments-differ
         """Method to handle incoming ``GET`` requests.
-        Note that all the arguments are already prepared in the :py:meth:`.prepare` method.
+        Note that all the arguments are already prepared in the :py:meth:`BaseRequestHandler.prepare` method.
         """
         retVal = yield IOLoop.current().run_in_executor(*self._prepareExecutor(args))
         self._finishFuture(retVal)
 
     @gen.coroutine
     def post(self, *args, **kwargs):  # pylint: disable=arguments-differ
         """Method to handle incoming ``POST`` requests.
-        Note that all the arguments are already prepared in the :py:meth:`.prepare` method.
+        Note that all the arguments are already prepared in the :py:meth:`BaseRequestHandler.prepare` method.
         """
         retVal = yield IOLoop.current().run_in_executor(*self._prepareExecutor(args))
         self._finishFuture(retVal)
@@ -135,9 +133,7 @@ def post(self, *args, **kwargs):  # pylint: disable=arguments-differ
 
     @staticmethod
     def web_echo(data):
-        """
-        This method used for testing the performance of a service
-        """
+        """This method used for testing the performance of a service"""
         return S_OK(data)
 
     auth_whoami = ["authenticated"]

src/DIRAC/Core/Tornado/Server/TornadoService.py

@@ -84,13 +84,16 @@ def export_streamToClient(self, myDataToSend, token):
           with open(myFileToSend, 'r') as fd:
             return fd.read()
 
-
     Note that because we inherit from :py:class:`tornado.web.RequestHandler`
     and we are running using executors, the methods you export cannot write
     back directly to the client. Please see inline comments for more details.
 
-    In order to pass information around and keep some states, we use instance attributes.
-    These are initialized in the :py:meth:`.initialize` method.
+    For compatibility with the existing :py:class:`DIRAC.Core.DISET.TransferClient.TransferClient`,
+    the handler can define a method ``export_streamToClient``. This is the method that will be called
+    whenever ``TransferClient.receiveFile`` is called. It is the equivalent of the DISET
+    ``transfer_toClient``.
+    Note that this is here only for compatibility, and we discourage using it for new purposes, as it is
+    bound to disappear.
 
     The handler only define the ``post`` verb. Please refer to :py:meth:`.post` for the details.
 

src/DIRAC/Core/Tornado/Server/private/BaseRequestHandler.py

@@ -38,15 +38,20 @@
 
 
 class TornadoResponse(object):
-    """This class registers tornadoes with arguments in the order they are called
+    """This class registers tornado methods with arguments in the order they are called
     from TornadoResponse to call them later.
 
-    Use::
+    This is used in exceptional cases, in most cases it is not required,
+    just use `return S_OK(data)` instead.
+
+    Usage example::
 
       def web_myEndpoint(self):
-          resp = TornadoResponse("data")
-          resp.set_status(400)
-          return resp
+          # We need not only to return the result but also to set a header.
+          resp = TornadoResponse('data')
+          resp.set_header("Content-Type", "application/x-tar")
+          # And for example redirect to another place
+          return resp.redirect('https://server')
     """
 
     __attrs = inspect.getmembers(RequestHandler)
@@ -75,7 +80,9 @@ def __setAction(self, mName, *args, **kwargs):
         return self
 
     def _runActions(self, reqObj):
-        """Calling methods in the order of their registration
+        """Calling methods in the order of their registration.
+        This method is called at the end of the request, when the main work is already done in the thread.
+        Look the `_finishFuture` method.
 
         :param reqObj: RequestHandler instance
         """
@@ -88,7 +95,7 @@ def _runActions(self, reqObj):
 
 
 class BaseRequestHandler(RequestHandler):
-    """Base class for all the Handlers.
+    """Base class for all the Handlers that uses HTTP tornado framework on server side.
     It directly inherits from :py:class:`tornado.web.RequestHandler`
 
     Each HTTP request is served by a new instance of this class.
@@ -104,8 +111,8 @@ class BaseRequestHandler(RequestHandler):
 
       class TornadoInstance(BaseRequestHandler):
 
-        # Prefix of methods names
-        METHOD_PREFIX = "export_"
+        # Prefix of methods names if need to use a special prefix. By default its "export_".
+        METHOD_PREFIX = "web_"
 
         @classmethod
         def _getServiceName(cls, request):
@@ -115,7 +122,7 @@ def _getServiceName(cls, request):
 
         @classmethod
         def _getServiceInfo(cls, serviceName, request):
-          ''' Fill service information.
+          ''' Fill service information. By default return empty dictionary.
           '''
           return {'serviceName': serviceName,
                   'serviceSectionPath': PathFinder.getServiceSection(serviceName),
@@ -156,12 +163,8 @@ def post(self, *args, **kwargs):  # pylint: disable=arguments-differ
           # retVal is :py:class:`tornado.concurrent.Future`
           self._finishFuture(retVal)
 
-    For compatibility with the existing :py:class:`DIRAC.Core.DISET.TransferClient.TransferClient`,
-    the handler can define a method ``export_streamToClient``. This is the method that will be called
-    whenever ``TransferClient.receiveFile`` is called. It is the equivalent of the DISET
-    ``transfer_toClient``.
-    Note that this is here only for compatibility, and we discourage using it for new purposes, as it is
-    bound to disappear.
+    In order to pass information around and keep some states, we use instance attributes.
+    These are initialized in the :py:meth:`.initialize` method.
     """
 
     # Because we initialize at first request, we use a flag to know if it's already done
@@ -342,9 +345,8 @@ def initializeHandler(cls, serviceInfo):
         And it must be a class method. This method is called only one time,
         at the first request
 
-        :param dict ServiceInfoDict: infos about services, it contains
-                                      'serviceName', 'serviceSectionPath',
-                                      'csPaths' and 'URL'
+        :param dict serviceInfo: infos about services, it contains information about service, e.g.:
+                                 'serviceName', 'serviceSectionPath', 'csPaths' and 'URL'
         """
         pass
 
@@ -424,15 +426,18 @@ def _getMethodAuthProps(self):
 
         :return: list
         """
+        # Cover default authorization requirements to list
         if self.AUTH_PROPS and not isinstance(self.AUTH_PROPS, (list, tuple)):
             self.AUTH_PROPS = [p.strip() for p in self.AUTH_PROPS.split(",") if p.strip()]
+        # Use auth_< method name > as primary value of the authorization requirements
         return getattr(self, "auth_" + self.mehtodName, self.AUTH_PROPS)
 
     def _getMethod(self):
         """Get method function to call.
 
         :return: function
         """
+        # Get method object using prefix and method name from request
         methodObj = getattr(self, "%s%s" % (self.METHOD_PREFIX, self.mehtodName), None)
         if not callable(methodObj):
             sLog.error("Invalid method", self.mehtodName)

src/DIRAC/FrameworkSystem/DB/AuthDB.py

@@ -1,4 +1,4 @@
-""" Auth class is a front-end to the Auth Database
+""" AuthDB class is a front-end to the AuthDB MySQL Database (via SQLAlchemy)
 """
 from __future__ import absolute_import
 from __future__ import division

src/DIRAC/FrameworkSystem/DB/TokenDB.py

@@ -1,4 +1,6 @@
-""" Auth class is a front-end to the Auth Database
+""" Token class is a front-end to the TokenDB Database.
+
+    Long-term user tokens are stored here, which can be used to obtain new tokens.
 """
 from __future__ import absolute_import
 from __future__ import division
@@ -26,19 +28,21 @@
 
 
 class Token(Model, OAuth2TokenMixin):
+    """This class describe token fields"""
+
     __tablename__ = "Token"
     __table_args__ = {"mysql_engine": "InnoDB", "mysql_charset": "utf8"}
     # access_token too large for varchar(255)
     # 767 bytes is the stated prefix limitation for InnoDB tables in MySQL version 5.6
     # https://stackoverflow.com/questions/1827063/mysql-error-key-specification-without-a-key-length
-    id = Column(Integer, autoincrement=True, primary_key=True)
-    kid = Column(String(255))
-    user_id = Column(String(255))
-    provider = Column(String(255))
-    expires_at = Column(Integer, nullable=False, default=0)
+    id = Column(Integer, autoincrement=True, primary_key=True)  # Unique token ID
+    kid = Column(String(255))  # Unique secret key ID for token encryption
+    user_id = Column(String(255))  # User identificator that registred in an identity provider, token owner
+    provider = Column(String(255))  # Provider name registred in DIRAC
+    expires_at = Column(Integer, nullable=False, default=0)  # When the access token is expired
     access_token = Column(Text, nullable=False)
     refresh_token = Column(Text, nullable=False)
-    rt_expires_at = Column(Integer, nullable=False, default=0)
+    rt_expires_at = Column(Integer, nullable=False, default=0)  # When the refresh token is expired
 
 
 class TokenDB(SQLAlchemyDB):
@@ -54,7 +58,10 @@ def __init__(self):
         self.session = scoped_session(self.sessionMaker_o)
 
     def __initializeDB(self):
-        """Create the tables"""
+        """Create the tables
+
+        :return: S_OK()/S_ERROR()
+        """
         tablesInDB = self.inspector.get_table_names()
 
         # Token
@@ -72,7 +79,7 @@ def getTokenForUserProvider(self, userID, provider):
         :param str userID: user ID
         :param str provider: provider
 
-        :return: S_OK(dict)/S_ERROR()
+        :return: S_OK(OAuth2Token)/S_ERROR() -- return an OAuth2Token object, which is also a dict
         """
         session = self.session()
         try:
@@ -88,34 +95,40 @@ def getTokenForUserProvider(self, userID, provider):
         return self.__result(session, S_OK(OAuth2Token(self.__rowToDict(token)) if token else None))
 
     def updateToken(self, token, userID, provider, rt_expired_in):
-        """Update tokens
+        """Update tokens for user and identity provider
 
         :param dict token: token info
-        :param str userID: user ID
-        :param str provider: provider
+        :param str userID: user ID that comes from identity provider
+        :param str provider: provider name
         :param int rt_expired_in: refresh token lifetime
 
-        :return: S_OK(list)/S_ERROR()
+        :return: S_OK(list)/S_ERROR() -- return old tokens that should be revoked.
         """
+        # Prepare a token to write to the database
         token["user_id"] = userID
         token["provider"] = provider
+        # If the token expiration date is not specified, we will try to determine it
         if not token.get("rt_expires_at"):
             try:
+                # This value can be contained in the token itself if it is a JWT
                 token["rt_expires_at"] = int(
                     jwt.decode(token["refresh_token"], options=dict(verify_signature=False, verify_aud=False))["exp"]
                 )
             except Exception as e:
                 self.log.debug("Cannot get refresh token expires time: %s" % repr(e))
-
+        # Otherwise, we set this value
         token["rt_expires_at"] = int(token.get("rt_expires_at", rt_expired_in + int(time.time())))
+        # We ignore expired tokens
         if token["rt_expires_at"] < time.time():
             return S_ERROR("Cannot store expired refresh token.")
 
         attrts = dict((k, v) for k, v in dict(token).items() if k in list(Token.__dict__.keys()))
         self.log.debug("Store token:", pprint.pformat(attrts))
         session = self.session()
         try:
+            # Remove expired tokens
             session.query(Token).filter(Token.expires_at < time.time()).delete()
+            # When we update existing tokens, the old tokens should be revoked
             oldTokens = session.query(Token).filter(Token.user_id == userID).filter(Token.provider == provider).all()
             session.add(Token(**attrts))
             session.query(Token).filter(Token.user_id == userID).filter(Token.provider == provider).filter(
@@ -128,12 +141,12 @@ def updateToken(self, token, userID, provider, rt_expired_in):
         return self.__result(session, S_OK([self.__rowToDict(t) for t in oldTokens] if oldTokens else []))
 
     def removeToken(self, access_token=None, refresh_token=None, user_id=None):
-        """Remove token
+        """Remove token from DB
 
         :param str access_token: access token
         :param str refresh_token: refresh token
 
-        :return: S_OK(object)/S_ERROR()
+        :return: S_OK(str)/S_ERROR()
         """
         session = self.session()
         try:
@@ -148,6 +161,12 @@ def removeToken(self, access_token=None, refresh_token=None, user_id=None):
         return self.__result(session, S_OK("Token successfully removed"))
 
     def getTokensByUserID(self, userID):
+        """Return tokens for user ID
+
+        :param str userID: user ID that return identity provider
+
+        :return: S_OK(list)/S_ERROR() -- tokens as OAuth2Token objects
+        """
         session = self.session()
         try:
             tokens = session.query(Token).filter(Token.user_id == userID).all()
@@ -158,6 +177,13 @@ def getTokensByUserID(self, userID):
         return self.__result(session, S_OK([OAuth2Token(self.__rowToDict(t)) for t in tokens]))
 
     def __result(self, session, result=None):
+        """Helper method
+
+        :param session: session instance
+        :param result: DIRAC result
+
+        :return: S_OK()/S_ERROR()
+        """
         try:
             if not result["OK"]:
                 session.rollback()