Merge branch 'master' into doc/pbs-extra-resources

Author: vkarak

docs/config_reference.rst

@@ -15,7 +15,7 @@ Even if a configuration object contains a list of other objects, this is not ref
 For example, by ``systems.partitions.name`` we designate the ``name`` property of any partition object inside the ``partitions`` property of any system object inside the top level ``systems`` object.
 If we were to use indices, that would be rewritten as ``systems[i].partitions[j].name`` where ``i`` indexes the systems and ``j`` indexes the partitions of the i-th system.
 For cases, where the objects in a list are not homogeneous, e.g., the logging handlers, we surround the object type with ``..``.
-For example, the ``logging.handlers..filelog..name`` syntax designates the ``name`` attribute of the ``filelog`` logging handler.
+For example, the ``logging.handlers_perflog..filelog..name`` syntax designates the ``name`` attribute of the ``filelog`` logging handler.
 
 .. |schemas/config.json| replace:: ``reframe/schemas/config.json``
 .. _schemas/config.json: https://github.com/reframe-hpc/reframe/blob/master/reframe/schemas/config.json
@@ -1138,12 +1138,10 @@ The additional properties for the ``file`` handler are the following:
 The ``filelog`` log handler
 ---------------------------
 
-This handler is meant primarily for performance logging and logs the performance of a test in one or more files.
+This handler is meant for performance logging only and logs the performance of a test in one or more files.
 The additional properties for the ``filelog`` handler are the following:
 
 
-.. py:attribute:: logging.handlers..filelog..basedir
-
 .. py:attribute:: logging.handlers_perflog..filelog..basedir
 
    :required: No
@@ -1152,13 +1150,11 @@ The additional properties for the ``filelog`` handler are the following:
    The base directory of performance data log files.
 
 
-.. py:attribute:: logging.handlers..filelog..prefix
-
 .. py:attribute:: logging.handlers_perflog..filelog..prefix
 
    :required: Yes
 
-   This is a directory prefix (usually dynamic), appended to the :attr:`~config.logging.handlers..filelog..basedir`, where the performance logs of a test will be stored.
+   This is a directory prefix (usually dynamic), appended to the :attr:`~config.logging.handlers_perflog..filelog..basedir`, where the performance logs of a test will be stored.
    This attribute accepts any of the check-specific `formatting placeholders <#config.logging.handlers_perflog.format>`__.
    This allows to create dynamic paths based on the current system, partition and/or programming environment a test executes with.
    For example, a value of ``%(check_system)s/%(check_partition)s`` would generate the following structure of performance log files:
@@ -1177,8 +1173,6 @@ The additional properties for the ``filelog`` handler are the following:
         ...
 
 
-.. py:attribute:: logging.handlers..filelog..append
-
 .. py:attribute:: logging.handlers_perflog..filelog..append
 
    :required: No
@@ -1196,20 +1190,16 @@ The additional properties for the ``filelog`` handler are the following:
 The ``graylog`` log handler
 ---------------------------
 
-This handler sends log records to a `Graylog <https://www.graylog.org/>`__ server.
+This handler is meant for performance logging only and sends log records to a `Graylog <https://www.graylog.org/>`__ server.
 The additional properties for the ``graylog`` handler are the following:
 
-.. py:attribute:: logging.handlers..graylog..address
-
 .. py:attribute:: logging.handlers_perflog..graylog..address
 
    :required: Yes
 
    The address of the Graylog server defined as ``host:port``.
 
 
-.. py:attribute:: logging.handlers..graylog..extras
-
 .. py:attribute:: logging.handlers_perflog..graylog..extras
 
    :required: No
@@ -1313,17 +1303,13 @@ The ``httpjson`` log handler
 This handler sends log records in JSON format to a server using HTTP POST requests.
 The additional properties for the ``httpjson`` handler are the following:
 
-.. py:attribute:: logging.handlers..httpjson..url
-
 .. py:attribute:: logging.handlers_perflog..httpjson..url
 
    :required: Yes
 
    The URL to be used in the HTTP(S) request server.
 
 
-.. py:attribute:: logging.handlers..httpjson..extra_headers
-
 .. py:attribute:: logging.handlers_perflog..httpjson..extra_headers
 
    :required: No
@@ -1335,8 +1321,6 @@ The additional properties for the ``httpjson`` handler are the following:
    .. versionadded:: 4.2
 
 
-.. py:attribute:: logging.handlers..httpjson..extras
-
 .. py:attribute:: logging.handlers_perflog..httpjson..extras
 
    :required: No
@@ -1345,8 +1329,6 @@ The additional properties for the ``httpjson`` handler are the following:
    A set of optional key/value pairs to be passed with each log record to the server.
    These may depend on the server configuration.
 
-.. py:attribute:: logging.handlers..httpjson..ignore_keys
-
 .. py:attribute:: logging.handlers_perflog..httpjson..ignore_keys
 
    :required: No
@@ -1376,8 +1358,6 @@ An example configuration of this handler for performance logging is shown here:
 
 This handler transmits the whole log record, meaning that all the information will be available and indexable at the remote end.
 
-.. py:attribute:: logging.handlers..httpjson..debug
-
 .. py:attribute:: logging.handlers_perflog..httpjson..debug
 
    :required: No
@@ -1389,8 +1369,6 @@ This handler transmits the whole log record, meaning that all the information wi
    .. versionadded:: 4.1
 
 
-.. py:attribute:: logging.handlers..httpjson..json_formatter
-
 .. py:attribute:: logging.handlers_perflog..httpjson..json_formatter
 
    A callable for converting the log record into JSON.
@@ -1402,8 +1380,8 @@ This handler transmits the whole log record, meaning that all the information wi
       :arg record: The prepared log record.
          The log record is a simple Python object with all the attributes listed in :attr:`~config.logging.handlers.format`, as well as all the default Python `log record <https://docs.python.org/3.8/library/logging.html#logrecord-attributes>`__ attributes.
          In addition to those, there is also the special :attr:`__rfm_check__` attribute that contains a reference to the actual test for which the performance is being logged.
-      :arg extras: Any extra attributes specified in :attr:`~config.logging.handlers..httpjson..extras`.
-      :arg ignore_keys: The set of keys specified in :attr:`~config.logging.handlers..httpjson..ignore_keys`.
+      :arg extras: Any extra attributes specified in :attr:`~config.logging.handlers_perflog..httpjson..extras`.
+      :arg ignore_keys: The set of keys specified in :attr:`~config.logging.handlers_perflog..httpjson..ignore_keys`.
          ReFrame always adds the default Python log record attributes in this set.
       :returns: A string representation of the JSON record to be sent to the server or :obj:`None` if the record should not be sent to the server.
 

docs/index.rst

@@ -18,7 +18,7 @@ Finally, ReFrame offers a powerful and efficient runtime for running and managin
 Publications
 ============
 
-* Slides [`Google Slides <https://docs.google.com/presentation/d/1GmO2Uf29SaLg36bPB9g9eeaKMN-bLlDJ5IvLGLQJfD8/edit?usp=share_link>`__] @ `8th EasyBuild User Meeting 2023 <https://easybuild.io/eum23/>`__.
+* Slides [`part 1 <https://docs.google.com/presentation/d/1GmO2Uf29SaLg36bPB9g9eeaKMN-bLlDJ5IvLGLQJfD8/edit?usp=share_link>`__][`part 2 <https://drive.google.com/file/d/1gZwch0BPc1wDEkMwbM4vxCpMzWIx-Lo1/view?usp=sharing>`__][`talk <https://youtu.be/0ApEKc185Bw>`__] @ `8th EasyBuild User Meeting 2023 <https://easybuild.io/eum23/>`__.
 * Slides [`pdf <https://drive.google.com/file/d/1vmaWyRHgtq3DrYhSCVBzR8U5ErKbxGNf/view?usp=sharing>`__] @ `7th EasyBuild User Meeting 2022 <https://easybuild.io/eum22/>`__.
 * Slides [`pdf <https://drive.google.com/file/d/1kNZu1QNBDDsbKarzwNWYjTGKgOukg-96/view?usp=sharing>`__] @ `6th EasyBuild User Meeting 2021 <https://easybuild.io/eum21/>`__.
 * Slides [`pdf <https://drive.google.com/open?id=1W7R5lfRkXvBpVDSZ7dVBadk_d3K4dFrS>`__] @ `5th EasyBuild User Meeting 2020 <https://github.com/easybuilders/easybuild/wiki/5th-EasyBuild-User-Meeting>`__.

docs/manpage.rst

@@ -1321,7 +1321,7 @@ Whenever an environment variable is associated with a configuration option, its
 
       ================================== ==================
       Associated command line option     N/A
-      Associated configuration parameter :attr:`~config.logging.handlers..graylog..address`
+      Associated configuration parameter :attr:`~config.logging.handlers_perflog..graylog..address`
       ================================== ==================
 
 
@@ -1338,7 +1338,7 @@ Whenever an environment variable is associated with a configuration option, its
 
       ================================== ==================
       Associated command line option     N/A
-      Associated configuration parameter :attr:`~config.logging.handlers..httpjson..url`
+      Associated configuration parameter :attr:`~config.logging.handlers_perflog..httpjson..url`
       ================================== ==================
 
 
@@ -1445,7 +1445,7 @@ Whenever an environment variable is associated with a configuration option, its
 
       ================================== ==================
       Associated command line option     :option:`--perflogdir`
-      Associated configuration parameter :attr:`~config.logging.handlers..filelog..basedir`
+      Associated configuration parameter :attr:`~config.logging.handlers_perflog..filelog..basedir`
       ================================== ==================
 
 

reframe/schemas/config.json

@@ -412,11 +412,9 @@
                         "items": {
                             "anyOf": [
                                 {"$ref": "#/defs/file_handler"},
-                                {"$ref": "#/defs/filelog_handler"},
                                 {"$ref": "#/defs/graylog_handler"},
                                 {"$ref": "#/defs/stream_handler"},
-                                {"$ref": "#/defs/syslog_handler"},
-                                {"$ref": "#/defs/httpjson_handler"}
+                                {"$ref": "#/defs/syslog_handler"}
                             ]
                         }
                     },
@@ -425,11 +423,9 @@
                         "items": {
                             "anyOf": [
                                 {"$ref": "#/defs/file_handler"},
-                                {"$ref": "#/defs/filelog_handler"},
                                 {"$ref": "#/defs/graylog_handler"},
                                 {"$ref": "#/defs/stream_handler"},
-                                {"$ref": "#/defs/syslog_handler"},
-                                {"$ref": "#/defs/httpjson_handler"}
+                                {"$ref": "#/defs/syslog_handler"}
                             ]
                         },
                         "minItems": 1
@@ -442,7 +438,8 @@
                                 {"$ref": "#/defs/filelog_handler"},
                                 {"$ref": "#/defs/graylog_handler"},
                                 {"$ref": "#/defs/stream_handler"},
-                                {"$ref": "#/defs/syslog_handler"}
+                                {"$ref": "#/defs/syslog_handler"},
+                                {"$ref": "#/defs/httpjson_handler"}
                             ]
                         }
                     },
@@ -566,17 +563,17 @@
         "logging/handlers*/file_append": false,
         "logging/handlers*/file_name": "",
         "logging/handlers*/file_timestamp": false,
-        "logging/handlers*/filelog_append": true,
-        "logging/handlers*/filelog_basedir": "./perflogs",
-        "logging/handlers*/graylog_extras": {},
-        "logging/handlers*/httpjson_extras": {},
-        "logging/handlers*/httpjson_ignore_keys": [],
-        "logging/handlers*/httpjson_json_formatter": null,
-        "logging/handlers*/httpjson_extra_headers": {},
-        "logging/handlers*/httpjson_debug": false,
         "logging/handlers*/stream_name": "stdout",
         "logging/handlers*/syslog_socktype": "udp",
         "logging/handlers*/syslog_facility": "user",
+        "logging/handlers_perflog/filelog_append": true,
+        "logging/handlers_perflog/filelog_basedir": "./perflogs",
+        "logging/handlers_perflog/graylog_extras": {},
+        "logging/handlers_perflog/httpjson_extras": {},
+        "logging/handlers_perflog/httpjson_ignore_keys": [],
+        "logging/handlers_perflog/httpjson_json_formatter": null,
+        "logging/handlers_perflog/httpjson_extra_headers": {},
+        "logging/handlers_perflog/httpjson_debug": false,
         "modes/options": [],
         "modes/target_systems": ["*"],
         "systems/descr": "",
@@ -609,9 +606,9 @@
         "systems/partitions/time_limit": null,
         "systems/partitions/devices": [],
         "systems/partitions/extras": {},
-        "{systems,systems/partitions}/sched_options/ignore_reqnodenotavail": false,
-        "{systems,systems/partitions}/sched_options/job_submit_timeout": 60,
-        "{systems,systems/partitions}/sched_options/resubmit_on_errors": [],
-        "{systems,systems/partitions}/sched_options/use_nodes_option": false
+        "systems*/sched_options/ignore_reqnodenotavail": false,
+        "systems*/sched_options/job_submit_timeout": 60,
+        "systems*/sched_options/resubmit_on_errors": [],
+        "systems*/sched_options/use_nodes_option": false
     }
 }

unittests/test_config.py

@@ -301,6 +301,12 @@ def test_select_subconfig(site_config):
     assert site_config.get('systems/0/sched_options') == {
         'job_submit_timeout': 10
     }
+    assert site_config.get(
+        'systems/0/sched_options/resubmit_on_errors'
+    ) == []
+    assert site_config.get(
+        'systems/0/partitions/@gpu/sched_options/resubmit_on_errors'
+    ) == []
     assert site_config.get('environments/@PrgEnv-gnu/cc') == 'cc'
     assert site_config.get('environments/1/cxx') == 'CC'
     assert site_config.get('general/0/check_search_path') == ['c:d']