qa/tasks/cephadm: add option to limit what matches in log error scraping

This is specifically being added with the orch/cephadm suite
in mind, where coming up with a viable ignorelist has proved
difficult. The orch testing does a lot of actions that can
cause thigns like an OSD or MON daemon to be down very
briefly, and I've found the vast majority of the time we
really don't want to fail the test when these pop up as cephadm
testing really only benefits from catching the CEPHADM_ errors/
warnings rather than eveyr possible one. Rather than continuing to
play whack-a-mole with the errors in the logs, this
patch should allow us to limit what we fail on to at
least get the suite in a good spot again. We can always
phase out the uses of this new "log-only_match" option
later in a more controlled way, and adding it shouldn't
affect log scraping for any of the tests that aren't
facing a similar issue.

Signed-off-by: Adam King <[email protected]>

Author: adk3798

qa/tasks/cephadm.py

@@ -430,7 +430,7 @@ def ceph_log(ctx, config):
 
     finally:
         log.info('Checking cluster log for badness...')
-        def first_in_ceph_log(pattern, excludes):
+        def first_in_ceph_log(pattern, excludes, only_match):
             """
             Find the first occurrence of the pattern specified in the Ceph log,
             Returns None if none found.
@@ -445,6 +445,8 @@ def first_in_ceph_log(pattern, excludes):
                 '/var/log/ceph/{fsid}/ceph.log'.format(
                     fsid=fsid),
             ]
+            if only_match:
+                args.extend([run.Raw('|'), 'egrep', '|'.join(only_match)])
             if excludes:
                 for exclude in excludes:
                     args.extend([run.Raw('|'), 'egrep', '-v', exclude])
@@ -460,14 +462,22 @@ def first_in_ceph_log(pattern, excludes):
                 return stdout
             return None
 
+        # NOTE: technically the first and third arg to first_in_ceph_log
+        # are serving a similar purpose here of being something we
+        # look for in the logs. The reason they are separate args is that
+        # we want '\[ERR\]|\[WRN\]|\[SEC\]' to always have to be in the thing
+        # we match even if the test yaml specifies nothing else, and then the
+        # log-only-match options are for when a test only wants to fail on
+        # a specific subset of log lines that '\[ERR\]|\[WRN\]|\[SEC\]' matches
         if first_in_ceph_log('\[ERR\]|\[WRN\]|\[SEC\]',
-                             config.get('log-ignorelist')) is not None:
+                             config.get('log-ignorelist'),
+                             config.get('log-only-match')) is not None:
             log.warning('Found errors (ERR|WRN|SEC) in cluster log')
             ctx.summary['success'] = False
             # use the most severe problem as the failure reason
             if 'failure_reason' not in ctx.summary:
                 for pattern in ['\[SEC\]', '\[ERR\]', '\[WRN\]']:
-                    match = first_in_ceph_log(pattern, config['log-ignorelist'])
+                    match = first_in_ceph_log(pattern, config['log-ignorelist'], config.get('log-only-match'))
                     if match is not None:
                         ctx.summary['failure_reason'] = \
                             '"{match}" in cluster log'.format(