Challenge the way we pass or fail several CISA Exchange Online tests

[tdcthosc]

I'd like to start a discussion on how we generally fail and pass many CISA Exchange Online tests based on whether they have standard and strict presets enabled, even though they pass the recommendation with other policies. I believe this was introduced simply because in many cases the remediation suggestion by CISA is to enable presets – not configure a policy accordingly. However, in some cases they do describe the configuration needed, but using presets as starting point.

We do this in several tests and it means that tenants, that have decided not to use standard and strict, will always fail these checks even if they have a default or custom policy passing the test.

Example: CISA.MS.EXO.11.1

CISA specifies that impersonation protection should be enabled. In our test we check if at least 1 policy has impersonation protection enabled and list it with pass or failed.

However, if not both standard and strict is enabled, we determine the overall test failed.

This also gives false indications of coverage, as standard and strict may be applied to only a few users.

@soulemike I'd love to hear your thoughts on this as you originally implemented these tests (Thank you – you are awesome!)

[soulemike]

No disagreement this could be improved. Here are a couple of thoughts:

• Add additional more nuanced tests, that are under the Maester specific tests. I still feel like this is a general gap, with cross-referencing tests that are similar. CISA, CIS, EIDSCA, MT, etc... each of those may have their own version of the same setting guideline. My wish list around this is complex. haha
• Use the strict or other switch parameter to provide our preference versus the CISA perspective.

[tdcthosc]

Given that you've already done a lot of the preparations by simply not only checking if strict and standard is enabled, but actually evaluating all policies, I think that a switch would make sense.

This way we still give attribution the source of the test (CISA) – and we know about the unsung heroes behind the scenes (you) 😄

Let's get some more opinions before getting to work and also discuss how to implement it. E.g. should it be a switch, tags, etc. on invoke or perhaps expanding and utilizing the measter-config file.

[tdcthosc]

I'd love to hear your thoughts @SamErde and @merill