Stabilize BDN tests by adding a flag on tests that only Warn on fail instead of causing pipeline to fail (#1119)

* Change to the BDN to allow to set expected values to only report as a warning instead of a fail. This is for tests that are not very deterministic / stable and we don't want false fails to be showing. The test is still ran and the data is still saved for the charts.  Docs on website were updated as well.

* a .bak file was left behind so removed it

---------

Co-authored-by: Badrish Chandramouli <[email protected]>

Author: darrenge