Feature: add a "preview" mode?

[woodruffw]

As zizmor's adoption has increased, I've become more conservative about adding new audits (even when I think they're correct) because of the noise/fatigue risk associated with them.

One possible way to mitigate this would be to add "preview" functionality -- audits would be released in new versions behind a "preview" flag/toggle, and users would have to opt into them. Then, a subsequent minor release would take the new audit out of preview.

The UI/CLI for this would look something like:

# run with previewed audits enabled
zizmor --preview-audits ...

# or
ZIZMOR_PREVIEW_AUDITS=1 zizmor ...

(or similar, this could use bikeshedding.)

The "pro" of this is that it would allow me to ship audits more rapidly and hopefully get user feedback on them while they're still in preview/not disrupting as many people. The "con" is that I strongly suspect most people won't enable an off-by-default preview feature. I'm eager/interested for user thoughts on it.

[funnelfiasco]

I'm tempted to move this into Discussions with a big ol' smirk on my face.

I think you articulate the pros and cons very well. Most people won't enable an off-by-default preview feature, but those who do will probably be the ones who can give them the most detailed workout (e.g. weird edge cases).

I'm in favor of the idea, generally. Do you envision a standard "promotion" process or will each one be handled as a unique case? Like is it "a preview audit will become not-preview in X releases"?

[woodruffw]

I'm tempted to move this into Discussions with a big ol' smirk on my face.

Please!

I'm in favor of the idea, generally. Do you envision a standard "promotion" process or will each one be handled as a unique case? Like is it "a preview audit will become not-preview in X releases"?

Yeah, I was kind of imagining it being mostly "preview for one release, and then promoted to stable in the next 1-2 releases." The exception would be if a previewed audit gets lots of feedback around noise or quality.

[woodruffw]

One other problem/design nuance here is going to be around adding new findings to existing audits -- those would ideally be flammable as "preview" as well, but that's a layer beneath the audit ID layer. Not sure yet how best to approach that (if at all).