Some warnings not making build fail

[lgoettgens]

Today, we noticed mutiple Warning: Unexpected Julia interpolation in the Markdown. in our CI logs. We expected the documentation building job to fail in case of any warning due to passing warnonly=false, but this kind of warning doesn't get converted to an error.
Could this be changed, so that contributers have some chance of knowing that they introduced such an issue? In the current situation, I had to bisect multiple weeks of activity to find the offending commit.

cc @fingolfin

[mortenpi]

Making this error by default would be breaking. But I think it's a good idea to have an opt-in to turn this into an error.

[fingolfin]

@aaruni96 can you please have a look into this and see if you can make a PR?

[fingolfin]

@mortenpi just to clarify: how should such an extra option look?
Specific to this case, e.g.treat_markdown_warnings_as_error (or so)? Or some more general mechanism (but how could that look like?)

How would we explain which warnings are turned into error by warnonly=false and which not?

[lgoettgens]

Making this error by default would be breaking. But I think it's a good idea to have an opt-in to turn this into an error.

One could also argue that this is a bugfix (yes, still making a lot of CI go red).

From the documentation of makedocs:

warnonly can be used to control whether the makedocs build fails with an error, or simply prints a warning if it detects any issues with the document.

So from reading the docs I would expect that all warnings are converted to errors.

(Not that I wanna break a lot of people's CI, but that could certainly be another viewpoint on the situation)

[mortenpi]

One could also argue that this is a bugfix

That's fair. But unfortunately this one is probably quite widespread and difficult for people to debug. Arguably, it's maybe something that should somehow be addressed in the Markdown parser.

What I could suggest is that we add an opt-in for it for now, encourage people to turn it on, and then maybe in the future we can change the default? We could also update the warning message to say that "This may become an error in the future." etc. It would at least give a bit of a "deprecation" period.

[fingolfin]

So @aaruni96 now made PR #2792 addressing this. But on top of it he also made PR #2793 which improves the diagnostics for this.

So perhaps we can roll this out, and at some point perhaps we can switch the default for treat_markdown_warnings_as_error to true if warnonly=false and see how big the fallout is... I.e.: put it into a release, and then wait if lots of people complain about the failures; if so it could just be rolled back...

(Here we suffer from the fact that there is no analogue to NanoSoldier but for testing Documentation issues...)