CI: remove invalidations checks

These don't work well in practice; lots of fluctuations caused by
changes in Julia and dependencies. The README for
`julia-actions/julia-invalidations` states:

> WARNING: this action may place blame in the wrong place. In most
> cases invalidation is the "fault" of the victim of invalidation,
> not the perpetrator

Author: fingolfin