Currently each developer has to setup flake8 on their own and look into the test_flake8.py for what rules should be ignored and what files to run it on.
Unrelated to this but, @dirk-thomas what's your opinion about switching to black or leave it up to each extension to decide the formatter/linter of choice?
I've used black consistently, it has excellent editors support and I haven't seen any notable bug when using it. It can be used both as a linter or as a formatter, and we can also lose the test_flake8.py file as well.