Test coverage support

[bergie]

It would be nice to be able to get test coverage reports from fbp-spec. Something like https://www.npmjs.com/package/nyc

[bergie]

Note: nyc claims suppor for applications that spawn subprocesses so fbp-spec might be possible to make work.

[jonnor]

Does nyc produce a report file somewhere from running the tests? Have you tried wrapping the runtime command in it to see if it just works?

[jonnor]

Generally this is something that needs to happen on the runtime side, and in a runtime-specific manner, as how to compute coverage depends entirely on the language/execution-environment in use.

[bergie]

@jonnor tried wrapping fbp-spec (via its mocha interface) to nyc, and at least that didn't work:

noflo/noflo-core#108

Getting zero coverage for the components we test via fbp-spec (but normal coverage for the ones we test via Mocha)

[jonnor]

@bergie what do you mean does not work? Looks fine here

[bergie]

@jonnor yes, see zero coverage reported for Repeat and RepeatAsync, which are fbp-spec tested. Everything else is tested via plain Mocha

[jonnor]

@bergie ok. Does not work when running nyc infront of fbp-spec either, does not pick up any files at all.
So either subprocess tracking is off, it cannot handle the indirect file loading (though noflo.ComponentLoader seems to work for other), there is a problem with .js versus .coffee files or similar

[bergie]

@jonnor for .coffee files you will need to require coffee-coverage

See https://github.com/benbria/coffee-coverage/blob/master/docs/HOWTO-nyc.md

[jonnor]

Presumably that needs to be done in the process which uses the coffee files. Which would explain why it is not working in fbp-spec case: It is the runtime that requires the files (not fbp-spec), and there is no such registration there. noflo-nodejs does not have such an option either, I think?

[bergie]

@jonnor correct, we should probably add a mocha-style --require option to noflo-nodejs.

But first to verify this, it might make sense to add this manually to noflo-nodejs

Update: tried patching that to noflo-nodejs locally, and still got zero coverage