Feature to add completely uncovered files to the coverage report #335
Description
In the generated report, Devel::Cover completely ignores files which have no coverage (because no part of the test suite ever ran them). For our use case, this makes the report a bit deceptive and dangerous because it gives the impression that we have reasonable coverage, when actually we have scary big holes, where we shouldn't be refactoring (yet)
So I wondereded if it's possible to force uncovered files to be treated as 0%. I asked about this on the london.pm list but it seems no-one had attempted this. So here goes...
I hacked together a script for work - cover-touch.pl
Without this the coverage report looks like this:
file stmt bran cond sub pod time total
bin/batch_monitoring.pl 98.1 83.3 n/a 93.3 n/a 0.3 96.0
bin/check_payment_notification.pl 98.4 75.0 44.4 93.7 n/a 0.1 86.2
bin/data_export_to_storage.pl 89.6 37.5 n/a 88.2 n/a 0.5 84.3
bin/monitor_payment_files.pl 93.1 75.0 n/a 100.0 n/a 0.0 91.6
bin/run_shasum.pl 97.7 75.0 n/a 92.8 n/a 0.0 95.2
bin/statement_monitoring.pl 90.0 78.5 47.6 89.4 n/a 0.6 81.4
lib/Processor/BankStatement.pm 98.2 75.0 66.6 100.0 100.0 0.1 95.1
...
lib/app/StatementMonitoring/UK.pm 100.0 n/a n/a 100.0 0.0 0.4 85.0
lib/logic/BankingDays.pm 98.0 91.6 100.0 100.0 0.0 1.0 94.7
lib/notify_slack.pl 92.0 50.0 n/a 100.0 n/a 0.3 88.5
lib/send_sms.pl 97.5 93.7 100.0 100.0 n/a 0.3 96.7
Total 94.9 74.5 58.1 95.8 11.4 100.0 88.7
With this:
file stmt bran cond sub pod time total
bin/batch_monitoring.pl 98.1 83.3 n/a 93.3 n/a 0.3 96.0
bin/check_bank_holiday.pl n/a n/a n/a n/a n/a n/a 0.0
bin/check_payment_notification.pl 98.4 75.0 44.4 93.7 n/a 0.1 86.2
bin/data_export_to_storage.pl 89.6 37.5 n/a 88.2 n/a 0.6 84.3
bin/gen_directories.pl n/a n/a n/a n/a n/a n/a 0.0
bin/import_statement_from_csv.pl n/a n/a n/a n/a n/a n/a 0.0
bin/manual_fail_rbs_transaction.pl n/a n/a n/a n/a n/a n/a 0.0
bin/monitor_payment_files.pl 93.1 75.0 n/a 100.0 n/a 0.0 91.6
bin/next_bank_holiday.pl n/a n/a n/a n/a n/a n/a 0.0
bin/rbs_narrative_soaker.pl n/a n/a n/a n/a n/a n/a 0.0
bin/read_links.pl n/a n/a n/a n/a n/a n/a 0.0
bin/run_shasum.pl 97.7 75.0 n/a 92.8 n/a 0.0 95.2
bin/statement_monitoring.pl 90.0 78.5 47.6 89.4 n/a 0.7 81.4
lib/Processor/BankStatement.pm 98.2 75.0 66.6 100.0 100.0 0.1 95.1
...
lib/app/StatementMonitoring/UK.pm 100.0 n/a n/a 100.0 0.0 0.5 85.0
lib/commify.pl n/a n/a n/a n/a n/a n/a 0.0
lib/logic/BankingDays.pm 98.0 91.6 100.0 100.0 0.0 1.0 94.7
lib/notify_slack.pl 92.0 50.0 n/a 100.0 n/a 0.3 88.5
lib/send_sms.pl 97.5 93.7 100.0 100.0 n/a 0.3 96.7
Total 94.9 74.5 58.1 95.8 11.4 100.0 88.7
we see that the UNKNOWN UNKNOWNS are now improved to KNOWN UNKNOWNS.
(Output stolen from the talk http://act.yapc.eu/gpw2024/talk/7872 - video not online yet)
I have no idea if this is generating "correct" (let alone "optimal") cover_db runs, but it does seem to do the job. However, the report generation code then spews bazillions of use of uninitialized value warnings because some part of a data structure is (partially) created. It is not expecting this - it appears that the "outer" level reference that now exists for that file causes code to be entered that wan't previously, but this code then assumes a multi-level structure exists with values for statement, branch and condition coverage, and so is reading undef when it expects a number.
This patch "shuts it up" but I don't think that it's really correct, as we end up with the "average number" for the file being 0.0 (seen above) whereas I'd like it to be n/a
diff --git a/lib/perl5/site_perl/5.32.0/x86_64-linux/Devel/Cover/Report/Html_minimal.pm b/lib/perl5/site_perl/5.32.0/x86_64-linux/Devel/Cover/Report/Html_minimal.pm
index ede35908d..5ffa77f0f 100644
--- a/lib/perl5/site_perl/5.32.0/x86_64-linux/Devel/Cover/Report/Html_minimal.pm
+++ b/lib/perl5/site_perl/5.32.0/x86_64-linux/Devel/Cover/Report/Html_minimal.pm
@@ -482,10 +482,14 @@ sub print_file_report {
my ($show, $th) = get_showing_headers($db, $opt);
my $file_data = $db->cover->file($fin);
+ # If we have injected files into the DB with no coverage, we don't want to
+ # accidentally autovivify a "total" structure for them, as creating the
+ # hashref here "confuses" get_summary_for_file().
+ my $total = $db->{summary}{$fin}{total};
print_html_header($out, "File Coverage: $fin");
print_summary($out, 'File Coverage', $fin,
- $db->{summary}{$fin}{total}{percentage},
- $db->{summary}{$fin}{total}{error},
+ $total->{percentage},
+ $total->{error},
$db);
print_th($out, ['line', @$th, 'code']);
(I believe there's at least one talk on YouTube that says work is on 5.32.0 ("this week"), so this isn't a leak)
I'm not sure where to take this from here. I guess
- add (yet another) option to
coverto create these placeholder runs - fix the report generation not to warn (roughly that patch)
- improve the report generation to output
n/a n/a n/a n/a n/a n/a n/a(instead ofn/a n/a n/a n/a n/a n/a 0.0) - ideally treat the average calculation so that a file with that last "total" column of
n/acounts as part of the denominator
I don't know how to submit a patch to do any of this. Or even if it's the right plan.
Right now (I think) if I have 5 files with totals of 0.6, 0.7, 0.8, 0.9 and 1.0, and 5 files completly uncovered, my overal "average" is 0.8, which is highlighted as orange.
With my hack, those 5 files will now show up, but the averaging ignores them. I'd rather like the averaging to treat the overal totals as 0.4, which is very red.
It's actually a bit of a "bug" - right now if I delete the test for a module with low coverage, my reported average coverage goes up. Game the system!