dxCompiler `-projectWideReuse` flag does not pick up changes to Structs, causing RuntimeError

[Ben-Habermeyer]

Background

We compile our WDL workflows using -projectWideReuse argument, to only re-compile applets whose underlying source code or docker image runtimes have changed.

Recently we have come across an issue where applets which take as input a Struct type do not get re-compiled when the Struct is changed. This later leads to a RuntimeError as there are extra or missing keys from what the applet expects.

For example, a task such as the following fails if any of the structs are edited to contain additional or less content

java.lang.Exception: keys (Set(contam_pass_status, discordance, effective_panel_depth, error_rate, median_insert_size, negative_control_log_likelihood_ratio, positive_control_log_likelihood_ratio)) have members that do not appear in struct MetricThresholds

task write_metrics_and_thresholds_json {
    input {
        SampleMetrics sample_metrics
        MetricThresholds failure_metric_thresholds
        WarningMetricThresholds warning_metric_thresholds
        String sample_name
    }
    command <<<
        # Comment to force rebuild
        set -exo pipefail
        
        cat "~{write_json(sample_metrics)}" > ~{sample_name}.sample_metrics.json
        cat "~{write_json(failure_metric_thresholds)}" > ~{sample_name}.failure_metric_thresholds.json
        cat "~{write_json(warning_metric_thresholds)}" > ~{sample_name}.warning_metric_thresholds.json
    >>>
    output {
        File sample_metrics_json = "~{sample_name}.sample_metrics.json"
        File failure_metric_thresholds_json = "~{sample_name}.failure_metric_thresholds.json"
        File warning_metric_thresholds_json = "~{sample_name}.warning_metric_thresholds.json"
    }
}

The workaround is to manually add some comment or whitespace to the WDL to tasks like this every time the a struct is edited, such that their applets are rebuilt. Or always rebuild all applets.

Requested change

dxCompiler should check that the types to input tasks have not changed (specifically when accepting Struct inputs) and recompile the applets if their definition has changed.