You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
author = {Clark, Andrew G. and Foster, Michael and Prifling, Benedikt and Walkinshaw, Neil and Hierons, Robert M. and Schmidt, Volker and Turner, Robert D.},
Copy file name to clipboardExpand all lines: paper.md
+19-99Lines changed: 19 additions & 99 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,6 +12,9 @@ authors:
12
12
orcid: 0000-0001-8233-9873
13
13
affiliation: 1
14
14
corresponding: true
15
+
- name: Andrew Clark
16
+
orcid: 0000-0002-6830-0566
17
+
affiliation: 2
15
18
- name: Christopher Wild
16
19
orcid: 0009-0009-1195-1497
17
20
affiliation: 1
@@ -20,14 +23,20 @@ authors:
20
23
- name: Richard Somers
21
24
orcid: 0009-0009-1195-1497
22
25
affiliation: 1
26
+
- name: Nicholas Lattimer
27
+
orcid: 0000-0001-5304-5585
28
+
affiliation: 1
23
29
- name: Neil Walkinshaw
24
30
orcid: 0000-0003-2134-6548
25
31
affiliation: 1
26
-
- name: Nicolas Lattimer
32
+
- name: Rob Hierons
33
+
orcid: 0000-0003-2134-6548
27
34
affiliation: 1
28
35
affiliations:
29
36
- name: University of Sheffield, UK
30
37
index: 1
38
+
- name: Wherever Andy works now, UK
39
+
index: 2
31
40
date: 2 December 2024
32
41
bibliography: paper.bib
33
42
---
@@ -39,21 +48,21 @@ Causal Inference works by using domain knowledge to identify and mitigate for bi
39
48
The Causal Testing Framework is a software testing framework that uses Causal Inference techniques to establish causal effects between software variables from pre-existing runtime data rather than having to collect bespoke, highly curated datasets especially for testing.
40
49
41
50
# Statement of need
42
-
Metamorphic Testing is a popular technique for testing computational models (and other traditionally "hard to test" software).
51
+
Metamorphic Testing @[chen1998metamorphic]is a popular technique for testing computational models (and other traditionally "hard to test" software).
43
52
Test goals are expressed as _metamorphic relations_ that specify how changing an input in a particular way should affect the software output.
44
-
Nondeterministic software can be tested using statistical metamorphic testing, which uses statistical tests over multiple executions of the software to determine whether the specified metamorphic relations hold.
53
+
Nondeterministic software can be tested using Statistical Metamorphic Testing @[guderlei2007smt], which uses statistical tests over multiple executions of the software to determine whether the specified metamorphic relations hold.
45
54
However, this requires the software to be executed repeatedly for each set of parameters of interest, so is computationally expensive, and is constrained to testing properties over software inputs that can be directly and precisely controlled.
46
-
Statistical metamorphic testing cannot be used to test properties that relate internal variables or outputs to each other, since these cannot be controlled a priori.
55
+
Statistical Metamorphic Testing cannot be used to test properties that relate internal variables or outputs to each other, since these cannot be controlled a priori.
47
56
48
57
By employing domain knowledge in the form of a causal graph --- a lightweight model specifying the expected relationships between key software variables --- the Causal Testing Framework circumvents both of these problems by enabling models to be tested using pre-existing runtime data.
49
-
The causal testing framework is written in python but is language agnostic in terms of the system under test.
58
+
The Causal Testing Framework is written in python but is language agnostic in terms of the system under test.
50
59
All that is required is a set of properties to be validated, a causal model, and a set of software runtime data.
51
60
52
61
# Causal Testing
53
-
Causal Testing has four main steps, outlined in \ref{fig:schematic}.
62
+
Causal Testing @[clark2023testing]has four main steps, outlined in \ref{fig:schematic}.
54
63
Firstly, the user supplies a causal model, which takes the form of a directed acyclic graph (DAG) in which an edge $X \to Y$ represents variable $X$ having a direct causal effect on variable $Y$.
55
64
Secondly, the user supplies a set of causal properties to be tested.
56
-
Such properties can be generated from the causal DAG: for each $X \to Y$ edge, a test to validate the presence of a causal effect is generated, and for each missing edge, a test to validate independence is generated.
65
+
Such properties can be generated from the causal DAG @[clark2023metamorphic]: for each $X \to Y$ edge, a test to validate the presence of a causal effect is generated, and for each missing edge, a test to validate independence is generated.
57
66
The user may also refine tests to validate the nature of a particular relationship.
58
67
Next, the user supplies a set of runtime data in the form of a table with each column representing a variable and rows containing the value of each variable for a particular run of the software.
59
68
Finally, the Causal Testing Framework automatically validates the supplied causal properties by using the supplied causal DAG and data to calculate a causal effect estimate, and validating this against the expected causal relationship.
@@ -62,106 +71,17 @@ Finally, the Causal Testing Framework automatically validates the supplied causa
62
71
63
72
## Test Adequacy
64
73
Because the properties being tested are completely separate from the data used to validate them, traditional coverage-based metrics are not appropriate here.
65
-
The Causal Testing Framework instead evaluates the adequacy of a particular dataset by calculating a statistical metric based on the stability of the causal effect estimate, with numbers closer to zero representing more adequate data.
74
+
The Causal Testing Framework instead evaluates the adequacy of a particular dataset by calculating a statistical metric @[foster2024adequacy]based on the stability of the causal effect estimate, with numbers closer to zero representing more adequate data.
66
75
67
76
## Missing Variables
68
77
Causal Testing works by using the supplied causal DAG to identify those variables which need to be statistically controlled for to remove their biassing effect on the causal estimate.
69
78
This typically means we need to know their values.
70
79
However, the Causal Testing Framework can still sometimes estimate unbiased causal effects using Instrumental Variables, an advanced Causal Inference technique.
71
80
72
-
## Feedback
81
+
## Feedback Over Time
73
82
Many scientific models involve iterating several interacting processes over time.
74
83
These processes often feed into each other, and can create feedback cycles.
75
84
Traditional Causal Inference cannot handle this, however the Causal Testing Framework uses another advanced Causal Inference technique, g-methods, to enable the estimation of causal effects even when there are feedback cycles between variables.
Booktitle = {Galactic Dynamics: Second Edition, by James Binney and Scott Tremaine.~ISBN 978-0-691-13026-2 (HB).~Published by Princeton University Press, Princeton, NJ USA, 2008.},
0 commit comments