small changes to poisson_line_process.ipynb

Author: f-allian

examples/poisson-line-process/poisson_line_process.ipynb

@@ -19,16 +19,23 @@
    "id": "56965fba-b90b-4233-a819-bb747ecd9d81",
    "metadata": {},
    "source": [
-    "# Poisson Line Process Case Study: Statistical Metamorphic Testing"
+    "# Poisson Line Process Tutorial: Statistical Metamorphic Testing"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5adf7cdc-fd96-47a4-a194-f1f060a4c0c5",
+   "metadata": {},
+   "source": [
+    "##  Overview"
    ]
   },
   {
    "cell_type": "markdown",
    "id": "5b5f1661-9d24-43e3-88ee-add75b744e87",
    "metadata": {},
    "source": [
-    "##  Overview\n",
-    "In this tutorial, we demonstrate using the **core Python API** how the Causal Testing Framework (CTF) can be employed to implement statistical metamorphic testing. Broadly speaking, this example involves running a series of causal test cases that incrementally change the width and height of the sampling window of a Poisson Line Tessellation (PLT) model. We then show how statistical estimation can produce similar results using only a fraction of the data. Further details on the methodology can be found in Section 5.1 of our [paper](https://dl.acm.org/doi/10.1145/3607184)."
+    "The purpose of this tutorial is to use the Causal Testing Framework's **core Python API** to demonstrate how it can be employed to implement statistical metamorphic testing. More specifically, this example involves running a series of causal test cases that incrementally change the width and height of the sampling window of a Poisson Line Tessellation (PLT) model. Further details on the methodology can be found in Section 5.1 of our [paper](https://dl.acm.org/doi/10.1145/3607184) and additional resources for this tutorial can be found at the end of this notebook."
    ]
   },
   {
@@ -38,7 +45,7 @@
    "source": [
     "### Step  1: Defining your Input Configurations\n",
     "\n",
-    "A good first step is to define your file paths, including your input configurations:"
+    "Before diving into the details, a good first step is to define your file paths, including your input configurations:"
    ]
   },
   {
@@ -80,7 +87,9 @@
     "\n",
     "- `Meta` variables, which are not directly observable but can be related to the system.\n",
     "\n",
-    "Secondly, the causal DAG encodes information about the expected causal structure of the system through nodes representing variables and directed edges representing causal relationships, which is a model of how the data could have been generated. Together, the Causal DAG and modelling scenario form the `Causal Specification`."
+    "Secondly, the causal DAG encodes information about the expected causal structure of the system through nodes representing variables and directed edges representing causal relationships, which is a model of how the data could have been generated. Together, the Causal DAG and modelling scenario form the `Causal Specification`.\n",
+    "\n",
+    "**Note**: The CTF doesn't support native visualisation tools, but it is possible to use existing frameworks such as NetworkX to visualise your DAG. Alternatively, browser-based environments such as [DAGitty](https://www.dagitty.net/) may also be useful."
    ]
   },
   {
@@ -96,12 +105,12 @@
    "id": "40f85921-40b7-46e5-aede-606900582f4b",
    "metadata": {},
    "source": [
-    "At this point, it might be worthwhile to interrogate your data and apply any pre-processing, transforming or cleaning as necessary. However, for the purposes of this example, there won't be any additional processing required. Section 5.13 of the paper explains how this dataset was generated."
+    "At this point, it might be worthwhile to interrogate your data and apply any pre-processing, transforming or cleaning as necessary. However, for the purposes of this tutorial, there won't be any additional processing required. Section 5.13 of our [paper](https://dl.acm.org/doi/10.1145/3607184) explains in detail how this dataset was generated."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 3,
    "id": "d7d27532-7995-4d76-b40e-e6ae9e7cc645",
    "metadata": {},
    "outputs": [
@@ -281,7 +290,7 @@
        "[1000 rows x 7 columns]"
       ]
      },
-     "execution_count": 2,
+     "execution_count": 3,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -304,7 +313,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 4,
    "id": "ac297d2d-5a2f-4c33-bbdc-967d54e24e3f",
    "metadata": {},
    "outputs": [],
@@ -352,18 +361,24 @@
     "causal_specification = CausalSpecification(scenario, causal_dag) "
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "877d413d-ff96-4481-953f-891c19493531",
+   "metadata": {},
+   "source": [
+    "### Step 3: Create Causal Test Cases"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "be854667-44de-4f40-a37d-fb35588f047a",
    "metadata": {},
    "source": [
-    "### Step 3: Create a Causal Test Case\n",
-    "   \n",
     "Now that we've created our Causal Specification, we're ready to create our Causal Tests. Causal tests are essentially metamorphic tests that are executed using statistical causal inference. A causal test expresses the change in a given output that we expect to see when we change a particular input in some way. \n",
     "\n",
     "Firstly, a `base test case`, which specifies the relationship between the given output and input and the desired effect, is required to build a `causal test case`. Together, the causal test case forms the complete executable test, which is the minimum required to perform identification on the DAG.\n",
     "\n",
-    "The two metamorphic relations we would like to investigate are the following:\n",
+    "In this tutorial, the two metamorphic relations we would like to investigate are the following:\n",
     "\n",
     "1. Doubling the intensity should cause the number of polygons per unit area to increase by a factor of 4.\n",
     "2. The number of polygons per unit area should be independent of width and height."
@@ -379,7 +394,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 5,
    "id": "9b8491ab-0a90-4061-baee-8e1ecef7371d",
    "metadata": {},
    "outputs": [],
@@ -400,24 +415,24 @@
    "id": "e8026067-4df6-43f4-8927-6ac9415b9232",
    "metadata": {},
    "source": [
-    "Following this, we can now create our causal test case. The minimum parameter's well need to create this are: the expected causal effect as a `CausalEffect` object (e.g. `ExactValue`), the estimate type, which is a `str` specifying the type of estimate to return, and an estimator, which can be is an `Estimator` object. Note, since the relation we're investigating is inherently linear, we can use the `LinearRegressionEstimator` class to build our causal test case."
+    "Following this, we can now create our causal test case. The minimum parameter's well need to create this are: the expected causal effect as a `CausalEffect` object (e.g. `ExactValue`), the estimate type, which is a `str` specifying the type of estimate to return, and an estimator, which can be is an `Estimator` object. Since the relation we're investigating is inherently linear, we can use the `LinearRegressionEstimator` class to build our causal test case."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 6,
    "id": "fa53a888-68e1-4f6f-babf-16d3a206ea49",
    "metadata": {},
    "outputs": [],
    "source": [
     "import numpy as np\n",
     "from causal_testing.estimation.linear_regression_estimator import LinearRegressionEstimator\n",
     "\n",
-    "control_values, treatment_values = 2 ** np.arange(0, 4), 2 ** np.arange(1, 5)\n",
+    "control_values, treatment_values = 2 ** np.arange(0, 4), 2 ** np.arange(1, 5) # Initialise the dummy intensity variables\n",
     "\n",
-    "intensity_results = []\n",
+    "intensity_results = [] # Initiate an empty list to store the causal test results\n",
     "\n",
-    "for (control, treatment) in zip(control_values, treatment_values):\n",
+    "for (control, treatment) in zip(control_values, treatment_values): # Simultaneously loop over control and treatment\n",
     "    \n",
     "    estimator=LinearRegressionEstimator(\n",
     "                    df=df, # Pass in the dataframe\n",
@@ -453,12 +468,12 @@
    "id": "d6793dc5-6425-4722-b430-f57aaf3fd181",
    "metadata": {},
    "source": [
-    "Finally, we can parse the causal test results as a `pandas` dataframe and export it to a `.csv` file."
+    "Finally, we can parse the causal test results as a `pandas` dataframe and optionally export it to a `.csv` file."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 7,
    "id": "6bc8be40-bc95-4187-8771-4ce096acc7b5",
    "metadata": {},
    "outputs": [
@@ -535,7 +550,7 @@
        "3      8      16        8         16    3.699311"
       ]
      },
-     "execution_count": 6,
+     "execution_count": 7,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -548,12 +563,20 @@
     "# intensity_results_df.to_csv(\"intensity_test_results.csv\", index=0) # Uncomment this to save as a csv."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "f8ad21d8-6451-4f6c-a69d-cb939ea4f96b",
+   "metadata": {},
+   "source": [
+    "### Summary"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "e62216b2-60ed-49b7-a2eb-0ad755bd91fc",
    "metadata": {},
    "source": [
-    "From the above causal test results and the risk ratios, we can conclude that doubling the intensity **does not** cause the number of polygons per unit area to increase by a factor of 4 as we expected - but by factors ranging from 2.8 - 3.7."
+    "From the above causal test results and the risk ratios, we can conclude that doubling the intensity **does not** cause the number of polygons per unit area to increase by a factor of 4 as we expected - but by factors ranging from 2.8 - 3.7, meaning that the metamorphic relation is not satisfied. This is a significant result since our hypothesis was that "
    ]
   },
   {
@@ -571,23 +594,23 @@
    "source": [
     "In a very similar way to the method above, we can test our second metamorphic relation that the number of polygons per unit area should be independent of sample width and height. Since we are only interested in whether there is some effect, we use the average treatment effect (ATE) instead of the risk ratio from above, which quantifies the additive change in outcome caused by the intervention.\n",
     "\n",
-    "To investigate whether the width affects number of polygons per unit area, we need to execute a new set of test cases, but this time fixing the intensity and varying the width. Note: we don't need to redefine the causal specification, nor the perform identification again; but we have to redefine our base test case."
+    "To investigate whether the width affects number of polygons per unit area, we need to execute a new set of test cases, but this time fixing the intensity and varying the width. Note: we don't need to redefine the causal specification, nor the perform identification again; but we have to redefine our base test case since we're now considering the Polygon's width as the treatment variable."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 8,
    "id": "67bf5061-720f-4b3a-a371-3ff3092e81e1",
    "metadata": {},
    "outputs": [],
    "source": [
-    "control_values, treatment_values = np.arange(1,10), np.arange(1, 17)\n",
+    "control_values, treatment_values = np.arange(1,10), np.arange(1, 17) # Initialise the dummy width variables\n",
     "\n",
-    "width_results = [] # Empty list for storing test case results\n",
+    "width_results = [] # Empty list for storing test case results \n",
     "\n",
     "base_test_case = BaseTestCase(treatment_variable=width, outcome_variable=num_shapes_unit) # Create the base test case\n",
     "\n",
-    "adjustment_set = causal_specification.causal_dag.identification(base_test_case)\n",
+    "adjustment_set = causal_specification.causal_dag.identification(base_test_case) # Calculate the adjustment set again (if it exists)\n",
     "\n",
     "for intensity in treatment_values:\n",
     "   \n",
@@ -626,7 +649,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
+   "execution_count": 9,
    "id": "6c54392c-4e6b-42b3-b39a-e0d1d0ab25b7",
    "metadata": {},
    "outputs": [
@@ -768,7 +791,7 @@
        "9      1.0        2.0          2 -7.378642 -16.381136   1.623851"
       ]
      },
-     "execution_count": 12,
+     "execution_count": 9,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -779,20 +802,30 @@
     "width_results_df.head(10)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "8c085e53-a56e-4f4a-b273-e6af46beba72",
+   "metadata": {},
+   "source": [
+    "### Summary"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "7fcfe837-5d71-4603-b8a5-d3e3f50ceccc",
    "metadata": {},
    "source": [
-    "The causal test results in this case demonstrate that the ATE values for width increases from 1 → 2 through 9 → 10, revealing that while most changes produce non-significant effects (ATEs ranging from -2.7097 to -0.5308 with confidence intervals containing zero), the width change from 1 → 2 produces a statistically significant negative effect of -7.3786 with a confidence interval of [-13.9182, -0.8390]. This either indicates there is a problem with either the program, or the metamorphic property itself. A likely interpretation is that, geometrically, lines are less likely to intersect a smaller sample window. As the sample window becomes larger, there is more area to average over. Therefore, the metamorphic relations should ideally specify a minimum window size to which they apply."
+    "The causal test results in this case demonstrate that the ATE values for width increases from `1 → 2` through `9 → 10`, revealing that while most changes produce non-significant effects (ATEs ranging from `-2.7097` to `-0.5308` with confidence intervals containing zero), the width change from `1 → 2` produces a statistically significant negative effect of `-7.3786` with a confidence interval of `[-13.9182, -0.8390]`. This either indicates there is a problem with either the program, or the metamorphic property itself. A likely interpretation is that, geometrically, lines are less likely to intersect a smaller sample window. As the sample window becomes larger, there is more area to average over. Therefore, the metamorphic relations should ideally specify a minimum window size to which they apply.\n",
+    "\n",
+    "Additionally, in the paper we further demonstrate that these results show that the CTF was able to identify the same discrepancy as conventional statistical metamorphic testing, but using only a fifth of the data. Ultimately, this highlights the potential of causal inference-driven approaches to offer economical alternatives to testing techniques that depend on repeated potentially costly executions of the system under test."
    ]
   },
   {
    "cell_type": "markdown",
    "id": "41545617-60e9-468d-a356-9dc1f433953d",
    "metadata": {},
    "source": [
-    "# Resources"
+    "## Additional Resources"
    ]
   },
   {