Merge pull request #650 from aai-institute/fix/dul_notebook

Improvements to DUL and associated notebook

Author: mdbenito

.github/workflows/run-notebook-tests-workflow.yaml

@@ -23,6 +23,11 @@ jobs:
   run-tests:
     runs-on: ubuntu-22.04
     steps:
+    - name: Free Disk Space (Ubuntu)
+      uses: jlumbroso/free-disk-space@main
+      with:
+        large-packages: false
+        docker-images: false
     - uses: actions/checkout@v4
       with:
         fetch-depth: 0

CHANGELOG.md

@@ -5,6 +5,9 @@
 
 ### Added
 
+- Introduced `UtilityModel` and two implementations `IndicatorUtilityModel`
+  and `DeepSetsUtilityModel` for data utility learning
+  [PR #650](https://github.com/aai-institute/pyDVL/pull/650)
 - Introduced the concept of `ResultUpdater` in order to allow samplers to
   declare the proper strategy to use by valuations 
   [PR #641](https://github.com/aai-institute/pyDVL/pull/641)
@@ -56,8 +59,9 @@
 - Fixed `show_warnings=False` not being respected in subprocesses
   [PR #647](https://github.com/aai-institute/pyDVL/pull/647)
 - Fixed several bugs in diverse stopping criteria, including: iteration counts,
-  computing completion and resetting
+  computing completion, resetting, nested composition
   [PR #641](https://github.com/aai-institute/pyDVL/pull/641)
+  [PR #650](https://github.com/aai-institute/pyDVL/pull/650)
 - Fixed all weights of all samplers to ensure that mix-and-matching samplers and
   semi-value methods always works, for all possible combinations
   [PR #641](https://github.com/aai-institute/pyDVL/pull/641)
@@ -88,6 +92,8 @@
   [PR #641](https://github.com/aai-institute/pyDVL/pull/641)
 - Updated Shapley spotify notebook
   [PR #628](https://github.com/aai-institute/pyDVL/pull/628)
+- Updated Data Utility notebook
+  [PR #650](https://github.com/aai-institute/pyDVL/pull/650)
 - Restructured and generalized `StratifiedSampler` to allow using heuristics,
   thus subsuming Variance-Reduced stratified sampling into a unified framework.
   Implemented the heuristics proposed in that paper

notebooks/data_oob.ipynb

@@ -42,8 +42,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "%load_ext autoreload\n",
-    "\n",
     "import os\n",
     "import random\n",
     "\n",

notebooks/influence_imagenet.ipynb

@@ -41,24 +41,6 @@
     "## Imports and setup"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {
-    "editable": true,
-    "nbsphinx": "hidden",
-    "slideshow": {
-     "slide_type": ""
-    },
-    "tags": [
-     "hide"
-    ]
-   },
-   "outputs": [],
-   "source": [
-    "%load_ext autoreload"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": 2,
@@ -74,7 +56,6 @@
    },
    "outputs": [],
    "source": [
-    "%autoreload\n",
     "%matplotlib inline\n",
     "import logging\n",
     "import os\n",

notebooks/influence_sentiment_analysis.ipynb

@@ -45,19 +45,6 @@
     "## Setup"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {
-    "tags": [
-     "hide-input"
-    ]
-   },
-   "outputs": [],
-   "source": [
-    "%load_ext autoreload"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -67,22 +54,13 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "metadata": {
     "tags": [
      "hide-output"
     ]
    },
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/home/jakob/Documents/pyDVL/src/pydvl/parallel/config.py:31: FutureWarning: The `ParallelConfig` class was deprecated in v0.9.0 and will be removed in v0.10.0\n",
-      "  warnings.warn(\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "import os\n",
     "from copy import deepcopy\n",
@@ -667,18 +645,9 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/home/jakob/Documents/pyDVL/venv/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py:222: UserWarning: There is a performance drop because we have not yet implemented the batching rule for aten::masked_fill.Tensor. Please file us an issue on GitHub so that we can prioritize its implementation. (Triggered internally at ../aten/src/ATen/functorch/BatchedFallback.cpp:82.)\n",
-      "  scores = scores.masked_fill(\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
     "ekfac_train_influences = ekfac_influence_model.influences(\n",
     "    test_input,\n",

notebooks/influence_synthetic.ipynb

@@ -60,24 +60,6 @@
     "## Imports"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "ea082430",
-   "metadata": {
-    "editable": true,
-    "slideshow": {
-     "slide_type": ""
-    },
-    "tags": [
-     "hide-input"
-    ]
-   },
-   "outputs": [],
-   "source": [
-    "%load_ext autoreload"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": 2,
@@ -91,7 +73,6 @@
    },
    "outputs": [],
    "source": [
-    "%autoreload\n",
     "%matplotlib inline\n",
     "\n",
     "import os\n",

notebooks/influence_wine.ipynb

@@ -40,23 +40,6 @@
     "## Imports"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "cef17bfc",
-   "metadata": {
-    "slideshow": {
-     "slide_type": ""
-    },
-    "tags": [
-     "hide-input"
-    ]
-   },
-   "outputs": [],
-   "source": [
-    "%load_ext autoreload"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": 2,
@@ -71,7 +54,6 @@
    },
    "outputs": [],
    "source": [
-    "%autoreload\n",
     "%matplotlib inline\n",
     "\n",
     "import os\n",