Merge pull request #62 from Quantmetry/dev

Dev

Author: JulienRoussel77

.github/workflows/test.yml

@@ -1,6 +1,12 @@
 name: Unit test Qolmat
 
-on: [push, pull_request, workflow_dispatch]
+on:
+  push:
+    branches:
+      -dev
+      -main
+  pull_request:
+  workflow_dispatch:
 
 jobs:
   build-linux:

CONTRIBUTING.rst

@@ -36,7 +36,7 @@ If you need to use tensorflow, enter the command:
 
 .. code:: sh
 
-    $ pip install -e .[tensorflow]
+    $ pip install -e .[pytorch]
 
 Once the environment is installed, pre-commit is installed, but need to be activated using the following command:
 

HISTORY.rst

@@ -1,6 +1,19 @@
 =======
 History
 =======
+
+0.0.15 (2023-??-??)
+-------------------
+
+* Hyperparameters are now optimized in hyperparameters.py, with the maintained module hyperopt
+* The Imputer classes do not possess a dictionary attribute anymore, and all list attributes have
+been changed into tuple attributes so that all are not immutable
+* All the tests from scikit-learn's check_estimator now pass for the class Imputer
+* Fix MLP imputer, created a builder for MLP imputer
+* Switch tensorflow by pytorch. Change Test, environment, benchmark and imputers for pytorch
+* Add new datasets
+* Added dcor metrics with a pattern-wise computation on data with missing values
+
 0.0.14 (2023-06-14)
 -------------------
 

README.rst

@@ -62,7 +62,7 @@ To install directly from the github repository :
 
 Let us start with a basic imputation problem. Here, we generate one-dimensional noisy time series.
 
-.. code:: sh
+.. code-block:: python
 
     import matplotlib.pyplot as plt
     import numpy as np
@@ -75,7 +75,7 @@ Let us start with a basic imputation problem. Here, we generate one-dimensional
 
 For this demonstration, let us create artificial holes in our dataset.
 
-.. code:: sh
+.. code-block:: python
 
     from qolmat.utils.data import add_holes
     plt.rcParams.update({'font.size': 18})
@@ -101,7 +101,7 @@ For this demonstration, let us create artificial holes in our dataset.
 To impute missing data, there are several methods that can be imported with ``from qolmat.imputations import imputers``.
 The creation of an imputation dictionary will enable us to benchmark the various imputations.
 
-.. code:: sh
+.. code-block:: python
 
     from sklearn.linear_model import LinearRegression
     from qolmat.imputations import imputers
@@ -146,7 +146,7 @@ The creation of an imputation dictionary will enable us to benchmark the various
 
 It is possible to define a parameter dictionary for an imputer with three pieces of information: min, max and type. The aim of the dictionary is to determine the optimal parameters for data imputation. Here, we call this dictionary ``dict_config_opti``.
 
-.. code:: sh
+.. code-block:: python
 
     search_params = {
         "RPCA_opti": {
@@ -157,7 +157,7 @@ It is possible to define a parameter dictionary for an imputer with three pieces
 
 Then with the comparator function in ``from qolmat.benchmark import comparator``, we can compare the different imputation methods. This **does not use knowledge on missing values**, but it relies data masking instead. For more details on how imputors and comparator work, please see the following `link <https://qolmat.readthedocs.io/en/latest/explanation.html>`_.
 
-.. code:: sh
+.. code-block:: python
 
     from qolmat.benchmark import comparator
 
@@ -175,7 +175,7 @@ Then with the comparator function in ``from qolmat.benchmark import comparator``
 
 We can observe the benchmark results.
 
-.. code:: sh
+.. code-block:: python
 
     dfs_imputed =  imputer_tsmle.fit_transform(df_with_nan)
 
@@ -196,7 +196,7 @@ We can observe the benchmark results.
 
 Finally, we keep the best ``TSMLE`` imputor we represent.
 
-.. code:: sh
+.. code-block:: python
 
     dfs_imputed =  imputer_tsmle.fit_transform(df_with_nan)
 

environment.ci.yml

@@ -6,12 +6,13 @@ dependencies:
     - pip=23.0.1
     - pip:
           - codecov
-          - flake8==6.0.0
-          - matplotlib==3.6.2
-          - mypy==1.1.1
-          - numpydoc==1.5.0
-          - pytest==7.2.0
-          - pytest-cov==4.0.0
-          - pytest-mock==3.10.0
-          - tensorflow
+          - flake8
+          - matplotlib
+          - mypy
+          - numpy
+          - numpydoc
+          - pytest
+          - pytest-cov
+          - pytest-mock
+          - torch==2.0.1
           - -e .

environment.dev.yml

@@ -5,17 +5,18 @@ channels:
 dependencies:
     - bump2version=1.0.1
     - dcor=0.6
-    - ipykernel=5.1.4
+    - ipykernel=6.21.0
     - jupyter=1.0.0
     - jupyterlab=1.2.6
     - jupytext=1.14.4
-    - numpy=1.21
+    - hyperopt=0.2.7
+    - numpy=1.24.4
     - packaging=23.1
     - pandas=2.0.1
+    - python=3.8
     - pip=23.0.1
     - scipy=1.10.1
     - scikit-learn=1.2.2
-    - scikit-optimize=0.9
     - sphinx=6.2.1
     - sphinx-gallery=0.13.0
     - sphinx_rtd_theme=1.2.0

examples/RPCA.md

@@ -1,15 +1,16 @@
 ---
 jupyter:
   jupytext:
+    formats: ipynb,md
     text_representation:
       extension: .md
       format_name: markdown
       format_version: '1.3'
       jupytext_version: 1.14.4
   kernelspec:
-    display_name: env_qolmat
+    display_name: Python 3 (ipykernel)
     language: python
-    name: env_qolmat
+    name: python3
 ---
 
 ```python
@@ -28,33 +29,27 @@ from math import pi
 from qolmat.utils import plot, data
 from qolmat.imputations.rpca.rpca_pcp import RPCAPCP
 from qolmat.imputations.rpca.rpca_noisy import RPCANoisy
+from qolmat.utils.data import generate_artificial_ts
 ```
 
 **Generate synthetic data**
 
 ```python
 n_samples = 1000
+periods = [100, 20]
+amp_anomalies = 0.5
+ratio_anomalies = 0.05
+amp_noise = 0.1
 
-mesh = np.arange(n_samples)
-X_true = np.zeros(n_samples)
-A_true = np.zeros(n_samples)
-E_true = np.zeros(n_samples)
-p1 = 100
-p2 = 20
-X_true = 1 + np.sin(2 * pi * mesh / p1) + np.sin(2 * pi * mesh / p2)
-noise = np.random.uniform(size=n_samples)
-amplitude_A = .5
-freq_A = .05
-A_true = amplitude_A * np.where(noise < freq_A, -np.log(noise), 0) * (2 * (np.random.uniform(size=n_samples) > .5) - 1)
-amplitude_E = .1
-E_true = amplitude_E * np.random.normal(size=n_samples)
-
-signal = X_true + E_true
-signal[A_true != 0] = A_true[A_true != 0]
-signal = signal.reshape(-1, 1)
+X_true, A_true, E_true = generate_artificial_ts(n_samples, periods, amp_anomalies, ratio_anomalies, amp_noise)
+
+signal = X_true + A_true + E_true
 
 # Adding missing data
-signal[5:20, 0] = np.nan
+#signal[5:20] = np.nan
+mask = np.random.choice(len(signal), round(len(signal) / 20))
+signal[mask] = np.nan
+
 ```
 
 ```python
@@ -73,7 +68,7 @@ plt.plot(E_true)
 
 ax = fig.add_subplot(4, 1, 4)
 ax.title.set_text("Corrupted signal")
-plt.plot(signal[:, 0])
+plt.plot(signal)
 
 plt.show()
 ```
@@ -82,26 +77,29 @@ plt.show()
 
 ```python
 %%time
+rpca_pcp = RPCAPCP(period=100, max_iterations=100, mu=.5, lam=0.1)
+X, A = rpca_pcp.decompose_rpca_signal(signal)
+imputed = signal - A
+```
 
-rpca_pcp = RPCAPCP(period=100, max_iter=5, mu=.5, lam=1)
-X = rpca_pcp.fit_transform(signal)
-corruptions = signal - X
+```python
+fig = plt.figure(figsize=(12, 4))
+plt.plot(X, color="black")
+plt.plot(imputed)
 ```
 
 ## Temporal RPCA
 
 ```python
-rpca_noisy = RPCANoisy(period=10, tau=2, lam=0.3, list_periods=[10], list_etas=[0.01], norm="L2")
-X = rpca_noisy.fit_transform(signal)
-corruptions = signal - X
-plot.plot_signal([signal[:,0], X[:,0], corruptions[:, 0]])
+%%time
+rpca_noisy = RPCANoisy(period=10, tau=1, lam=0.4, list_periods=[10], list_etas=[0.01], norm="L2")
+X, A = rpca_noisy.decompose_rpca_signal(signal)
 ```
 
 ```python
-rpca_noisy = RPCANoisy(period=10, tau=2, lam=0.3, list_periods=[], list_etas=[], norm="L2")
-X = rpca_noisy.fit_transform(signal)
-corruptions = signal - X
-plot.plot_signal([signal[:,0], X[:,0], corruptions[:, 0]])
+fig = plt.figure(figsize=(12, 4))
+plt.plot(X, color="black")
+plt.plot(imputed)
 ```
 
 ```python