make release-tag: Merge branch 'master' into stable

sarahmish · sarahmish · commit ebee314d5b30 · 2023-08-17T15:57:10.000-04:00
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -52,6 +52,9 @@ jobs:
       matrix:
         python-version: [3.6, 3.7, 3.8]
         os: [ubuntu-20.04, macos-latest]
+        exclude:
+          - os: macos-latest
+            python-version: '3.7'
     steps:
     - uses: actions/checkout@v1
     - name: Set up Python ${{ matrix.python-version }}
@@ -70,6 +73,9 @@ jobs:
       matrix:
         python-version: [3.6, 3.7, 3.8]
         os: [ubuntu-20.04, macos-latest, windows-latest]
+        exclude:
+          - os: macos-latest
+            python-version: '3.7'
     steps:
     - uses: actions/checkout@v1
     - name: Set up Python ${{ matrix.python-version }}
@@ -88,6 +94,9 @@ jobs:
       matrix:
         python-version: [3.6, 3.7, 3.8]
         os: [ubuntu-20.04, macos-latest]
+        exclude:
+          - os: macos-latest
+            python-version: '3.7'
     steps:
     - uses: actions/checkout@v1
     - name: Set up Python ${{ matrix.python-version }}
@@ -106,6 +115,9 @@ jobs:
       matrix:
         python-version: [3.6, 3.7, 3.8]
         os: [ubuntu-20.04, macos-latest, windows-latest]
+        exclude:
+          - os: macos-latest
+            python-version: '3.7'
     steps:
     - uses: actions/checkout@v1
     - name: Set up Python ${{ matrix.python-version }}
diff --git a/HISTORY.md b/HISTORY.md
@@ -1,6 +1,17 @@
 History
 =======
 
+## 0.5.1 - 2023-08-16
+
+This version introduces a new dataset to the benchmark.
+
+### Issues resolved
+
+* Add UCR dataset to the benchmark – [Issue #443](https://github.com/signals-dev/Orion/issues/443) by @sarahmish
+* docker image build failed – [Issue #439](https://github.com/signals-dev/Orion/issues/439) by @sarahmish
+* Edit interval settings in ``azure`` pipeline – [Issue #436](https://github.com/signals-dev/Orion/issues/436) by @sarahmish
+
+
 ## 0.5.0 - 2023-05-23
 
 This version uses ``ml-stars`` package instead of ``mlprimitives``.
diff --git a/README.md b/README.md
@@ -124,17 +124,17 @@ The output of the previous command will be a ``pandas.DataFrame`` containing a t
 # Leaderboard
 In every release, we run Orion benchmark. We maintain an up-to-date leaderboard with the current scoring of the verified pipelines according to the benchmarking procedure.
 
-We run the benchmark on **11** datasets with their known grounth truth. We record the score of the pipelines on each datasets. To compute the leaderboard table, we showcase the number of wins each pipeline has over the ARIMA pipeline.
+We run the benchmark on **12** datasets with their known grounth truth. We record the score of the pipelines on each datasets. To compute the leaderboard table, we showcase the number of wins each pipeline has over the ARIMA pipeline.
 
 | Pipeline                  |  Outperforms ARIMA |
 |---------------------------|--------------------|
-| AER                       |         10         |
+| AER                       |         11         |
 | TadGAN                    |          7         |
 | LSTM Dynamic Thresholding |          7         |
-| LSTM Autoencoder          |          6         |
-| Dense Autoencoder         |          6         |
-| VAE                       |          6         |
-| [GANF](https://arxiv.org/pdf/2202.07857.pdf)                                                  |          6         |
+| LSTM Autoencoder          |          8         |
+| Dense Autoencoder         |          7         |
+| VAE                       |          7         |
+| [GANF](https://arxiv.org/pdf/2202.07857.pdf)                                                  |          7         |
 | [Azure](https://azure.microsoft.com/en-us/products/cognitive-services/anomaly-detector/)      |          0         |
 
 
diff --git a/benchmark/leaderboard.md b/benchmark/leaderboard.md
@@ -1,10 +1,10 @@
 | Pipeline                  |  Outperforms ARIMA |
 |---------------------------|--------------------|
-| AER					    |          10        |
+| AER					    |          11        |
 | TadGAN					|          7         |
 | LSTM Dynamic Thresholding |          7         |
-| LSTM Autoencoder			|          6         |
-| Dense Autoencoder			|          6         |
-| VAE					    |          6         |
-| GANF					    |          6         |
+| LSTM Autoencoder			|          8         |
+| Dense Autoencoder			|          7         |
+| VAE					    |          7         |
+| GANF					    |          7         |
 | Azure						|          0         |
diff --git a/benchmark/leaderboard.xlsx b/benchmark/leaderboard.xlsx
diff --git a/benchmark/results/0.3.2.csv b/benchmark/results/0.3.2.csv
diff --git a/benchmark/results/0.5.0.csv b/benchmark/results/0.5.0.csv
diff --git a/benchmark/results/0.5.1.csv b/benchmark/results/0.5.1.csv
diff --git a/docker/orion-jupyter.Dockerfile b/docker/orion-jupyter.Dockerfile
@@ -7,17 +7,17 @@ COPY setup.py /app
 RUN pip install -e /app && pip install jupyter
 
 COPY orion /app/orion
-COPY notebooks /app/notebooks
-RUN mkdir -p /app/orion/notebooks/data \
-    && cd /app/orion/notebooks/data \
-    && wget https://d3-ai-orion.s3.amazonaws.com/S-1.csv \
-    && wget https://d3-ai-orion.s3.amazonaws.com/S-2.csv \
-    && wget https://d3-ai-orion.s3.amazonaws.com/E-1.csv \
-    && wget https://d3-ai-orion.s3.amazonaws.com/P-1.csv \
+COPY tutorials /app/tutorials
+RUN mkdir -p /app/orion/tutorials/data \
+    && cd /app/orion/tutorials/data \
+    && wget https://sintel-orion.s3.amazonaws.com/S-1.csv \
+    && wget https://sintel-orion.s3.amazonaws.com/S-2.csv \
+    && wget https://sintel-orion.s3.amazonaws.com/E-1.csv \
+    && wget https://sintel-orion.s3.amazonaws.com/P-1.csv \
     && mkdir -p /app/orion/data \
     && cd /app/orion/data \
-    && wget https://d3-ai-orion.s3.amazonaws.com/anomalies.csv \
-    && cp /app/orion/notebooks/data/* .
+    && wget https://sintel-orion.s3.amazonaws.com/anomalies.csv \
+    && cp /app/orion/tutorials/data/* .
 
 RUN adduser jupyter --uid 1000 --disabled-password --system && chown -R jupyter /app
 
diff --git a/docker/package.sh b/docker/package.sh
@@ -8,20 +8,20 @@ mv orion-jupyter.tar Orion
 
 # Download the demo data inside the new repository
 cd Orion
-wget https://d3-ai-orion.s3.amazonaws.com/B-1.csv
+wget https://sintel-orion.s3.amazonaws.com/B-1.csv
 
-mkdir -p notebooks/data
-cd notebooks/data
-wget https://d3-ai-orion.s3.amazonaws.com/S-1.csv
-wget https://d3-ai-orion.s3.amazonaws.com/S-2.csv
-wget https://d3-ai-orion.s3.amazonaws.com/E-1.csv
-wget https://d3-ai-orion.s3.amazonaws.com/P-1.csv
+mkdir -p tutorials/data
+cd tutorials/data
+wget https://sintel-orion.s3.amazonaws.com/S-1.csv
+wget https://sintel-orion.s3.amazonaws.com/S-2.csv
+wget https://sintel-orion.s3.amazonaws.com/E-1.csv
+wget https://sintel-orion.s3.amazonaws.com/P-1.csv
 cd ../..
 
 mkdir -p orion/data
-cp notebooks/data/* orion/data
+cp tutorials/data/* orion/data
 cd orion/data
-wget https://d3-ai-orion.s3.amazonaws.com/anomalies.csv
+wget https://sintel-orion.s3.amazonaws.com/anomalies.csv
 
 cd ../../../
 
diff --git a/orion/__init__.py b/orion/__init__.py
@@ -2,7 +2,7 @@
 
 __author__ = """MIT Data To AI Lab"""
 __email__ = 'dailabmit@gmail.com'
-__version__ = '0.5.0'
+__version__ = '0.5.1.dev2'
 
 import os
 
diff --git a/orion/pipelines/verified/aer/aer_ucr.json b/orion/pipelines/verified/aer/aer_ucr.json
@@ -0,0 +1,7 @@
+{
+    "mlstars.custom.timeseries_preprocessing.time_segments_aggregate#1": {
+        "time_column": "timestamp",
+        "interval": 300,
+        "method": "mean"
+    }
+}
diff --git a/orion/pipelines/verified/arima/arima_ucr.json b/orion/pipelines/verified/arima/arima_ucr.json
@@ -0,0 +1,30 @@
+{
+    "mlstars.custom.timeseries_preprocessing.time_segments_aggregate#1": {
+        "time_column": "timestamp",
+        "interval": 300,
+        "method": "mean"
+    },
+    "sklearn.preprocessing.MinMaxScaler#1": {
+        "feature_range": [
+            -1,
+            1
+        ]
+    },
+    "mlstars.custom.timeseries_preprocessing.rolling_window_sequences#1": {
+        "target_column": 0,
+        "window_size": 250
+    },
+    "numpy.reshape#1": {
+        "newshape": [
+            -1,
+            250
+        ]
+    },
+    "statsmodels.tsa.arima_model.Arima#1": {
+        "steps": 1
+    },
+    "orion.primitives.timeseries_anomalies.find_anomalies#1": {
+        "window_size": 2000,
+        "window_step_size": 100
+    }
+}
diff --git a/orion/pipelines/verified/azure/azure_artificialwithanomaly.json b/orion/pipelines/verified/azure/azure_artificialwithanomaly.json
@@ -7,6 +7,6 @@
     "orion.primitives.azure_anomaly_detector.detect_anomalies#1": {
         "interval": 600,
         "granularity": "minutely",
-        "custom_interval": 5
+        "custom_interval": 10
     }
 }
diff --git a/orion/pipelines/verified/azure/azure_realadexchange.json b/orion/pipelines/verified/azure/azure_realadexchange.json
@@ -6,7 +6,7 @@
     },
     "orion.primitives.azure_anomaly_detector.detect_anomalies#1": {
         "interval": 3600,
-        "granularity": "minutely",
-        "custom_interval": 5
+        "granularity": "hourly",
+        "custom_interval": 1
     }
 }
diff --git a/orion/pipelines/verified/azure/azure_realawscloudwatch.json b/orion/pipelines/verified/azure/azure_realawscloudwatch.json
@@ -7,6 +7,6 @@
     "orion.primitives.azure_anomaly_detector.detect_anomalies#1": {
         "interval": 600,
         "granularity": "minutely",
-        "custom_interval": 5
+        "custom_interval": 10
     }
 }
diff --git a/orion/pipelines/verified/azure/azure_realtraffic.json b/orion/pipelines/verified/azure/azure_realtraffic.json
@@ -7,6 +7,6 @@
     "orion.primitives.azure_anomaly_detector.detect_anomalies#1": {
         "interval": 600,
         "granularity": "minutely",
-        "custom_interval": 5
+        "custom_interval": 10
     }
 }
diff --git a/orion/pipelines/verified/azure/azure_realtweets.json b/orion/pipelines/verified/azure/azure_realtweets.json
@@ -7,6 +7,6 @@
     "orion.primitives.azure_anomaly_detector.detect_anomalies#1": {
         "interval": 600,
         "granularity": "minutely",
-        "custom_interval": 5
+        "custom_interval": 10
     }
 }
diff --git a/orion/pipelines/verified/azure/azure_ucr.json b/orion/pipelines/verified/azure/azure_ucr.json
@@ -0,0 +1,12 @@
+{
+    "mlstars.custom.timeseries_preprocessing.time_segments_aggregate#1": {
+        "time_column": "timestamp",
+        "interval": 300,
+        "method": "mean"
+    },
+    "orion.primitives.azure_anomaly_detector.detect_anomalies#1": {
+        "interval": 300,
+        "granularity": "minutely",
+        "custom_interval": 5
+    }
+}
diff --git a/orion/pipelines/verified/dense_autoencoder/dense_autoencoder_ucr.json b/orion/pipelines/verified/dense_autoencoder/dense_autoencoder_ucr.json
@@ -0,0 +1,6 @@
+{
+    "mlstars.custom.timeseries_preprocessing.time_segments_aggregate#1": {
+        "time_column": "timestamp",
+        "interval": 300
+    }
+}
diff --git a/orion/pipelines/verified/lstm_autoencoder/lstm_autoencoder_ucr.json b/orion/pipelines/verified/lstm_autoencoder/lstm_autoencoder_ucr.json
@@ -0,0 +1,6 @@
+{
+    "mlstars.custom.timeseries_preprocessing.time_segments_aggregate#1": {
+        "time_column": "timestamp",
+        "interval": 300
+    }
+}
diff --git a/orion/pipelines/verified/lstm_dynamic_threshold/lstm_dynamic_threshold_ucr.json b/orion/pipelines/verified/lstm_dynamic_threshold/lstm_dynamic_threshold_ucr.json
@@ -0,0 +1,17 @@
+{
+    "mlstars.custom.timeseries_preprocessing.time_segments_aggregate#1": {
+        "time_column": "timestamp",
+        "interval": 300
+    },
+    "sklearn.preprocessing.MinMaxScaler#1": {
+        "feature_range": [-1, 1]
+    },
+    "mlstars.custom.timeseries_preprocessing.rolling_window_sequences#1": {
+        "target_column": 0,
+        "window_size": 250
+    },
+    "keras.Sequential.LSTMTimeSeriesRegressor#1": {
+        "epochs": 35,
+        "verbose": false
+    }
+}
diff --git a/orion/pipelines/verified/tadgan/tadgan_ucr.json b/orion/pipelines/verified/tadgan/tadgan_ucr.json
@@ -0,0 +1,12 @@
+{
+    "mlstars.custom.timeseries_preprocessing.time_segments_aggregate#1": {
+        "time_column": "timestamp",
+        "interval": 300,
+        "method": "mean"
+    },
+    "orion.primitives.tadgan.score_anomalies#1": {
+        "rec_error_type": "dtw",
+        "comb": "sum",
+        "lambda_rec": 0.5
+    }
+}
diff --git a/orion/pipelines/verified/vae/vae_ucr.json b/orion/pipelines/verified/vae/vae_ucr.json
@@ -0,0 +1,6 @@
+{
+    "mlstars.custom.timeseries_preprocessing.time_segments_aggregate#1": {
+        "time_column": "timestamp",
+        "interval": 300
+    }
+}
diff --git a/orion/results.py b/orion/results.py
@@ -6,6 +6,7 @@
 DATASET_FAMILY = {
     "MSL": "NASA",
     "SMAP": "NASA",
+    "UCR": "UCR",
     "YAHOOA1": "YAHOO",
     "YAHOOA2": "YAHOO",
     "YAHOOA3": "YAHOO",
@@ -20,6 +21,7 @@
 DATASET_ABBREVIATION = {
     "MSL": "MSL",
     "SMAP": "SMAP",
+    "UCR": "UCR",
     "YAHOOA1": "A1",
     "YAHOOA2": "A2",
     "YAHOOA3": "A3",
@@ -136,13 +138,13 @@ def add_sheet(df, name, writer, cell_fmt, header_fmt):
 
         # horizontal
         worksheet.merge_range(0, 1, 0, 2, columns[1][0], header_fmt)
-        worksheet.merge_range(0, 3, 0, 6, columns[3][0], header_fmt)
-        worksheet.merge_range(0, 7, 0, 11, columns[7][0], header_fmt)
+        worksheet.merge_range(0, 4, 0, 7, columns[4][0], header_fmt)
+        worksheet.merge_range(0, 8, 0, 12, columns[8][0], header_fmt)
 
         # vertical
         worksheet.merge_range(0, 0, 1, 0, columns[0][0], header_fmt)
-        worksheet.merge_range(0, 12, 1, 12, columns[12][0], header_fmt)
         worksheet.merge_range(0, 13, 1, 13, columns[13][0], header_fmt)
+        worksheet.merge_range(0, 14, 1, 14, columns[14][0], header_fmt)
 
     for idx, width in enumerate(widths):
         worksheet.set_column(idx, idx, width + 1, cell_fmt)
diff --git a/setup.cfg b/setup.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.5.0
+current_version = 0.5.1.dev2
 commit = True
 tag = True
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\.(?P<release>[a-z]+)(?P<candidate>\d+))?
diff --git a/setup.py b/setup.py
@@ -27,7 +27,7 @@
     'scikit-learn>=0.22,<1.2',
     'tabulate>=0.8.3,<0.9',
     'numba>=0.48,<0.52',
-    'pyts>=0.9,<0.11',
+    'pyts>=0.9,<0.12',
     'azure-cognitiveservices-anomalydetector>=0.3,<0.4',
     'xlsxwriter>=1.3.6,<1.4',
     'tqdm>=4.36.1',
@@ -126,6 +126,6 @@
     test_suite='tests',
     tests_require=tests_require,
     url='https://github.com/sintel-dev/Orion',
-    version='0.5.0',
+    version='0.5.1.dev2',
     zip_safe=False,
 )
diff --git a/tests/requirement_files/latest_requirements.txt b/tests/requirement_files/latest_requirements.txt
