Comprehensive Enhancements and Improvements

NewChanges.md

@@ -0,0 +1,75 @@
+
+# Document for Changes to Informer Model
+
+This document outlines the modifications made to the *Dataset_Custom* class, *Exp_Informer* class and perhaps a few other classes to enhance their functionality, adaptability and customization, for different use cases. The changes include adjustments to parameter defaults, introduction of new parameters, enabling detailed tracking of performance metrics and improvements in the data scaling process.
+
+## Changes to Dataset_Custom Class
+
+
+### Parameter Changes:
+
+        features: Default value changed from 'S' to 'MS', enabling multivariate scaling by default.
+        data_path: Default value changed from 'ETTh1.csv' to 'data.csv' to reflect a more generic placeholder for data files.
+        target: Default value changed from 'OT' to 'Close', typically used in financial datasets to denote closing prices.
+        freq: Frequency of data aggregation changed from hourly ('h') to business days ('b'), which is common in financial time series analysis.
+
+### New Parameters Introduced:
+
+        kind_of_scaler: Allows specifying the type of scaler to use ('Standard' or 'MinMax'). If none provided, defaults to 'Standard'.
+        test_size: Allows specifying the proportion of the dataset to be reserved for testing. The remaining data minus the test set is used for training.
+
+### Size Configuration:
+
+        Updated default sequence lengths to more compact settings, presumably to fit smaller or more specific windows of time series data:
+            seq_len: From 24*4*4 to 1*5*2 (presumed to represent a smaller window for analysis).
+            label_len and pred_len: Both simplified from 24*4 to 1*1, indicating shorter periods for labels and predictions.
+
+### Data Handling Enhancements:
+
+        Dynamic Train/Test Split: The training dataset size is dynamically adjusted based on the test_size, with the validation set size being calculated as the residual.
+        Enhanced Scaling Logic: Individual columns can now be scaled using either StandardScaler or MinMaxScaler. The scaler for the target column is preserved separately, allowing for easy inverse transformations if needed.
+
+### Code Structure and Execution:
+
+        __read_data__ Method: This method has been restructured to accommodate the flexible scaling of individual columns based on the kind_of_scaler setting.
+        Scaler Persistence: The scaler used for the target column is saved as a .pkl file, facilitating model deployment or further analysis without re-training the scaler.
+
+### Enhanced Functionality
+
+    Flexibility in Data Scaling: Users can now choose how each column is scaled, which is particularly useful in datasets where different features have different ranges and distributions.
+    Improved Customizability: With parameters like kind_of_scaler and test_size, users have more control over the preprocessing and partitioning of the data, allowing the class to be better tailored to specific analytical needs.
+    Support for Diverse Data Frequencies: By supporting different frequencies like business days ('b'), the class becomes more suitable for financial time series analysis, where non-trading days do not need to be included.
+
+
+## Changes to Exp_Informer Class
+
+
+### Constructor ```__init__``` Enhancements:
+        Tracking Metrics: New lists train_losses, test_losses, actual_test_values, and predicted_test_values have been introduced to track detailed training and testing metrics throughout the model's operation.
+
+### Data Loader Configuration:
+        Shuffling Flags: The approach to handling data shuffling is now more customizable with shuffle_for_test, shuffle_for_pred, and shuffle_for_train flags introduced. These flags are pulled from the args passed during initialization, allowing for dynamic adjustments based on experimental needs.
+        Batch and Frequency Handling: Enhanced flexibility in configuring batch sizes and frequency parameters for different operation modes (test, pred, and default training mode).
+
+### Optimizer Selection:
+        Extended Optimizer Options: The method _select_optimizer has been significantly expanded to include a wide range of optimizers. This allows for a more fine-tuned approach to optimization based on specific use cases or performance objectives.
+        Customization Through args: The optimizer to be used can now be specified through args.kind_of_optim, enabling the selection of optimizers like AdamW, SparseAdam, SGD, RMSprop, RAdam, NAdam, LBFGS, Adamax, ASGD, Adadelta, and Adagrad. The default optimizer remains Adam if none is specified.
+
+
+## Introduction of Custom and Standard Loss Functions:
+
+### Custom Loss Functions: 
+        New loss functions such as WeightedMeanAbsolutePercentageError, SymmetricMeanAbsolutePercentageError, RMSELoss, QuantileLoss, HuberLoss, and PinballLoss have been introduced. These functions are sourced from utils.criter, a module presumably containing implementations tailored to specific use cases, especially for handling regression tasks where such metrics are crucial.
+        Standard Loss Functions: Integration of PyTorch's native loss functions like nn.L1Loss (Mean Absolute Error) and nn.MSELoss (Mean Squared Error) ensures that users have access to well-established methods for baseline comparisons.
+
+### *Dynamic Loss Function Selection*
+        The method leverages self.args.criter, a parameter from the model's argument list, to dynamically select the appropriate loss function based on user input. This design ensures flexibility and easy customization of the model training process.
+
+By parameterizing the choice of loss function, the _select_criterion method allows users to easily experiment with different types of errors and loss metrics, which is particularly useful in exploring the best setups for various prediction tasks.
+
+
+## Conclusion 
+These enhancements make the Exp_Informer class a more robust and versatile tool for handling complex machine learning workflows. By enabling detailed tracking of performance metrics and offering customizable data handling and optimization strategies, the class is well-suited to meet the diverse needs of users engaged in advanced data analysis and modeling tasks. The introduction of dynamic shuffling flags and a wide selection of optimizers provides flexibility and fine-tuning capabilities essential for optimizing model training and evaluation processes.
+
+
+The enhanced _select_criterion method provides the Exp_Informer class with a robust and flexible framework for selecting loss functions, critical for optimizing model performance across different types of prediction tasks. This update facilitates more refined model tuning and enables users to achieve better alignment with specific performance metrics and objectives. By supporting a wide range of loss functions, the class becomes significantly more versatile and powerful, suitable for advanced machine learning challenges.

README.md

@@ -1,4 +1,6 @@
 # Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (AAAI'21 Best Paper)
+
+
 ![Python 3.6](https://img.shields.io/badge/python-3.6-green.svg?style=plastic)
 ![PyTorch 1.2](https://img.shields.io/badge/PyTorch%20-%23EE4C2C.svg?style=plastic)
 ![cuDNN 7.3.1](https://img.shields.io/badge/cudnn-7.3.1-green.svg?style=plastic)
@@ -73,21 +75,6 @@ for file in `ls scripts`; do make run_module module="bash scripts/$script"; done
 ```
 
 ## Usage
-<span id="colablink">Colab Examples:</span> We provide google colabs to help reproduce and customize our repo, which includes `experiments(train and test)`, `prediction`, `visualization` and `custom data`.
-[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1_X7O2BkFLvqyCdZzDZvV2MB0aAvYALLC)
-
-Commands for training and testing the model with *ProbSparse* self-attention on Dataset ETTh1, ETTh2 and ETTm1 respectively:
-
-```bash
-# ETTh1
-python -u main_informer.py --model informer --data ETTh1 --attn prob --freq h
-
-# ETTh2
-python -u main_informer.py --model informer --data ETTh2 --attn prob --freq h
-
-# ETTm1
-python -u main_informer.py --model informer --data ETTm1 --attn prob --freq t
-```
 
 More parameter information please refer to `main_informer.py`.
 

data/data_loader.py

@@ -1,12 +1,12 @@
 import os
 import numpy as np
 import pandas as pd
-
+import joblib
 import torch
 from torch.utils.data import Dataset, DataLoader
-# from sklearn.preprocessing import StandardScaler
+from sklearn.preprocessing import MinMaxScaler, StandardScaler
 
-from utils.tools import StandardScaler
+#from utils.tools import StandardScaler
 from utils.timefeatures import time_features
 
 import warnings
@@ -38,6 +38,7 @@ def __init__(self, root_path, flag='train', size=None,
         self.timeenc = timeenc
         self.freq = freq
 
+
         self.root_path = root_path
         self.data_path = data_path
         self.__read_data__()
@@ -186,15 +187,26 @@ def inverse_transform(self, data):
 
 
 class Dataset_Custom(Dataset):
-    def __init__(self, root_path, flag='train', size=None, 
-                 features='S', data_path='ETTh1.csv', 
-                 target='OT', scale=True, inverse=False, timeenc=0, freq='h', cols=None):
+    def __init__(self, 
+                 root_path, 
+                 flag='train', 
+                 size=None, 
+                 features='MS',
+                 data_path='data.csv', 
+                 target='Close', 
+                 scale=True, 
+                 inverse=False,
+                 timeenc=0, 
+                 freq='b', 
+                 cols=None, 
+                 kind_of_scaler = None,
+                 test_size = 0.2):
         # size [seq_len, label_len, pred_len]
         # info
         if size == None:
-            self.seq_len = 24*4*4
-            self.label_len = 24*4
-            self.pred_len = 24*4
+            self.seq_len = 1*5*2
+            self.label_len = 1*1
+            self.pred_len = 1*1
         else:
             self.seq_len = size[0]
             self.label_len = size[1]
@@ -203,7 +215,9 @@ def __init__(self, root_path, flag='train', size=None,
         assert flag in ['train', 'test', 'val']
         type_map = {'train':0, 'val':1, 'test':2}
         self.set_type = type_map[flag]
-
+
+        self.test_size = test_size
+        self.train_size = 0.90 - self.test_size
         self.features = features
         self.target = target
         self.scale = scale
@@ -213,12 +227,13 @@ def __init__(self, root_path, flag='train', size=None,
         self.cols=cols
         self.root_path = root_path
         self.data_path = data_path
+        self.kind_of_scaler = kind_of_scaler if kind_of_scaler is not None else 'Standard'
         self.__read_data__()
 
     def __read_data__(self):
-        self.scaler = StandardScaler()
+
         df_raw = pd.read_csv(os.path.join(self.root_path,
-                                          self.data_path))
+                                              self.data_path))
         '''
         df_raw.columns: ['date', ...(other features), target feature]
         '''
@@ -230,8 +245,8 @@ def __read_data__(self):
             cols = list(df_raw.columns); cols.remove(self.target); cols.remove('date')
         df_raw = df_raw[['date']+cols+[self.target]]
 
-        num_train = int(len(df_raw)*0.7)
-        num_test = int(len(df_raw)*0.2)
+        num_train = int(len(df_raw)*self.train_size)
+        num_test = int(len(df_raw)*self.test_size)
         num_vali = len(df_raw) - num_train - num_test
         border1s = [0, num_train-self.seq_len, len(df_raw)-num_test-self.seq_len]
         border2s = [num_train, num_train+num_vali, len(df_raw)]
@@ -244,10 +259,39 @@ def __read_data__(self):
         elif self.features=='S':
             df_data = df_raw[[self.target]]
 
+
         if self.scale:
-            train_data = df_data[border1s[0]:border2s[0]]
-            self.scaler.fit(train_data.values)
-            data = self.scaler.transform(df_data.values)
+            if self.features=='S' or self.features=='MS':
+                col_scaled = []
+                for col in df_data.columns:
+                    col_data = df_data[[col]].values
+                    if self.kind_of_scaler == 'MinMax':
+                        if col == self.target:
+                            self.scaler = MinMaxScaler()
+                        else:
+                            scaler = MinMaxScaler()
+                    else:
+                        if col == self.target:
+                            self.scaler = StandardScaler()
+                        else:
+                            scaler = StandardScaler()
+                    if col == self.target:
+                        self.scaler.fit(col_data[border1s[0]:border2s[0]])
+                        joblib.dump(self.scaler, os.path.join(self.root_path, 'scaler.pkl'))
+                        col_temp = self.scaler.transform(col_data)
+                    else:
+                        scaler.fit(col_data[border1s[0]:border2s[0]])
+                        col_temp = scaler.transform(col_data)
+                    col_scaled.append(col_temp)
+                if len(col_scaled) == 1:
+                    data = col_scaled[0]
+                else:
+                    data = np.concatenate(col_scaled, axis = 1)
+            else:
+                self.scaler = StandardScaler()
+                train_data = df_data[border1s[0]:border2s[0]]
+                self.scaler.fit(train_data.values)
+                data = self.scaler.transform(df_data.values)                
         else:
             data = df_data.values
 
@@ -286,8 +330,9 @@ def inverse_transform(self, data):
 
 class Dataset_Pred(Dataset):
     def __init__(self, root_path, flag='pred', size=None, 
-                 features='S', data_path='ETTh1.csv', 
-                 target='OT', scale=True, inverse=False, timeenc=0, freq='15min', cols=None):
+                 features='MS', data_path='data.csv', 
+                 target='Close', scale=True, inverse=False, timeenc=0, freq='b', cols=None,
+                 kind_of_scaler = None, test_size = None):
         # size [seq_len, label_len, pred_len]
         # info
         if size == None:
@@ -300,7 +345,10 @@ def __init__(self, root_path, flag='pred', size=None,
             self.pred_len = size[2]
         # init
         assert flag in ['pred']
-
+
+        self.kind_of_scaler = kind_of_scaler if kind_of_scaler is not None else 'Standard'
+        self.test_size = test_size
+        self.train_size = None
         self.features = features
         self.target = target
         self.scale = scale
@@ -336,8 +384,36 @@ def __read_data__(self):
             df_data = df_raw[[self.target]]
 
         if self.scale:
-            self.scaler.fit(df_data.values)
-            data = self.scaler.transform(df_data.values)
+            if self.features=='S' or self.features=='MS':
+                col_scaled = []
+                for col in df_data.columns:
+                    col_data = df_data[[col]].values
+                    if self.kind_of_scaler == 'MinMax':
+                        if col == self.target:
+                            self.scaler = MinMaxScaler()
+                        else:
+                            scaler = MinMaxScaler()
+                    else:
+                        if col == self.target:
+                            self.scaler = StandardScaler()
+                        else:
+                            scaler = StandardScaler()
+                    if col == self.target:
+                        self.scaler.fit(col_data)
+                        joblib.dump(self.scaler, os.path.join(self.root_path, 'scaler.pkl'))
+                        col_temp = self.scaler.transform(col_data)
+                    else:
+                        scaler.fit(col_data)
+                        col_temp = scaler.transform(col_data)
+                    col_scaled.append(col_temp)
+                if len(col_scaled) == 1:
+                    data = col_scaled[0]
+                else:
+                    data = np.concatenate(col_scaled, axis = 1)
+            else:
+                self.scaler = StandardScaler()
+                self.scaler.fit(df_data.values)
+                data = self.scaler.transform(df_data.values)                
         else:
             data = df_data.values