From 7c458662a51dcbb6a165bd5746433b9d2c777586 Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Thu, 6 Mar 2025 19:53:00 +0900 Subject: [PATCH 01/14] Update README.md --- README.md | 156 +++--------------------------------------------------- 1 file changed, 6 insertions(+), 150 deletions(-) diff --git a/README.md b/README.md index 08583bc8..b72c8186 100644 --- a/README.md +++ b/README.md @@ -1,155 +1,11 @@ -# iTransformer +# RxiTransformer -The repo is the official implementation for the paper: [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://arxiv.org/abs/2310.06625). [[Slides]](https://cloud.tsinghua.edu.cn/f/175ff98f7e2d44fbbe8e/), [[Poster]](https://cloud.tsinghua.edu.cn/f/36a2ae6c132d44c0bd8c/). +RxiTransformer is a machine learning model for time series forecasting that combines elements of RLinear with iTransformer. +The source code is almost the same as that of iTransformer. -# Updates +The main changes are described below. -:triangular_flag_on_post: **News** (2024.10) [TimeXer](https://arxiv.org/abs/2402.19072), a Transformer for predicting with exogenous variables, is released. Code is available [here](https://github.com/thuml/TimeXer). +# Model: RxiTransformer -:triangular_flag_on_post: **News** (2024.05) Many thanks for the great efforts from [lucidrains](https://github.com/lucidrains/iTransformer). A pip package for the usage of iTransformer variants can be simply installed via ```pip install iTransformer``` - -:triangular_flag_on_post: **News** (2024.04) iTransformer has benn included in [NeuralForecast](https://github.com/Nixtla/neuralforecast/blob/main/neuralforecast/models/itransformer.py). Special thanks to the contributor @[Marco](https://github.com/marcopeix)! - -:triangular_flag_on_post: **News** (2024.03) Introduction of our work in [Chinese](https://mp.weixin.qq.com/s/-pvBnA1_NSloNxa6TYXTSg) is available. - -:triangular_flag_on_post: **News** (2024.02) iTransformer has been accepted as **ICLR 2024 Spotlight**. - -:triangular_flag_on_post: **News** (2023.12) iTransformer available in [GluonTS](https://github.com/awslabs/gluonts/pull/3017) with probablistic head and support for static covariates. Notebook is available [here](https://github.com/awslabs/gluonts/blob/dev/examples/iTransformer.ipynb). - -:triangular_flag_on_post: **News** (2023.12) We received lots of valuable suggestions. A [revised version](https://arxiv.org/pdf/2310.06625v2.pdf) (**24 Pages**) is now available. - -:triangular_flag_on_post: **News** (2023.10) iTransformer has been included in [[Time-Series-Library]](https://github.com/thuml/Time-Series-Library) and achieves state-of-the-art in Lookback-$96$ forecasting. - -:triangular_flag_on_post: **News** (2023.10) All the scripts for the experiments in our [paper](https://arxiv.org/pdf/2310.06625.pdf) are available. - - -## Introduction - -🌟 Considering the characteristics of multivariate time series, iTransformer breaks the conventional structure without modifying any Transformer modules. **Inverted Transformer is all you need in MTSF**. - -

- -

- -🏆 iTransformer achieves the comprehensive state-of-the-art in challenging multivariate forecasting tasks and solves several pain points of Transformer on extensive time series data. - -

- -

- - -## Overall Architecture - -iTransformer regards **independent time series as variate tokens** to **capture multivariate correlations by attention** and **utilize layernorm and feed-forward networks to learn series representations**. - -

- -

- -The pseudo-code of iTransformer is as simple as the following: - -

- -

- -## Usage - -1. Install Pytorch and necessary dependencies. - -``` -pip install -r requirements.txt -``` - -1. The datasets can be obtained from [Google Drive](https://drive.google.com/file/d/1l51QsKvQPcqILT3DwfjCgx8Dsg2rpjot/view?usp=drive_link) or [Baidu Cloud](https://pan.baidu.com/s/11AWXg1Z6UwjHzmto4hesAA?pwd=9qjr). - -2. Train and evaluate the model. We provide all the above tasks under the folder ./scripts/. You can reproduce the results as the following examples: - -``` -# Multivariate forecasting with iTransformer -bash ./scripts/multivariate_forecasting/Traffic/iTransformer.sh - -# Compare the performance of Transformer and iTransformer -bash ./scripts/boost_performance/Weather/iTransformer.sh - -# Train the model with partial variates, and generalize on the unseen variates -bash ./scripts/variate_generalization/ECL/iTransformer.sh - -# Test the performance on the enlarged lookback window -bash ./scripts/increasing_lookback/Traffic/iTransformer.sh - -# Utilize FlashAttention for acceleration -bash ./scripts/efficient_attentions/iFlashTransformer.sh -``` - -## Main Result of Multivariate Forecasting - -We evaluate the iTransformer on challenging multivariate forecasting benchmarks (**generally hundreds of variates**). **Comprehensive good performance** (MSE/MAE $\downarrow$) is achieved. - - - -### Online Transaction Load Prediction of Alipay Trading Platform (Avg Results) - -

- -

- -## General Performance Boosting on Transformers - -By introducing the proposed framework, Transformer and its variants achieve **significant performance improvement**, demonstrating the **generality of the iTransformer approach** and **benefiting from efficient attention mechanisms**. - -

- -

- -## Zero-Shot Generalization on Variates - -**Technically, iTransformer is able to forecast with arbitrary numbers of variables**. We train iTransformers on partial variates and forecast unseen variates with good generalizability. - -

- -

- -## Model Analysis - -Benefiting from inverted Transformer modules: - -- (Left) Inverted Transformers learn **better time series representations** (more similar [CKA](https://github.com/jayroxis/CKA-similarity)) favored by forecasting. -- (Right) The inverted self-attention module learns **interpretable multivariate correlations**. - -

- -

- -## Citation - -If you find this repo helpful, please cite our paper. - -``` -@article{liu2023itransformer, - title={iTransformer: Inverted Transformers Are Effective for Time Series Forecasting}, - author={Liu, Yong and Hu, Tengge and Zhang, Haoran and Wu, Haixu and Wang, Shiyu and Ma, Lintao and Long, Mingsheng}, - journal={arXiv preprint arXiv:2310.06625}, - year={2023} -} -``` - -## Acknowledgement - -We appreciate the following GitHub repos a lot for their valuable code and efforts. -- Reformer (https://github.com/lucidrains/reformer-pytorch) -- Informer (https://github.com/zhouhaoyi/Informer2020) -- FlashAttention (https://github.com/shreyansh26/FlashAttention-PyTorch) -- Autoformer (https://github.com/thuml/Autoformer) -- Stationary (https://github.com/thuml/Nonstationary_Transformers) -- Time-Series-Library (https://github.com/thuml/Time-Series-Library) -- lucidrains (https://github.com/lucidrains/iTransformer) - -This work was supported by Ant Group through the CCF-Ant Research Fund, and awarded as [Outstanding Projects of CCF Fund](https://mp.weixin.qq.com/s/PDLNbibZD3kqhcUoNejLfA). - -## Contact - -If you have any questions or want to use the code, feel free to contact: -* Yong Liu (liuyong21@mails.tsinghua.edu.cn) -* Haoran Zhang (z-hr20@mails.tsinghua.edu.cn) -* Tengge Hu (htg21@mails.tsinghua.edu.cn) +# Learnign Rate Scheduler: ReduceLRReverter From 0b5600da43a04bc22165cf0abe3609d0f1f4165e Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Thu, 6 Mar 2025 19:55:05 +0900 Subject: [PATCH 02/14] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b72c8186..7f0cc7a9 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # RxiTransformer -RxiTransformer is a machine learning model for time series forecasting that combines elements of RLinear with iTransformer. +RxiTransformer is a machine learning model for time series forecasting that combines elements of RLinear(https://github.com/plumprc/RTSF) with iTransformer(https://github.com/thuml/iTransformer). The source code is almost the same as that of iTransformer. From 3c4c6b6eed3f414580d1f846a5f36a21488fd699 Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Thu, 6 Mar 2025 19:57:05 +0900 Subject: [PATCH 03/14] Update README.md --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7f0cc7a9..8f8b0fff 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,14 @@ # RxiTransformer -RxiTransformer is a machine learning model for time series forecasting that combines elements of RLinear(https://github.com/plumprc/RTSF) with iTransformer(https://github.com/thuml/iTransformer). - The source code is almost the same as that of iTransformer. The main changes are described below. # Model: RxiTransformer +RxiTransformer is a machine learning model for time series forecasting that combines elements of RLinear(https://github.com/plumprc/RTSF) with iTransformer(https://github.com/thuml/iTransformer). + + # Learnign Rate Scheduler: ReduceLRReverter + +# Minor Changes From e592b2fad91394ffa7189d8026f9779298c6ca59 Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Fri, 7 Mar 2025 18:18:39 +0900 Subject: [PATCH 04/14] Add files via upload --- moving/RxiTransformer.py | 83 +++++++++ moving/RxiTransformer7.py | 89 +++++++++ moving/exp_reverter.py | 366 ++++++++++++++++++++++++++++++++++++++ moving/requirements.txt | 6 + moving/reverter.py | 170 ++++++++++++++++++ moving/torch.txt | 4 + 6 files changed, 718 insertions(+) create mode 100644 moving/RxiTransformer.py create mode 100644 moving/RxiTransformer7.py create mode 100644 moving/exp_reverter.py create mode 100644 moving/requirements.txt create mode 100644 moving/reverter.py create mode 100644 moving/torch.txt diff --git a/moving/RxiTransformer.py b/moving/RxiTransformer.py new file mode 100644 index 00000000..20eb46b7 --- /dev/null +++ b/moving/RxiTransformer.py @@ -0,0 +1,83 @@ +import torch +import torch.nn as nn +import torch.nn.functional as F +from layers.Transformer_EncDec import Encoder, EncoderLayer +from layers.SelfAttention_Family import FullAttention, AttentionLayer +from layers.Embed import DataEmbedding_inverted +import numpy as np +from layers.Invertible import RevIN + + +class Model(nn.Module): + # Parrarel model of iTransformer and RLinear-CI + + def __init__(self, configs): + super(Model, self).__init__() + self.seq_len = configs.seq_len + self.pred_len = configs.pred_len + self.output_attention = configs.output_attention + self.use_norm = configs.use_norm + + # iTransoformer + self.enc_embedding = DataEmbedding_inverted(configs.seq_len, configs.d_model, configs.embed, configs.freq, + configs.dropout) + self.class_strategy = configs.class_strategy + self.encoder = Encoder( + [ + EncoderLayer( + AttentionLayer( + FullAttention(False, configs.factor, attention_dropout=configs.dropout, + output_attention=configs.output_attention), configs.d_model, configs.n_heads), + configs.d_model, + configs.d_ff, + dropout=configs.dropout, + activation=configs.activation + ) for l in range(configs.e_layers) + ], + norm_layer=torch.nn.LayerNorm(configs.d_model) + ) + self.projector = nn.Linear(configs.d_model, configs.pred_len, bias=True) + + # RLinear-CI + self.Linear = nn.ModuleList([ + nn.Linear(configs.seq_len, configs.pred_len) for _ in range(configs.enc_in) + ]) + self.dropout = nn.Dropout(configs.dropout) + self.rev = RevIN(configs.enc_in) if configs.rev else None + + def forecast(self, x_enc, x_mark_enc, x_dec, x_mark_dec): + # Saving x_enc + x_enc_save = x_enc + + #iTransformer + if self.use_norm: + # Normalization from RevIN + x_normed = self.rev(x_enc_save, 'norm') if self.rev else x_enc_save + _, _, N = x_enc.shape # B L N + # B: batch_size; E: d_model; # L: seq_len; S: pred_len; + # N: number of variate (tokens), can also includes covariates + # B L N -> B S N + enc_out = self.enc_embedding(x_normed, x_mark_enc) + enc_out, attns = self.encoder(enc_out, attn_mask=None) + enc_out = self.projector(enc_out).permute(0, 2, 1)[:, :, :N] + if self.use_norm: + # De-Normalization from Non-stationary Transformer + enc_out = self.rev(enc_out, 'denorm') if self.rev else enc_out + + # RLinear-CI + x = self.dropout(x_normed) + pred = torch.zeros(x.size(0), self.pred_len, x.size(2)).to('cuda') + for idx, proj in enumerate(self.Linear): + pred[:, :, idx] = proj(x[:, :, idx]) + rl_out = self.rev(pred, 'denorm') if self.rev else pred + + # Add + out = enc_out*0.5 + rl_out*0.5 + + + return out + + + def forward(self, x_enc, x_mark_enc, x_dec, x_mark_dec, mask=None): + dec_out = self.forecast(x_enc, x_mark_enc, x_dec, x_mark_dec) + return dec_out[:, -self.pred_len:, :] # [B, L, D] \ No newline at end of file diff --git a/moving/RxiTransformer7.py b/moving/RxiTransformer7.py new file mode 100644 index 00000000..91081d84 --- /dev/null +++ b/moving/RxiTransformer7.py @@ -0,0 +1,89 @@ +import torch +import torch.nn as nn +import torch.nn.functional as F +from layers.Transformer_EncDec import Encoder, EncoderLayer +from layers.SelfAttention_Family import FullAttention, AttentionLayer +from layers.Embed import DataEmbedding_inverted +import numpy as np +from layers.Invertible import RevIN + + +class Model(nn.Module): + # Parrarel model of iTransformer and RLinear-CI + + def __init__(self, configs): + super(Model, self).__init__() + self.seq_len = configs.seq_len + self.pred_len = configs.pred_len + self.output_attention = configs.output_attention + self.use_norm = configs.use_norm + self.enc_in = configs.enc_in + + # iTransoformer + self.enc_embedding = DataEmbedding_inverted(configs.seq_len, configs.d_model, configs.embed, configs.freq, + configs.dropout) + self.class_strategy = configs.class_strategy + self.encoder = Encoder( + [ + EncoderLayer( + AttentionLayer( + FullAttention(False, configs.factor, attention_dropout=configs.dropout, + output_attention=configs.output_attention), configs.d_model, configs.n_heads), + configs.d_model, + configs.d_ff, + dropout=configs.dropout, + activation=configs.activation + ) for l in range(configs.e_layers) + ], + norm_layer=torch.nn.LayerNorm(configs.d_model) + ) + self.projector = nn.Linear(configs.d_model, configs.pred_len, bias=True) + + # RLinear-CI + self.Linear = nn.ModuleList([ + nn.Linear(configs.seq_len, configs.pred_len) for _ in range(configs.enc_in) + ]) + self.dropout = nn.Dropout(configs.dropout) + self.rev = RevIN(configs.enc_in) if configs.rev else None + + # DistributionLayer + distributer = [] + for i in range(configs.enc_in): + distributer.append(0.5) + self.distributer = nn.Parameter(torch.tensor(distributer)) + + def forecast(self, x_enc, x_mark_enc, x_dec, x_mark_dec): + # Saving x_enc + x_enc_save = x_enc + + # Normalization + x_normed = self.rev(x_enc_save, 'norm') if self.rev else x_enc_save + + # iTransformer + _, _, N = x_enc.shape + enc_out = self.enc_embedding(x_normed, x_mark_enc) + enc_out, attns = self.encoder(enc_out, attn_mask=None) + enc_out = self.projector(enc_out).permute(0, 2, 1)[:, :, :N] + + # RLinear-CI + x = self.dropout(x_normed) + rl_out = torch.zeros(x.size(0), self.pred_len, x.size(2)).to('cuda') + for idx, proj in enumerate(self.Linear): + rl_out[:, :, idx] = proj(x[:, :, idx]) + + # Add + out = torch.zeros_like(enc_out) + for i in range(self.enc_in): + rate = self.distributer[i] + out[:, :, i] = enc_out[:, :, i]*rate + rl_out[:, :, i]*(1 - rate) + + # Denormalization + out = self.rev(out, 'denorm') if self.rev else out + + + return out + + + def forward(self, x_enc, x_mark_enc, x_dec, x_mark_dec, mask=None): + dec_out = self.forecast(x_enc, x_mark_enc, x_dec, x_mark_dec) + return dec_out[:, -self.pred_len:, :] # [B, L, D] \ No newline at end of file diff --git a/moving/exp_reverter.py b/moving/exp_reverter.py new file mode 100644 index 00000000..87e01e46 --- /dev/null +++ b/moving/exp_reverter.py @@ -0,0 +1,366 @@ +from data_provider.data_factory import data_provider +from experiments.exp_basic import Exp_Basic +from utils.tools import EarlyStopping_v2, visual, clear_directory, SAVE +from utils.metrics import metric +import torch +import torch.nn as nn +from torch import optim +import os +import time +import warnings +import numpy as np + +warnings.filterwarnings('ignore') + + +class Exp_Long_Term_Forecast(Exp_Basic): + def __init__(self, args): + super(Exp_Long_Term_Forecast, self).__init__(args) + + def _build_model(self): + model = self.model_dict[self.args.model].Model(self.args).float() + + if self.args.use_multi_gpu and self.args.use_gpu: + model = nn.DataParallel(model, device_ids=self.args.device_ids) + return model + + def _get_data(self, flag): + data_set, data_loader = data_provider(self.args, flag) + return data_set, data_loader + + def _select_optimizer(self): + model_optim = optim.Adam(self.model.parameters(), lr=self.args.learning_rate) + return model_optim + + def _select_criterion(self): + criterion = nn.MSELoss() + return criterion + + def vali(self, vali_data, vali_loader, criterion): + total_loss = [] + self.model.eval() + with torch.no_grad(): + for i, (batch_x, batch_y, batch_x_mark, batch_y_mark) in enumerate(vali_loader): + batch_x = batch_x.float().to(self.device) + batch_y = batch_y.float().to(self.device) + if 'PEMS' in self.args.data or 'Solar' in self.args.data: + batch_x_mark = None + batch_y_mark = None + else: + batch_x_mark = batch_x_mark.float().to(self.device) + batch_y_mark = batch_y_mark.float().to(self.device) + + # decoder input + dec_inp = torch.zeros_like(batch_y[:, -self.args.pred_len:, :]).float() + dec_inp = torch.cat([batch_y[:, :self.args.label_len, :], dec_inp], dim=1).float().to(self.device) + # encoder - decoder + if self.args.use_amp: + with torch.cuda.amp.autocast(): + if self.args.output_attention: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)[0] + else: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark) + else: + if self.args.output_attention: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)[0] + else: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark) + f_dim = -1 if self.args.features == 'MS' else 0 + outputs = outputs[:, -self.args.pred_len:, f_dim:] + batch_y = batch_y[:, -self.args.pred_len:, f_dim:].to(self.device) + + pred = outputs.detach().cpu() + true = batch_y.detach().cpu() + + loss = criterion(pred, true) + + total_loss.append(loss) + total_loss = np.average(total_loss) + self.model.train() + return total_loss + + def train(self, setting): + train_data, train_loader = self._get_data(flag='train') + vali_data, vali_loader = self._get_data(flag='val') + test_data, test_loader = self._get_data(flag='test') + + path = os.path.join(self.args.checkpoints, setting) + if not os.path.exists(path): + os.makedirs(path) + path2 = os.path.join(self.args.checkpoints2, setting) + if not os.path.exists(path2): + os.makedirs(path2) + best_model_path = path + '/' + 'checkpoint.pth' + best_optimizer_path = path2 + '/' + 'checkpoint.pth' + clear_directory(best_model_path) + clear_directory(best_optimizer_path) + + time_now = time.time() + + train_steps = len(train_loader) + early_stopping = EarlyStopping_v2(patience=self.args.patience, verbose=True) + + model_optim = self._select_optimizer() + criterion = self._select_criterion() + + if self.args.use_amp: + scaler = torch.cuda.amp.GradScaler() + + for epoch in range(self.args.train_epochs): + iter_count = 0 + train_loss = [] + ##################################################### + if epoch == 0: + lr = self.args.learning_rate + print("lerning rate: {}".format(lr)) + ##################################################### + + self.model.train() + epoch_time = time.time() + for i, (batch_x, batch_y, batch_x_mark, batch_y_mark) in enumerate(train_loader): + iter_count += 1 + model_optim.zero_grad() + batch_x = batch_x.float().to(self.device) + batch_y = batch_y.float().to(self.device) + if 'PEMS' in self.args.data or 'Solar' in self.args.data: + batch_x_mark = None + batch_y_mark = None + else: + batch_x_mark = batch_x_mark.float().to(self.device) + batch_y_mark = batch_y_mark.float().to(self.device) + + # decoder input + dec_inp = torch.zeros_like(batch_y[:, -self.args.pred_len:, :]).float() + dec_inp = torch.cat([batch_y[:, :self.args.label_len, :], dec_inp], dim=1).float().to(self.device) + + # encoder - decoder + if self.args.use_amp: + with torch.cuda.amp.autocast(): + if self.args.output_attention: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)[0] + else: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark) + + f_dim = -1 if self.args.features == 'MS' else 0 + outputs = outputs[:, -self.args.pred_len:, f_dim:] + batch_y = batch_y[:, -self.args.pred_len:, f_dim:].to(self.device) + loss = criterion(outputs, batch_y) + train_loss.append(loss.item()) + else: + if self.args.output_attention: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)[0] + else: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark) + + f_dim = -1 if self.args.features == 'MS' else 0 + outputs = outputs[:, -self.args.pred_len:, f_dim:] + batch_y = batch_y[:, -self.args.pred_len:, f_dim:].to(self.device) + loss = criterion(outputs, batch_y) + train_loss.append(loss.item()) + + if (i + 1) % 100 == 0: + print("\titers: {0}, epoch: {1} | loss: {2:.7f}".format(i + 1, epoch + 1, loss.item())) + speed = (time.time() - time_now) / iter_count + left_time = speed * ((self.args.train_epochs - epoch) * train_steps - i) + print('\tspeed: {:.4f}s/iter; left time: {:.4f}s'.format(speed, left_time)) + iter_count = 0 + time_now = time.time() + + if self.args.use_amp: + scaler.scale(loss).backward() + scaler.step(model_optim) + scaler.update() + else: + loss.backward() + model_optim.step() + + print("Epoch: {} cost time: {}".format(epoch + 1, time.time() - epoch_time)) + train_loss = np.average(train_loss) + vali_loss = self.vali(vali_data, vali_loader, criterion) + test_loss = self.vali(test_data, test_loader, criterion) + + print("Epoch: {0}, Steps: {1} | Train Loss: {2:.7f} Vali Loss: {3:.7f} Test Loss: {4:.7f}".format( + epoch + 1, train_steps, train_loss, vali_loss, test_loss)) + + + # Added section ####################################################### + # Score calcuration + score = train_loss*(3/4) + vali_loss*(1/4) + print("SCORE: {}".format(score)) + + if epoch == 0: + best_score = score + + # Reverting + if score > best_score: + print("Score did not improve. Reverting model.") + self.model.load_state_dict(torch.load(best_model_path)) + model_optim.load_state_dict(torch.load(best_optimizer_path)) + lr *= 0.5 + for param_group in model_optim.param_groups: + param_group['lr'] = lr + # Saving + else: + if epoch == 0: + pass + else: + print("Score improved.\n{} >>>>> {}".format(best_score, score)) + SAVE(self.model, model_optim, path, path2) + best_score = score + lr *= 0.5 + for param_group in model_optim.param_groups: + param_group['lr'] = lr + + print('Updating learning rate to {}.'.format(lr)) + ######################################################################## + + # get_cka(self.args, setting, self.model, train_loader, self.device, epoch) + + self.model.load_state_dict(torch.load(best_model_path)) + + return self.model + + def test(self, setting, test=0): + test_data, test_loader = self._get_data(flag='test') + if test: + print('loading model') + self.model.load_state_dict(torch.load(os.path.join('./checkpoints/' + setting, 'checkpoint.pth'))) + + preds = [] + trues = [] + folder_path = './test_results/' + setting + '/' + if not os.path.exists(folder_path): + os.makedirs(folder_path) + + self.model.eval() + with torch.no_grad(): + for i, (batch_x, batch_y, batch_x_mark, batch_y_mark) in enumerate(test_loader): + batch_x = batch_x.float().to(self.device) + batch_y = batch_y.float().to(self.device) + + if 'PEMS' in self.args.data or 'Solar' in self.args.data: + batch_x_mark = None + batch_y_mark = None + else: + batch_x_mark = batch_x_mark.float().to(self.device) + batch_y_mark = batch_y_mark.float().to(self.device) + + # decoder input + dec_inp = torch.zeros_like(batch_y[:, -self.args.pred_len:, :]).float() + dec_inp = torch.cat([batch_y[:, :self.args.label_len, :], dec_inp], dim=1).float().to(self.device) + # encoder - decoder + if self.args.use_amp: + with torch.cuda.amp.autocast(): + if self.args.output_attention: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)[0] + else: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark) + else: + if self.args.output_attention: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)[0] + + else: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark) + + f_dim = -1 if self.args.features == 'MS' else 0 + outputs = outputs[:, -self.args.pred_len:, f_dim:] + batch_y = batch_y[:, -self.args.pred_len:, f_dim:].to(self.device) + outputs = outputs.detach().cpu().numpy() + batch_y = batch_y.detach().cpu().numpy() + if test_data.scale and self.args.inverse: + shape = outputs.shape + outputs = test_data.inverse_transform(outputs.squeeze(0)).reshape(shape) + batch_y = test_data.inverse_transform(batch_y.squeeze(0)).reshape(shape) + + pred = outputs + true = batch_y + + preds.append(pred) + trues.append(true) + if i % 20 == 0: + input = batch_x.detach().cpu().numpy() + if test_data.scale and self.args.inverse: + shape = input.shape + input = test_data.inverse_transform(input.squeeze(0)).reshape(shape) + gt = np.concatenate((input[0, :, -1], true[0, :, -1]), axis=0) + pd = np.concatenate((input[0, :, -1], pred[0, :, -1]), axis=0) + visual(gt, pd, os.path.join(folder_path, str(i) + '.pdf')) + + preds = np.array(preds) + trues = np.array(trues) + print('test shape:', preds.shape, trues.shape) + preds = preds.reshape(-1, preds.shape[-2], preds.shape[-1]) + trues = trues.reshape(-1, trues.shape[-2], trues.shape[-1]) + print('test shape:', preds.shape, trues.shape) + + # result save + folder_path = './results/' + setting + '/' + if not os.path.exists(folder_path): + os.makedirs(folder_path) + + mae, mse, rmse, mape, mspe = metric(preds, trues) + print('mse:{}, mae:{}'.format(mse, mae)) + f = open("result_long_term_forecast.txt", 'a') + f.write(setting + " \n") + f.write('mse:{}, mae:{}'.format(mse, mae)) + f.write('\n') + f.write('\n') + f.close() + + np.save(folder_path + 'metrics.npy', np.array([mae, mse, rmse, mape, mspe])) + np.save(folder_path + 'pred.npy', preds) + np.save(folder_path + 'true.npy', trues) + + return + + + def predict(self, setting, load=False): + pred_data, pred_loader = self._get_data(flag='pred') + + if load: + path = os.path.join(self.args.checkpoints, setting) + best_model_path = path + '/' + 'checkpoint.pth' + self.model.load_state_dict(torch.load(best_model_path)) + + preds = [] + + self.model.eval() + with torch.no_grad(): + for i, (batch_x, batch_y, batch_x_mark, batch_y_mark) in enumerate(pred_loader): + batch_x = batch_x.float().to(self.device) + batch_y = batch_y.float().to(self.device) + batch_x_mark = batch_x_mark.float().to(self.device) + batch_y_mark = batch_y_mark.float().to(self.device) + + # decoder input + dec_inp = torch.zeros_like(batch_y[:, -self.args.pred_len:, :]).float() + dec_inp = torch.cat([batch_y[:, :self.args.label_len, :], dec_inp], dim=1).float().to(self.device) + # encoder - decoder + if self.args.use_amp: + with torch.cuda.amp.autocast(): + if self.args.output_attention: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)[0] + else: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark) + else: + if self.args.output_attention: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)[0] + else: + outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark) + outputs = outputs.detach().cpu().numpy() + if pred_data.scale and self.args.inverse: + shape = outputs.shape + outputs = pred_data.inverse_transform(outputs.squeeze(0)).reshape(shape) + preds.append(outputs) + + preds = np.array(preds) + preds = preds.reshape(-1, preds.shape[-2], preds.shape[-1]) + + # result save + folder_path = './results/' + setting + '/' + if not os.path.exists(folder_path): + os.makedirs(folder_path) + + np.save(folder_path + 'real_prediction.npy', preds) + + return \ No newline at end of file diff --git a/moving/requirements.txt b/moving/requirements.txt new file mode 100644 index 00000000..5d3ac625 --- /dev/null +++ b/moving/requirements.txt @@ -0,0 +1,6 @@ +pandas==1.5.3 +scikit-learn==1.2.2 +numpy==1.23.5 +matplotlib==3.7.0 +reformer-pytorch==1.4.4 +datetime==5.5 \ No newline at end of file diff --git a/moving/reverter.py b/moving/reverter.py new file mode 100644 index 00000000..8a0d3a1f --- /dev/null +++ b/moving/reverter.py @@ -0,0 +1,170 @@ +import argparse +import torch +from experiments.exp_reverter import Exp_Long_Term_Forecast +from experiments.exp_long_term_forecasting_partial import Exp_Long_Term_Forecast_Partial +import random +import numpy as np + +if __name__ == '__main__': + fix_seed = 2023 + random.seed(fix_seed) + torch.manual_seed(fix_seed) + np.random.seed(fix_seed) + + parser = argparse.ArgumentParser(description='iTransformer') + + # basic config + parser.add_argument('--is_training', type=int, required=True, default=1, help='status') + parser.add_argument('--model_id', type=str, required=True, default='test', help='model id') + parser.add_argument('--model', type=str, required=True, default='iTransformer', + help='model name, options: [iTransformer, iInformer, iReformer, iFlowformer, iFlashformer, RxiTransformer]') + + # data loader + parser.add_argument('--data', type=str, required=True, default='custom', help='dataset type') + parser.add_argument('--root_path', type=str, default='./data/electricity/', help='root path of the data file') + parser.add_argument('--data_path', type=str, default='electricity.csv', help='data csv file') + parser.add_argument('--features', type=str, default='M', + help='forecasting task, options:[M, S, MS]; M:multivariate predict multivariate, S:univariate predict univariate, MS:multivariate predict univariate') + parser.add_argument('--target', type=str, default='OT', help='target feature in S or MS task') + parser.add_argument('--freq', type=str, default='h', + help='freq for time features encoding, options:[s:secondly, t:minutely, h:hourly, d:daily, b:business days, w:weekly, m:monthly], you can also use more detailed freq like 15min or 3h') + parser.add_argument('--checkpoints', type=str, default='./checkpoints/', help='location of model checkpoints') + parser.add_argument('--checkpoints2', type=str, default='./checkpoints2/', help='location of optimizer checkpoints') + + # forecasting task + parser.add_argument('--seq_len', type=int, default=96, help='input sequence length') + parser.add_argument('--label_len', type=int, default=48, help='start token length') # no longer needed in inverted Transformers + parser.add_argument('--pred_len', type=int, default=96, help='prediction sequence length') + + # model define + parser.add_argument('--enc_in', type=int, default=7, help='encoder input size') + parser.add_argument('--dec_in', type=int, default=7, help='decoder input size') + parser.add_argument('--c_out', type=int, default=7, help='output size') # applicable on arbitrary number of variates in inverted Transformers + parser.add_argument('--d_model', type=int, default=512, help='dimension of model') + parser.add_argument('--n_heads', type=int, default=8, help='num of heads') + parser.add_argument('--e_layers', type=int, default=2, help='num of encoder layers') + parser.add_argument('--d_layers', type=int, default=1, help='num of decoder layers') + parser.add_argument('--d_ff', type=int, default=2048, help='dimension of fcn') + parser.add_argument('--moving_avg', type=int, default=25, help='window size of moving average') + parser.add_argument('--factor', type=int, default=1, help='attn factor') + parser.add_argument('--distil', action='store_false', + help='whether to use distilling in encoder, using this argument means not using distilling', + default=True) + parser.add_argument('--dropout', type=float, default=0.1, help='dropout') + parser.add_argument('--embed', type=str, default='timeF', + help='time features encoding, options:[timeF, fixed, learned]') + parser.add_argument('--activation', type=str, default='gelu', help='activation') + parser.add_argument('--output_attention', action='store_true', help='whether to output attention in ecoder') + parser.add_argument('--do_predict', action='store_true', help='whether to predict unseen future data') + + # optimization + parser.add_argument('--num_workers', type=int, default=10, help='data loader num workers') + parser.add_argument('--itr', type=int, default=1, help='experiments times') + parser.add_argument('--train_epochs', type=int, default=10, help='train epochs') + parser.add_argument('--batch_size', type=int, default=32, help='batch size of train input data') + parser.add_argument('--patience', type=int, default=3, help='early stopping patience') + parser.add_argument('--learning_rate', type=float, default=0.01, help='optimizer learning rate') + parser.add_argument('--des', type=str, default='test', help='exp description') + parser.add_argument('--loss', type=str, default='MSE', help='loss function') + parser.add_argument('--lradj', type=str, default='type1', help='adjust learning rate') + parser.add_argument('--use_amp', action='store_true', help='use automatic mixed precision training', default=False) + + # GPU + parser.add_argument('--use_gpu', type=bool, default=True, help='use gpu') + parser.add_argument('--gpu', type=int, default=0, help='gpu') + parser.add_argument('--use_multi_gpu', action='store_true', help='use multiple gpus', default=False) + parser.add_argument('--devices', type=str, default='0,1,2,3', help='device ids of multile gpus') + + # iTransformer + parser.add_argument('--exp_name', type=str, required=False, default='MTSF', + help='experiemnt name, options:[MTSF, partial_train]') + parser.add_argument('--channel_independence', type=bool, default=False, help='whether to use channel_independence mechanism') + parser.add_argument('--inverse', action='store_true', help='inverse output data', default=False) + parser.add_argument('--class_strategy', type=str, default='projection', help='projection/average/cls_token') + parser.add_argument('--target_root_path', type=str, default='./data/electricity/', help='root path of the data file') + parser.add_argument('--target_data_path', type=str, default='electricity.csv', help='data file') + parser.add_argument('--efficient_training', type=bool, default=False, help='whether to use efficient_training (exp_name should be partial train)') # See Figure 8 of our paper for the detail + parser.add_argument('--use_norm', type=int, default=True, help='use norm and denorm') + parser.add_argument('--partial_start_index', type=int, default=0, help='the start index of variates for partial training, ' + 'you can select [partial_start_index, min(enc_in + partial_start_index, N)]') + + # RevIN + parser.add_argument('--rev', default=True, action='store_true', help='whether to apply RevIN') + + args = parser.parse_args() + args.use_gpu = True if torch.cuda.is_available() and args.use_gpu else False + + if args.use_gpu and args.use_multi_gpu: + args.devices = args.devices.replace(' ', '') + device_ids = args.devices.split(',') + args.device_ids = [int(id_) for id_ in device_ids] + args.gpu = args.device_ids[0] + + print('Args in experiment:') + print(args) + + if args.exp_name == 'partial_train': # See Figure 8 of our paper, for the detail + Exp = Exp_Long_Term_Forecast_Partial + else: # MTSF: multivariate time series forecasting + Exp = Exp_Long_Term_Forecast + + + if args.is_training: + for ii in range(args.itr): + # setting record of experiments + setting = '{}_{}_{}_{}_ft{}_sl{}_ll{}_pl{}_dm{}_nh{}_el{}_dl{}_df{}_fc{}_eb{}_dt{}_{}_{}'.format( + args.model_id, + args.model, + args.data, + args.features, + args.seq_len, + args.label_len, + args.pred_len, + args.d_model, + args.n_heads, + args.e_layers, + args.d_layers, + args.d_ff, + args.factor, + args.embed, + args.distil, + args.des, + args.class_strategy, ii) + + exp = Exp(args) # set experiments + print('>>>>>>>start training : {}>>>>>>>>>>>>>>>>>>>>>>>>>>'.format(setting)) + exp.train(setting) + + print('>>>>>>>testing : {}<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<'.format(setting)) + exp.test(setting) + + if args.do_predict: + print('>>>>>>>predicting : {}<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<'.format(setting)) + exp.predict(setting, True) + + torch.cuda.empty_cache() + else: + ii = 0 + setting = '{}_{}_{}_{}_ft{}_sl{}_ll{}_pl{}_dm{}_nh{}_el{}_dl{}_df{}_fc{}_eb{}_dt{}_{}_{}'.format( + args.model_id, + args.model, + args.data, + args.features, + args.seq_len, + args.label_len, + args.pred_len, + args.d_model, + args.n_heads, + args.e_layers, + args.d_layers, + args.d_ff, + args.factor, + args.embed, + args.distil, + args.des, + args.class_strategy, ii) + + exp = Exp(args) # set experiments + print('>>>>>>>testing : {}<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<'.format(setting)) + exp.test(setting, test=1) + torch.cuda.empty_cache() diff --git a/moving/torch.txt b/moving/torch.txt new file mode 100644 index 00000000..a6b1024d --- /dev/null +++ b/moving/torch.txt @@ -0,0 +1,4 @@ +--index-url https://download.pytorch.org/whl/cu118 +torch +torchvision +torchaudio \ No newline at end of file From 25b6a941185b4c427cbb1abcd10729028eb10c31 Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Fri, 7 Mar 2025 20:37:44 +0900 Subject: [PATCH 05/14] =?UTF-8?q?=E3=83=95=E3=82=A1=E3=82=A4=E3=83=AB?= =?UTF-8?q?=E3=81=AE=E7=BD=AE=E3=81=8D=E5=A0=B4=E3=81=AE=E4=BF=AE=E6=AD=A3?= =?UTF-8?q?=E3=80=81RxiTransformer=E3=81=A8RxiTransformer2=E3=81=AE?= =?UTF-8?q?=E8=A1=A8=E8=A8=98=E3=81=AE=E7=B5=B1=E4=B8=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- experiments/exp_basic.py | 5 +- {moving => experiments}/exp_reverter.py | 0 layers/Invertible.py | 58 +++++++++++++++++++ {moving => model}/RxiTransformer.py | 22 +++---- .../RxiTransformer2.py | 22 ++++--- moving/requirements.txt | 6 -- moving/torch.txt | 4 -- requirements.txt | 6 -- moving/reverter.py => run_reverter.py | 0 9 files changed, 86 insertions(+), 37 deletions(-) rename {moving => experiments}/exp_reverter.py (100%) create mode 100644 layers/Invertible.py rename {moving => model}/RxiTransformer.py (87%) rename moving/RxiTransformer7.py => model/RxiTransformer2.py (83%) delete mode 100644 moving/requirements.txt delete mode 100644 moving/torch.txt delete mode 100644 requirements.txt rename moving/reverter.py => run_reverter.py (100%) diff --git a/experiments/exp_basic.py b/experiments/exp_basic.py index 935d65ef..cee7e06b 100644 --- a/experiments/exp_basic.py +++ b/experiments/exp_basic.py @@ -1,7 +1,8 @@ import os import torch from model import Transformer, Informer, Reformer, Flowformer, Flashformer, \ - iTransformer, iInformer, iReformer, iFlowformer, iFlashformer + iTransformer, iInformer, iReformer, iFlowformer, iFlashformer, \ + RxiTransformer, RxiTransformer2 class Exp_Basic(object): @@ -18,6 +19,8 @@ def __init__(self, args): 'iReformer': iReformer, 'iFlowformer': iFlowformer, 'iFlashformer': iFlashformer, + 'RxiTransformer': RxiTransformer, + 'RxiTransformer2': RxiTransformer2, } self.device = self._acquire_device() self.model = self._build_model().to(self.device) diff --git a/moving/exp_reverter.py b/experiments/exp_reverter.py similarity index 100% rename from moving/exp_reverter.py rename to experiments/exp_reverter.py diff --git a/layers/Invertible.py b/layers/Invertible.py new file mode 100644 index 00000000..70a63ed6 --- /dev/null +++ b/layers/Invertible.py @@ -0,0 +1,58 @@ +import torch +import torch.nn as nn + +class RevIN(nn.Module): + def __init__(self, num_features: int, eps=1e-5, affine=True): + """ + :param num_features: the number of features or channels + :param eps: a value added for numerical stability + :param affine: if True, RevIN has learnable affine parameters + """ + super(RevIN, self).__init__() + + self.num_features = num_features + self.eps = eps + self.affine = affine + + if self.affine: + self._init_params() + + def forward(self, x, mode:str): + if mode == 'norm': + self._get_statistics(x) + x = self._normalize(x) + + elif mode == 'denorm': + x = self._denormalize(x) + + else: raise NotImplementedError + + return x + + def _init_params(self): + # initialize RevIN params: (C,) + self.affine_weight = nn.Parameter(torch.ones(self.num_features)) + self.affine_bias = nn.Parameter(torch.zeros(self.num_features)) + + def _get_statistics(self, x): + dim2reduce = tuple(range(1, x.ndim-1)) + self.mean = torch.mean(x, dim=dim2reduce, keepdim=True).detach() + self.stdev = torch.sqrt(torch.var(x, dim=dim2reduce, keepdim=True, unbiased=False) + self.eps).detach() + + def _normalize(self, x): + x = x - self.mean + x = x / self.stdev + if self.affine: + x = x * self.affine_weight + x = x + self.affine_bias + + return x + + def _denormalize(self, x): + if self.affine: + x = x - self.affine_bias + x = x / (self.affine_weight + self.eps*self.eps) + x = x * self.stdev + x = x + self.mean + + return x diff --git a/moving/RxiTransformer.py b/model/RxiTransformer.py similarity index 87% rename from moving/RxiTransformer.py rename to model/RxiTransformer.py index 20eb46b7..4d0a1b57 100644 --- a/moving/RxiTransformer.py +++ b/model/RxiTransformer.py @@ -10,6 +10,7 @@ class Model(nn.Module): # Parrarel model of iTransformer and RLinear-CI + # Simply added version def __init__(self, configs): super(Model, self).__init__() @@ -46,13 +47,13 @@ def __init__(self, configs): self.rev = RevIN(configs.enc_in) if configs.rev else None def forecast(self, x_enc, x_mark_enc, x_dec, x_mark_dec): - # Saving x_enc x_enc_save = x_enc - #iTransformer + # Normalization from RevIN if self.use_norm: - # Normalization from RevIN x_normed = self.rev(x_enc_save, 'norm') if self.rev else x_enc_save + + #iTransformer _, _, N = x_enc.shape # B L N # B: batch_size; E: d_model; # L: seq_len; S: pred_len; # N: number of variate (tokens), can also includes covariates @@ -60,20 +61,19 @@ def forecast(self, x_enc, x_mark_enc, x_dec, x_mark_dec): enc_out = self.enc_embedding(x_normed, x_mark_enc) enc_out, attns = self.encoder(enc_out, attn_mask=None) enc_out = self.projector(enc_out).permute(0, 2, 1)[:, :, :N] - if self.use_norm: - # De-Normalization from Non-stationary Transformer - enc_out = self.rev(enc_out, 'denorm') if self.rev else enc_out - # RLinear-CI + # RLinear-CI x = self.dropout(x_normed) - pred = torch.zeros(x.size(0), self.pred_len, x.size(2)).to('cuda') + rl_out = torch.zeros(x.size(0), self.pred_len, x.size(2)).to('cuda') for idx, proj in enumerate(self.Linear): - pred[:, :, idx] = proj(x[:, :, idx]) - rl_out = self.rev(pred, 'denorm') if self.rev else pred - + rl_out[:, :, idx] = proj(x[:, :, idx]) + # Add out = enc_out*0.5 + rl_out*0.5 + # De-Normalization from RevIN + if self.use_norm: + out = self.rev(enc_out, 'denorm') if self.rev else out return out diff --git a/moving/RxiTransformer7.py b/model/RxiTransformer2.py similarity index 83% rename from moving/RxiTransformer7.py rename to model/RxiTransformer2.py index 91081d84..fe525dfb 100644 --- a/moving/RxiTransformer7.py +++ b/model/RxiTransformer2.py @@ -10,6 +10,7 @@ class Model(nn.Module): # Parrarel model of iTransformer and RLinear-CI + # Weighted and added version def __init__(self, configs): super(Model, self).__init__() @@ -46,21 +47,24 @@ def __init__(self, configs): self.dropout = nn.Dropout(configs.dropout) self.rev = RevIN(configs.enc_in) if configs.rev else None - # DistributionLayer + # DistributionLayer(Weighting and Add) distributer = [] for i in range(configs.enc_in): distributer.append(0.5) self.distributer = nn.Parameter(torch.tensor(distributer)) def forecast(self, x_enc, x_mark_enc, x_dec, x_mark_dec): - # Saving x_enc x_enc_save = x_enc - # Normalization - x_normed = self.rev(x_enc_save, 'norm') if self.rev else x_enc_save + # Normalization from RevIN + if self.use_norm: + x_normed = self.rev(x_enc_save, 'norm') if self.rev else x_enc_save # iTransformer - _, _, N = x_enc.shape + _, _, N = x_enc.shape # B L N + # B: batch_size; E: d_model; # L: seq_len; S: pred_len; + # N: number of variate (tokens), can also includes covariates + # B L N -> B S N enc_out = self.enc_embedding(x_normed, x_mark_enc) enc_out, attns = self.encoder(enc_out, attn_mask=None) enc_out = self.projector(enc_out).permute(0, 2, 1)[:, :, :N] @@ -71,15 +75,15 @@ def forecast(self, x_enc, x_mark_enc, x_dec, x_mark_dec): for idx, proj in enumerate(self.Linear): rl_out[:, :, idx] = proj(x[:, :, idx]) - # Add + # Wegihting and Add out = torch.zeros_like(enc_out) for i in range(self.enc_in): rate = self.distributer[i] out[:, :, i] = enc_out[:, :, i]*rate + rl_out[:, :, i]*(1 - rate) - # Denormalization - out = self.rev(out, 'denorm') if self.rev else out - + # De-Normalization from RevIN + if self.use_norm: + out = self.rev(enc_out, 'denorm') if self.rev else out return out diff --git a/moving/requirements.txt b/moving/requirements.txt deleted file mode 100644 index 5d3ac625..00000000 --- a/moving/requirements.txt +++ /dev/null @@ -1,6 +0,0 @@ -pandas==1.5.3 -scikit-learn==1.2.2 -numpy==1.23.5 -matplotlib==3.7.0 -reformer-pytorch==1.4.4 -datetime==5.5 \ No newline at end of file diff --git a/moving/torch.txt b/moving/torch.txt deleted file mode 100644 index a6b1024d..00000000 --- a/moving/torch.txt +++ /dev/null @@ -1,4 +0,0 @@ ---index-url https://download.pytorch.org/whl/cu118 -torch -torchvision -torchaudio \ No newline at end of file diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index 3a8ff527..00000000 --- a/requirements.txt +++ /dev/null @@ -1,6 +0,0 @@ -pandas==1.5.3 -scikit-learn==1.2.2 -numpy==1.23.5 -matplotlib==3.7.0 -torch==2.0.0 -reformer-pytorch==1.4.4 diff --git a/moving/reverter.py b/run_reverter.py similarity index 100% rename from moving/reverter.py rename to run_reverter.py From 524fb13020a89991e9af8c6488fd73f208cd7513 Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Fri, 7 Mar 2025 20:40:50 +0900 Subject: [PATCH 06/14] =?UTF-8?q?RLinear-CI=E3=81=AE=E8=BF=BD=E5=8A=A0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- experiments/exp_basic.py | 3 ++- model/RLinearCI.py | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) create mode 100644 model/RLinearCI.py diff --git a/experiments/exp_basic.py b/experiments/exp_basic.py index cee7e06b..09fd3faf 100644 --- a/experiments/exp_basic.py +++ b/experiments/exp_basic.py @@ -2,7 +2,7 @@ import torch from model import Transformer, Informer, Reformer, Flowformer, Flashformer, \ iTransformer, iInformer, iReformer, iFlowformer, iFlashformer, \ - RxiTransformer, RxiTransformer2 + RxiTransformer, RxiTransformer2, RLinearCI class Exp_Basic(object): @@ -21,6 +21,7 @@ def __init__(self, args): 'iFlashformer': iFlashformer, 'RxiTransformer': RxiTransformer, 'RxiTransformer2': RxiTransformer2, + 'RLinearCI': RLinearCI, } self.device = self._acquire_device() self.model = self._build_model().to(self.device) diff --git a/model/RLinearCI.py b/model/RLinearCI.py new file mode 100644 index 00000000..940a0137 --- /dev/null +++ b/model/RLinearCI.py @@ -0,0 +1,33 @@ +import torch +import torch.nn as nn +import torch.nn.functional as F +from layers.Invertible import RevIN + + +class Model(nn.Module): + def __init__(self, configs): + super(Model, self).__init__() + + self.Linear = nn.ModuleList([ + nn.Linear(configs.seq_len, configs.pred_len) for _ in range(configs.enc_in) + ]) + + self.dropout = nn.Dropout(configs.dropout) + self.rev = RevIN(configs.enc_in) + self.pred_len = configs.pred_len + + def forecast(self, x_enc, x_mark_enc, x_dec, x_mark_dec): + # x: [B, L, D] + x = self.rev(x_enc, 'norm') if self.rev else x_enc + x = self.dropout(x) + pred = torch.zeros(x.size(0), self.pred_len, x.size(2)).to('cuda') + for idx, proj in enumerate(self.Linear): + pred[:, :, idx] = proj(x[:, :, idx]) + pred = self.rev(pred, 'denorm') if self.rev else pred + + return pred + + + def forward(self, x_enc, x_mark_enc, x_dec, x_mark_dec, mask=None): + dec_out = self.forecast(x_enc, x_mark_enc, x_dec, x_mark_dec) + return dec_out[:, -self.pred_len:, :] # [B, L, D] \ No newline at end of file From 2f6f0f73aab71745eb443cb6e2c427fe254f2ece Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Fri, 7 Mar 2025 20:47:57 +0900 Subject: [PATCH 07/14] README is revised. --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index 8f8b0fff..8eaa35e1 100644 --- a/README.md +++ b/README.md @@ -12,3 +12,8 @@ RxiTransformer is a machine learning model for time series forecasting that comb # Learnign Rate Scheduler: ReduceLRReverter # Minor Changes + +-exp_basic.py is revised to use added models. +-exp_reverter.py is added for impelmentation of ReduceLRReverter +-run_reverter.py is added for impelmentation of ReduceLRReverter +-requirements.txt is revised and torch is installed by torch.txt \ No newline at end of file From 6a42752a716b6010f6e933fd223bdf89d67a017c Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Fri, 7 Mar 2025 20:48:41 +0900 Subject: [PATCH 08/14] README is revised. --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 8eaa35e1..09299bf7 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,9 @@ RxiTransformer is a machine learning model for time series forecasting that comb # Minor Changes -exp_basic.py is revised to use added models. + -exp_reverter.py is added for impelmentation of ReduceLRReverter + -run_reverter.py is added for impelmentation of ReduceLRReverter + -requirements.txt is revised and torch is installed by torch.txt \ No newline at end of file From 2dca7038250bc2c4fbdba97187856910254b6f6b Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Fri, 7 Mar 2025 21:26:38 +0900 Subject: [PATCH 09/14] so many changes --- README.md | 6 ++++-- experiments/exp_long_term_forecasting.py | 5 +++++ experiments/exp_reverter.py | 5 +++++ run.py | 3 +++ 4 files changed, 17 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 09299bf7..b51e9926 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ The main changes are described below. RxiTransformer is a machine learning model for time series forecasting that combines elements of RLinear(https://github.com/plumprc/RTSF) with iTransformer(https://github.com/thuml/iTransformer). -# Learnign Rate Scheduler: ReduceLRReverter +# Learnig Rate Scheduler: ReduceLRReverter # Minor Changes @@ -19,4 +19,6 @@ RxiTransformer is a machine learning model for time series forecasting that comb -run_reverter.py is added for impelmentation of ReduceLRReverter --requirements.txt is revised and torch is installed by torch.txt \ No newline at end of file +-requirements.txt is revised and torch is installed by torch.txt + +-In run.py, "--rev" argument is added. \ No newline at end of file diff --git a/experiments/exp_long_term_forecasting.py b/experiments/exp_long_term_forecasting.py index 35170c6a..814b42b5 100644 --- a/experiments/exp_long_term_forecasting.py +++ b/experiments/exp_long_term_forecasting.py @@ -162,6 +162,11 @@ def train(self, setting): loss.backward() model_optim.step() + # Recording the time taken for the epoch + f = open("result_long_term_forecast.txt", 'a') + f.write("Epoch: {} cost time: {}".format(epoch + 1, time.time() - epoch_time) + "\n") + f.close() + print("Epoch: {} cost time: {}".format(epoch + 1, time.time() - epoch_time)) train_loss = np.average(train_loss) vali_loss = self.vali(vali_data, vali_loader, criterion) diff --git a/experiments/exp_reverter.py b/experiments/exp_reverter.py index 87e01e46..00fcc883 100644 --- a/experiments/exp_reverter.py +++ b/experiments/exp_reverter.py @@ -174,6 +174,11 @@ def train(self, setting): loss.backward() model_optim.step() + # Recording the time taken for the epoch + f = open("result_long_term_forecast.txt", 'a') + f.write("Epoch: {} cost time: {}".format(epoch + 1, time.time() - epoch_time) + "\n") + f.close() + print("Epoch: {} cost time: {}".format(epoch + 1, time.time() - epoch_time)) train_loss = np.average(train_loss) vali_loss = self.vali(vali_data, vali_loader, criterion) diff --git a/run.py b/run.py index a50bbdbf..3fb088df 100644 --- a/run.py +++ b/run.py @@ -87,6 +87,9 @@ parser.add_argument('--partial_start_index', type=int, default=0, help='the start index of variates for partial training, ' 'you can select [partial_start_index, min(enc_in + partial_start_index, N)]') + # RevIN + parser.add_argument('--rev', default=True, action='store_true', help='whether to apply RevIN') + args = parser.parse_args() args.use_gpu = True if torch.cuda.is_available() and args.use_gpu else False From de9d342868a3b70b39c8782ccc3888f1a7f23364 Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Fri, 7 Mar 2025 21:28:16 +0900 Subject: [PATCH 10/14] revising README --- README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index b51e9926..4e453749 100644 --- a/README.md +++ b/README.md @@ -15,10 +15,12 @@ RxiTransformer is a machine learning model for time series forecasting that comb -exp_basic.py is revised to use added models. --exp_reverter.py is added for impelmentation of ReduceLRReverter +-exp_reverter.py is added for impelmentation of ReduceLRReverter . --run_reverter.py is added for impelmentation of ReduceLRReverter +-run_reverter.py is added for impelmentation of ReduceLRReverter . --requirements.txt is revised and torch is installed by torch.txt +-requirements.txt is revised and torch is installed by torch.txt . --In run.py, "--rev" argument is added. \ No newline at end of file +-In run.py, "--rev" argument is added. + +-exp_long_term_forecasting is changed to measure the time taken for each epochs. \ No newline at end of file From 891db49abb2c6be3c970002712feb8abe6a2f6dc Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Fri, 7 Mar 2025 22:46:33 +0900 Subject: [PATCH 11/14] removing unneccessary import --- experiments/exp_reverter.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/experiments/exp_reverter.py b/experiments/exp_reverter.py index 00fcc883..ba38a46c 100644 --- a/experiments/exp_reverter.py +++ b/experiments/exp_reverter.py @@ -1,6 +1,6 @@ from data_provider.data_factory import data_provider from experiments.exp_basic import Exp_Basic -from utils.tools import EarlyStopping_v2, visual, clear_directory, SAVE +from utils.tools import visual, clear_directory, SAVE from utils.metrics import metric import torch import torch.nn as nn From 9f7e421fc44ed7e7cc0b3a6496577327234d9595 Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Fri, 7 Mar 2025 22:54:07 +0900 Subject: [PATCH 12/14] some adjustments --- experiments/exp_reverter.py | 1 - utils/tools.py | 29 +++++++++++++++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/experiments/exp_reverter.py b/experiments/exp_reverter.py index ba38a46c..2fdfa0cb 100644 --- a/experiments/exp_reverter.py +++ b/experiments/exp_reverter.py @@ -98,7 +98,6 @@ def train(self, setting): time_now = time.time() train_steps = len(train_loader) - early_stopping = EarlyStopping_v2(patience=self.args.patience, verbose=True) model_optim = self._select_optimizer() criterion = self._select_criterion() diff --git a/utils/tools.py b/utils/tools.py index c3b922ec..604a405b 100644 --- a/utils/tools.py +++ b/utils/tools.py @@ -4,6 +4,7 @@ import torch import matplotlib.pyplot as plt import pandas as pd +import shutil plt.switch_backend('agg') @@ -113,3 +114,31 @@ def adjustment(gt, pred): def cal_accuracy(y_pred, y_true): return np.mean(y_pred == y_true) + + +### Added section ####################################################### +def clear_directory(path): + if not os.path.exists(path): + print(f"no directory: {path}") + return + + if not os.path.isdir(path): + print(f"not directory: {path}") + return + + # clearing the directory + for filename in os.listdir(path): + file_path = os.path.join(path, filename) + try: + if os.path.isfile(file_path) or os.path.islink(file_path): + os.unlink(file_path) + elif os.path.isdir(file_path): + shutil.rmtree(file_path) + except Exception as e: + print(f"error: {file_path} - {e}") + +def SAVE(model, optimizer, path, path2): + print('SAVE: Saving model ...') + torch.save(model.state_dict(), path + '/' + 'checkpoint.pth') + torch.save(optimizer.state_dict(), path2 + '/' + 'checkpoint.pth') + return None \ No newline at end of file From b3c1d5fb7b32c7855cd84851d8cc4e6d47c528c9 Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Sat, 8 Mar 2025 11:15:48 +0900 Subject: [PATCH 13/14] 0308 --- .gitignore | 11 +- run_reverter.py | 2 +- scripts/boost_performance/ECL/iFlowformer.sh | 72 ------- scripts/boost_performance/ECL/iInformer.sh | 72 ------- scripts/boost_performance/ECL/iReformer.sh | 72 ------- scripts/boost_performance/ECL/iTransformer.sh | 72 ------- scripts/boost_performance/README.md | 29 --- .../boost_performance/Traffic/iFlowformer.sh | 76 -------- .../boost_performance/Traffic/iInformer.sh | 76 -------- .../boost_performance/Traffic/iReformer.sh | 76 -------- .../boost_performance/Traffic/iTransformer.sh | 76 -------- .../boost_performance/Weather/iFlowformer.sh | 77 -------- .../boost_performance/Weather/iInformer.sh | 77 -------- .../boost_performance/Weather/iReformer.sh | 77 -------- .../boost_performance/Weather/iTransformer.sh | 77 -------- .../increasing_lookback/ECL/iFlowformer.sh | 109 ----------- scripts/increasing_lookback/ECL/iInformer.sh | 109 ----------- scripts/increasing_lookback/ECL/iReformer.sh | 109 ----------- .../increasing_lookback/ECL/iTransformer.sh | 109 ----------- scripts/increasing_lookback/README.md | 23 --- .../Traffic/iFlowformer.sh | 109 ----------- .../increasing_lookback/Traffic/iInformer.sh | 109 ----------- .../increasing_lookback/Traffic/iReformer.sh | 109 ----------- .../Traffic/iTransformer.sh | 109 ----------- .../model_efficiency/ECL/iFlashTransformer.sh | 165 ----------------- scripts/model_efficiency/README.md | 40 ---- .../Traffic/iFlashTransformer.sh | 173 ----------------- .../Weather/iFlashTransformer.sh | 175 ------------------ .../ECL/iTransformer.sh | 89 --------- .../ETT/iTransformer_ETTh1.sh | 79 -------- .../ETT/iTransformer_ETTh2.sh | 79 -------- .../ETT/iTransformer_ETTm1.sh | 79 -------- .../ETT/iTransformer_ETTm2.sh | 79 -------- .../Exchange/iTransformer.sh | 80 -------- .../PEMS/iTransformer_03.sh | 85 --------- .../PEMS/iTransformer_04.sh | 87 --------- .../PEMS/iTransformer_07.sh | 89 --------- .../PEMS/iTransformer_08.sh | 87 --------- scripts/multivariate_forecasting/README.md | 57 ------ .../SolarEnergy/iTransformer.sh | 83 --------- .../Traffic/iTransformer.sh | 87 --------- .../Weather/iTransformer.sh | 82 -------- .../variate_generalization/ECL/iFlowformer.sh | 91 --------- .../variate_generalization/ECL/iInformer.sh | 91 --------- .../variate_generalization/ECL/iReformer.sh | 91 --------- .../ECL/iTransformer.sh | 92 --------- scripts/variate_generalization/README.md | 37 ---- .../SolarEnergy/iFlowformer.sh | 95 ---------- .../SolarEnergy/iInformer.sh | 95 ---------- .../SolarEnergy/iReformer.sh | 95 ---------- .../SolarEnergy/iTransformer.sh | 96 ---------- .../Traffic/iFlowformer.sh | 91 --------- .../Traffic/iInformer.sh | 91 --------- .../Traffic/iReformer.sh | 91 --------- .../Traffic/iTransformer.sh | 92 --------- 55 files changed, 11 insertions(+), 4669 deletions(-) delete mode 100644 scripts/boost_performance/ECL/iFlowformer.sh delete mode 100644 scripts/boost_performance/ECL/iInformer.sh delete mode 100644 scripts/boost_performance/ECL/iReformer.sh delete mode 100644 scripts/boost_performance/ECL/iTransformer.sh delete mode 100644 scripts/boost_performance/README.md delete mode 100644 scripts/boost_performance/Traffic/iFlowformer.sh delete mode 100644 scripts/boost_performance/Traffic/iInformer.sh delete mode 100644 scripts/boost_performance/Traffic/iReformer.sh delete mode 100644 scripts/boost_performance/Traffic/iTransformer.sh delete mode 100644 scripts/boost_performance/Weather/iFlowformer.sh delete mode 100644 scripts/boost_performance/Weather/iInformer.sh delete mode 100644 scripts/boost_performance/Weather/iReformer.sh delete mode 100644 scripts/boost_performance/Weather/iTransformer.sh delete mode 100644 scripts/increasing_lookback/ECL/iFlowformer.sh delete mode 100644 scripts/increasing_lookback/ECL/iInformer.sh delete mode 100644 scripts/increasing_lookback/ECL/iReformer.sh delete mode 100644 scripts/increasing_lookback/ECL/iTransformer.sh delete mode 100644 scripts/increasing_lookback/README.md delete mode 100644 scripts/increasing_lookback/Traffic/iFlowformer.sh delete mode 100644 scripts/increasing_lookback/Traffic/iInformer.sh delete mode 100644 scripts/increasing_lookback/Traffic/iReformer.sh delete mode 100644 scripts/increasing_lookback/Traffic/iTransformer.sh delete mode 100644 scripts/model_efficiency/ECL/iFlashTransformer.sh delete mode 100644 scripts/model_efficiency/README.md delete mode 100644 scripts/model_efficiency/Traffic/iFlashTransformer.sh delete mode 100644 scripts/model_efficiency/Weather/iFlashTransformer.sh delete mode 100644 scripts/multivariate_forecasting/ECL/iTransformer.sh delete mode 100644 scripts/multivariate_forecasting/ETT/iTransformer_ETTh1.sh delete mode 100644 scripts/multivariate_forecasting/ETT/iTransformer_ETTh2.sh delete mode 100644 scripts/multivariate_forecasting/ETT/iTransformer_ETTm1.sh delete mode 100644 scripts/multivariate_forecasting/ETT/iTransformer_ETTm2.sh delete mode 100644 scripts/multivariate_forecasting/Exchange/iTransformer.sh delete mode 100644 scripts/multivariate_forecasting/PEMS/iTransformer_03.sh delete mode 100644 scripts/multivariate_forecasting/PEMS/iTransformer_04.sh delete mode 100644 scripts/multivariate_forecasting/PEMS/iTransformer_07.sh delete mode 100644 scripts/multivariate_forecasting/PEMS/iTransformer_08.sh delete mode 100644 scripts/multivariate_forecasting/README.md delete mode 100644 scripts/multivariate_forecasting/SolarEnergy/iTransformer.sh delete mode 100644 scripts/multivariate_forecasting/Traffic/iTransformer.sh delete mode 100644 scripts/multivariate_forecasting/Weather/iTransformer.sh delete mode 100644 scripts/variate_generalization/ECL/iFlowformer.sh delete mode 100644 scripts/variate_generalization/ECL/iInformer.sh delete mode 100644 scripts/variate_generalization/ECL/iReformer.sh delete mode 100644 scripts/variate_generalization/ECL/iTransformer.sh delete mode 100644 scripts/variate_generalization/README.md delete mode 100644 scripts/variate_generalization/SolarEnergy/iFlowformer.sh delete mode 100644 scripts/variate_generalization/SolarEnergy/iInformer.sh delete mode 100644 scripts/variate_generalization/SolarEnergy/iReformer.sh delete mode 100644 scripts/variate_generalization/SolarEnergy/iTransformer.sh delete mode 100644 scripts/variate_generalization/Traffic/iFlowformer.sh delete mode 100644 scripts/variate_generalization/Traffic/iInformer.sh delete mode 100644 scripts/variate_generalization/Traffic/iReformer.sh delete mode 100644 scripts/variate_generalization/Traffic/iTransformer.sh diff --git a/.gitignore b/.gitignore index 104809f8..50718472 100644 --- a/.gitignore +++ b/.gitignore @@ -128,4 +128,13 @@ venv.bak/ dmypy.json # Pyre type checker -.pyre/ \ No newline at end of file +.pyre/ + +############################################### +/result_long_term_forecast.txt +/dataset +/scripts +/checkpoints +/checkpoints2 +/results +/test_results \ No newline at end of file diff --git a/run_reverter.py b/run_reverter.py index 8a0d3a1f..7698a5b3 100644 --- a/run_reverter.py +++ b/run_reverter.py @@ -63,7 +63,7 @@ parser.add_argument('--train_epochs', type=int, default=10, help='train epochs') parser.add_argument('--batch_size', type=int, default=32, help='batch size of train input data') parser.add_argument('--patience', type=int, default=3, help='early stopping patience') - parser.add_argument('--learning_rate', type=float, default=0.01, help='optimizer learning rate') + parser.add_argument('--learning_rate', type=float, default=0.001, help='optimizer learning rate') parser.add_argument('--des', type=str, default='test', help='exp description') parser.add_argument('--loss', type=str, default='MSE', help='loss function') parser.add_argument('--lradj', type=str, default='type1', help='adjust learning rate') diff --git a/scripts/boost_performance/ECL/iFlowformer.sh b/scripts/boost_performance/ECL/iFlowformer.sh deleted file mode 100644 index a3ad90e2..00000000 --- a/scripts/boost_performance/ECL/iFlowformer.sh +++ /dev/null @@ -1,72 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -model_name=iFlowformer -# model_name=Flowformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 diff --git a/scripts/boost_performance/ECL/iInformer.sh b/scripts/boost_performance/ECL/iInformer.sh deleted file mode 100644 index 69a6f889..00000000 --- a/scripts/boost_performance/ECL/iInformer.sh +++ /dev/null @@ -1,72 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -model_name=iInformer -# model_name=Informer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 diff --git a/scripts/boost_performance/ECL/iReformer.sh b/scripts/boost_performance/ECL/iReformer.sh deleted file mode 100644 index d6084ff8..00000000 --- a/scripts/boost_performance/ECL/iReformer.sh +++ /dev/null @@ -1,72 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -model_name=iReformer -# model_name=Reformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 diff --git a/scripts/boost_performance/ECL/iTransformer.sh b/scripts/boost_performance/ECL/iTransformer.sh deleted file mode 100644 index 175fba18..00000000 --- a/scripts/boost_performance/ECL/iTransformer.sh +++ /dev/null @@ -1,72 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -model_name=iTransformer -# model_name=Transformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 diff --git a/scripts/boost_performance/README.md b/scripts/boost_performance/README.md deleted file mode 100644 index 86974bbb..00000000 --- a/scripts/boost_performance/README.md +++ /dev/null @@ -1,29 +0,0 @@ -# Inverted Transformers Work Better for Time Series Forecasting - -This folder contains the comparison of the vanilla Transformer-based forecasters and the inverted versions. If you are new to this repo, we recommend you to have a look at this [README](../multivariate_forecasting/README.md) first. - -## Scripts - -In each folder named after the dataset, we compare the performance of iTransformers and the vanilla Transformers. - -``` -# iTransformer on the Traffic Dataset with gradually enlarged lookback windows. - -bash ./scripts/boost_performance/Traffic/iTransformer.sh -``` - -You can change the ```model_name``` in the script to select one Transformer variant and its inverted version. - -## Results -We compare the performance of Transformer and iTransformer on all six datasets, indicating that the attention and feed-forward network on the -inverted dimensions greatly empower Transformers in multivariate time series forecasting. - -

- -

- -We apply the proposed inverted framework to Transformer and its variants. It demonstrates that our iTransformers framework can consistently promote these Transformer variants, -and take advantage of the booming efficient attention mechanisms. -

- -

diff --git a/scripts/boost_performance/Traffic/iFlowformer.sh b/scripts/boost_performance/Traffic/iFlowformer.sh deleted file mode 100644 index 097a7eb0..00000000 --- a/scripts/boost_performance/Traffic/iFlowformer.sh +++ /dev/null @@ -1,76 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -model_name=iFlowformer -# model_name=Flowformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 diff --git a/scripts/boost_performance/Traffic/iInformer.sh b/scripts/boost_performance/Traffic/iInformer.sh deleted file mode 100644 index 4ba91a8c..00000000 --- a/scripts/boost_performance/Traffic/iInformer.sh +++ /dev/null @@ -1,76 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -# model_name=iInformer -model_name=Informer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 diff --git a/scripts/boost_performance/Traffic/iReformer.sh b/scripts/boost_performance/Traffic/iReformer.sh deleted file mode 100644 index bc616be3..00000000 --- a/scripts/boost_performance/Traffic/iReformer.sh +++ /dev/null @@ -1,76 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -model_name=iReformer -# model_name=Reformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 diff --git a/scripts/boost_performance/Traffic/iTransformer.sh b/scripts/boost_performance/Traffic/iTransformer.sh deleted file mode 100644 index 852b17fe..00000000 --- a/scripts/boost_performance/Traffic/iTransformer.sh +++ /dev/null @@ -1,76 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -model_name=iTransformer -#model_name=Transformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 diff --git a/scripts/boost_performance/Weather/iFlowformer.sh b/scripts/boost_performance/Weather/iFlowformer.sh deleted file mode 100644 index 4b245270..00000000 --- a/scripts/boost_performance/Weather/iFlowformer.sh +++ /dev/null @@ -1,77 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=iFlowformer -# model_name=Flowformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --itr 1 \ - --batch_size 128 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 diff --git a/scripts/boost_performance/Weather/iInformer.sh b/scripts/boost_performance/Weather/iInformer.sh deleted file mode 100644 index 5158cfd0..00000000 --- a/scripts/boost_performance/Weather/iInformer.sh +++ /dev/null @@ -1,77 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=iInformer -#model_name=Informer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --itr 1 \ - --batch_size 128 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 diff --git a/scripts/boost_performance/Weather/iReformer.sh b/scripts/boost_performance/Weather/iReformer.sh deleted file mode 100644 index 2ecdaab7..00000000 --- a/scripts/boost_performance/Weather/iReformer.sh +++ /dev/null @@ -1,77 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=iReformer -#model_name=Reformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --itr 1 \ - --batch_size 128 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 diff --git a/scripts/boost_performance/Weather/iTransformer.sh b/scripts/boost_performance/Weather/iTransformer.sh deleted file mode 100644 index 5b64ae8a..00000000 --- a/scripts/boost_performance/Weather/iTransformer.sh +++ /dev/null @@ -1,77 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=iTransformer -#model_name=Transformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --itr 1 \ - --batch_size 128 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 diff --git a/scripts/increasing_lookback/ECL/iFlowformer.sh b/scripts/increasing_lookback/ECL/iFlowformer.sh deleted file mode 100644 index 358a3368..00000000 --- a/scripts/increasing_lookback/ECL/iFlowformer.sh +++ /dev/null @@ -1,109 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -# model_name=Flowformer -model_name=iFlowformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_48_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 48 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005\ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005\ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_192_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 192 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_336_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 336 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_720_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 720 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 diff --git a/scripts/increasing_lookback/ECL/iInformer.sh b/scripts/increasing_lookback/ECL/iInformer.sh deleted file mode 100644 index 20a834bd..00000000 --- a/scripts/increasing_lookback/ECL/iInformer.sh +++ /dev/null @@ -1,109 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -# model_name=Informer -model_name=iInformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_48_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 48 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_192_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 192 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_336_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 336 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_720_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 720 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 diff --git a/scripts/increasing_lookback/ECL/iReformer.sh b/scripts/increasing_lookback/ECL/iReformer.sh deleted file mode 100644 index d94357d1..00000000 --- a/scripts/increasing_lookback/ECL/iReformer.sh +++ /dev/null @@ -1,109 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -# model_name=Reformer -model_name=iReformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_48_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 48 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_192_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 192 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_336_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 336 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_720_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 720 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 diff --git a/scripts/increasing_lookback/ECL/iTransformer.sh b/scripts/increasing_lookback/ECL/iTransformer.sh deleted file mode 100644 index bd856612..00000000 --- a/scripts/increasing_lookback/ECL/iTransformer.sh +++ /dev/null @@ -1,109 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -# model_name=Transformer -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_48_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 48 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_192_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 192 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_336_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 336 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_720_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 720 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 diff --git a/scripts/increasing_lookback/README.md b/scripts/increasing_lookback/README.md deleted file mode 100644 index c98c7813..00000000 --- a/scripts/increasing_lookback/README.md +++ /dev/null @@ -1,23 +0,0 @@ -# iTransformer for Enlarged Lookback Window - -This folder contains the implementation of the iTransformer for an enlarged lookback window. If you are new to this repo, we recommend you to read this [README](../multivariate_forecasting/README.md) first. - -## Scripts - -In each folder named after the dataset, we provide the iTransformers and the vanilla Transformers experiments under five increasing prediction lengths. - -``` -# iTransformer on the Traffic Dataset with gradually enlarged lookback windows. - -bash ./scripts/increasing_lookback/Traffic/iTransformer.sh -``` - -You can change the ```model_name``` in the script to switch the selection of the vanilla Transformer and the inverted version. - -## Results - -

- -

- -The inverted framework empowers Transformers with improved performance on the enlarged lookback window. \ No newline at end of file diff --git a/scripts/increasing_lookback/Traffic/iFlowformer.sh b/scripts/increasing_lookback/Traffic/iFlowformer.sh deleted file mode 100644 index b82ad89e..00000000 --- a/scripts/increasing_lookback/Traffic/iFlowformer.sh +++ /dev/null @@ -1,109 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -# model_name=Flowformer -model_name=iFlowformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_48_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 48 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_192_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 192 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_336_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 336 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_720_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 720 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 diff --git a/scripts/increasing_lookback/Traffic/iInformer.sh b/scripts/increasing_lookback/Traffic/iInformer.sh deleted file mode 100644 index eb06e41e..00000000 --- a/scripts/increasing_lookback/Traffic/iInformer.sh +++ /dev/null @@ -1,109 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -# model_name=Informer -model_name=iInformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_48_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 48 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_192_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 192 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_336_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 336 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_720_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 720 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 diff --git a/scripts/increasing_lookback/Traffic/iReformer.sh b/scripts/increasing_lookback/Traffic/iReformer.sh deleted file mode 100644 index 07e634fc..00000000 --- a/scripts/increasing_lookback/Traffic/iReformer.sh +++ /dev/null @@ -1,109 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -# model_name=Reformer -model_name=iReformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_48_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 48 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_192_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 192 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_336_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 336 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_720_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 720 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 \ No newline at end of file diff --git a/scripts/increasing_lookback/Traffic/iTransformer.sh b/scripts/increasing_lookback/Traffic/iTransformer.sh deleted file mode 100644 index 7ee2d8b6..00000000 --- a/scripts/increasing_lookback/Traffic/iTransformer.sh +++ /dev/null @@ -1,109 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -# model_name=Transformer -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_48_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 48 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --factor 3 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_192_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 192 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_336_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 336 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_720_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 720 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 diff --git a/scripts/model_efficiency/ECL/iFlashTransformer.sh b/scripts/model_efficiency/ECL/iFlashTransformer.sh deleted file mode 100644 index df981cbf..00000000 --- a/scripts/model_efficiency/ECL/iFlashTransformer.sh +++ /dev/null @@ -1,165 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -model_name=Flashformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 192 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 336 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 720 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -model_name=iFlashformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 192 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 336 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 720 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --itr 1 diff --git a/scripts/model_efficiency/README.md b/scripts/model_efficiency/README.md deleted file mode 100644 index 3d21877d..00000000 --- a/scripts/model_efficiency/README.md +++ /dev/null @@ -1,40 +0,0 @@ -# Efficiency Improvement of iTransformer - -Supposing the input multivariate time series has a shape of $T \times N$. The vanilla attention module has a complexity of $\mathcal{O}(L^2)$, where $L$ is the number of tokens. - -* In Transformer, we have $L=T$ because of the manner of time points as tokens. -* In iTransformer, we have $L=N$ because of the manner of variates as tokens. - -## Benefit from Efficient Attention - -Since the attention mechanism is applied on the variate dimension in the inverted structure, efficient attention with reduced complexity essentially addresses the problem of numerous variates, which is ubiquitous in real-world applications. - -We currently try out the linear complexity attention from [Flowformer](https://github.com/thuml/Flowformer), and the hardware-accelerated attention mechanism from [FlashAttention](https://github.com/shreyansh26/FlashAttention-PyTorch). It demonstrates efficiency improvement by adopting these novel attention mechanisms. - -### Scripts -We provide the iTransformers with the FlashAttention module: - -``` -# iTransformer on the Traffic Dataset with hardware-friendly FlashAttention for acceleration - -bash ./scripts/model_efficiency/Traffic/iFlashTransformer.sh -``` - - -## Efficient Training Strategy -With the input flexibility of attention, the token number can vary from training to inference, **our model is the first one to be capable of training on arbitrary numbers of series**. We propose a novel training strategy for high-dimensional multivariate series by taking advantage of the [variate generation capability](../variate_generalization/README.md). - -Concretely, we randomly choose part of the variates in each batch and only train the model with selected variates. Since the number of variate channels is flexible because of our inverting, the model can predict all the variates for predictions. - - -## Results - -**Environments**: The batch size of training is fixed as 16 with comparable model hyperparameters. The experiments run on P100 (16G). We comprehensively compare the training speed, memory footprint, and performance of the following. - -

- -

- -* The efficiency of iTransformer exceeds other Transformers in Weather with 21 variates. In Traffic with 862 variates, the memory footprints are basically the same, but iTransformer can be trained faster. -* iTransformer achieves particularly better performance on the dataset with numerous variates, since the multivariate correlations can be explicitly utilized. -* By adopting an efficient attention module or our proposed efficient training strategy on partial variates, iTransformer can enjoy the same level of speed and memory footprint as linear forecasters. \ No newline at end of file diff --git a/scripts/model_efficiency/Traffic/iFlashTransformer.sh b/scripts/model_efficiency/Traffic/iFlashTransformer.sh deleted file mode 100644 index 9c1ade70..00000000 --- a/scripts/model_efficiency/Traffic/iFlashTransformer.sh +++ /dev/null @@ -1,173 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=Flashformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 192 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 336 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 720 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -model_name=iFlashformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 192 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 336 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 720 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --itr 1 \ - --train_epochs 3 diff --git a/scripts/model_efficiency/Weather/iFlashTransformer.sh b/scripts/model_efficiency/Weather/iFlashTransformer.sh deleted file mode 100644 index 8e65743c..00000000 --- a/scripts/model_efficiency/Weather/iFlashTransformer.sh +++ /dev/null @@ -1,175 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -model_name=Flashformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --itr 1 \ - --batch_size 128 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 192 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 336 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 720 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -model_name=iFlashformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 \ - --train_epochs 3 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 192 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 336 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 720 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --batch_size 128 \ - --itr 1 \ No newline at end of file diff --git a/scripts/multivariate_forecasting/ECL/iTransformer.sh b/scripts/multivariate_forecasting/ECL/iTransformer.sh deleted file mode 100644 index f8609f09..00000000 --- a/scripts/multivariate_forecasting/ECL/iTransformer.sh +++ /dev/null @@ -1,89 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 - - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 \ No newline at end of file diff --git a/scripts/multivariate_forecasting/ETT/iTransformer_ETTh1.sh b/scripts/multivariate_forecasting/ETT/iTransformer_ETTh1.sh deleted file mode 100644 index e72e27e6..00000000 --- a/scripts/multivariate_forecasting/ETT/iTransformer_ETTh1.sh +++ /dev/null @@ -1,79 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTh1.csv \ - --model_id ETTh1_96_96 \ - --model $model_name \ - --data ETTh1 \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 256 \ - --d_ff 256 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTh1.csv \ - --model_id ETTh1_96_192 \ - --model $model_name \ - --data ETTh1 \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 256 \ - --d_ff 256 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTh1.csv \ - --model_id ETTh1_96_336 \ - --model $model_name \ - --data ETTh1 \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTh1.csv \ - --model_id ETTh1_96_720 \ - --model $model_name \ - --data ETTh1 \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --itr 1 \ No newline at end of file diff --git a/scripts/multivariate_forecasting/ETT/iTransformer_ETTh2.sh b/scripts/multivariate_forecasting/ETT/iTransformer_ETTh2.sh deleted file mode 100644 index 5da4439f..00000000 --- a/scripts/multivariate_forecasting/ETT/iTransformer_ETTh2.sh +++ /dev/null @@ -1,79 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTh2.csv \ - --model_id ETTh2_96_96 \ - --model $model_name \ - --data ETTh2 \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTh2.csv \ - --model_id ETTh2_96_192 \ - --model $model_name \ - --data ETTh2 \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTh2.csv \ - --model_id ETTh2_96_336 \ - --model $model_name \ - --data ETTh2 \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTh2.csv \ - --model_id ETTh2_96_720 \ - --model $model_name \ - --data ETTh2 \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 \ No newline at end of file diff --git a/scripts/multivariate_forecasting/ETT/iTransformer_ETTm1.sh b/scripts/multivariate_forecasting/ETT/iTransformer_ETTm1.sh deleted file mode 100644 index 64d83daa..00000000 --- a/scripts/multivariate_forecasting/ETT/iTransformer_ETTm1.sh +++ /dev/null @@ -1,79 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTm1.csv \ - --model_id ETTm1_96_96 \ - --model $model_name \ - --data ETTm1 \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTm1.csv \ - --model_id ETTm1_96_192 \ - --model $model_name \ - --data ETTm1 \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTm1.csv \ - --model_id ETTm1_96_336 \ - --model $model_name \ - --data ETTm1 \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTm1.csv \ - --model_id ETTm1_96_720 \ - --model $model_name \ - --data ETTm1 \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 \ No newline at end of file diff --git a/scripts/multivariate_forecasting/ETT/iTransformer_ETTm2.sh b/scripts/multivariate_forecasting/ETT/iTransformer_ETTm2.sh deleted file mode 100644 index e5de2f36..00000000 --- a/scripts/multivariate_forecasting/ETT/iTransformer_ETTm2.sh +++ /dev/null @@ -1,79 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTm2.csv \ - --model_id ETTm2_96_96 \ - --model $model_name \ - --data ETTm2 \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTm2.csv \ - --model_id ETTm2_96_192 \ - --model $model_name \ - --data ETTm2 \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTm2.csv \ - --model_id ETTm2_96_336 \ - --model $model_name \ - --data ETTm2 \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/ETT-small/ \ - --data_path ETTm2.csv \ - --model_id ETTm2_96_720 \ - --model $model_name \ - --data ETTm2 \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 7 \ - --dec_in 7 \ - --c_out 7 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 \ No newline at end of file diff --git a/scripts/multivariate_forecasting/Exchange/iTransformer.sh b/scripts/multivariate_forecasting/Exchange/iTransformer.sh deleted file mode 100644 index e2729fb8..00000000 --- a/scripts/multivariate_forecasting/Exchange/iTransformer.sh +++ /dev/null @@ -1,80 +0,0 @@ -export CUDA_VISIBLE_DEVICES=3 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/exchange_rate/ \ - --data_path exchange_rate.csv \ - --model_id Exchange_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 8 \ - --dec_in 8 \ - --c_out 8 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/exchange_rate/ \ - --data_path exchange_rate.csv \ - --model_id Exchange_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 8 \ - --dec_in 8 \ - --c_out 8 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/exchange_rate/ \ - --data_path exchange_rate.csv \ - --model_id Exchange_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 8 \ - --dec_in 8 \ - --c_out 8 \ - --des 'Exp' \ - --itr 1 \ - --d_model 128 \ - --d_ff 128 \ - --train_epochs 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/exchange_rate/ \ - --data_path exchange_rate.csv \ - --model_id Exchange_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 8 \ - --dec_in 8 \ - --c_out 8 \ - --des 'Exp' \ - --d_model 128 \ - --d_ff 128 \ - --itr 1 \ No newline at end of file diff --git a/scripts/multivariate_forecasting/PEMS/iTransformer_03.sh b/scripts/multivariate_forecasting/PEMS/iTransformer_03.sh deleted file mode 100644 index a218a1cf..00000000 --- a/scripts/multivariate_forecasting/PEMS/iTransformer_03.sh +++ /dev/null @@ -1,85 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS03.npz \ - --model_id PEMS03_96_12 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 12 \ - --e_layers 4 \ - --enc_in 358 \ - --dec_in 358 \ - --c_out 358 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS03.npz \ - --model_id PEMS03_96_24 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 24 \ - --e_layers 4 \ - --enc_in 358 \ - --dec_in 358 \ - --c_out 358 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --learning_rate 0.001 \ - --itr 1 - - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS03.npz \ - --model_id PEMS03_96_48 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 48 \ - --e_layers 4 \ - --enc_in 358 \ - --dec_in 358 \ - --c_out 358 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --learning_rate 0.001 \ - --itr 1 - - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS03.npz \ - --model_id PEMS03_96_96 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 358 \ - --dec_in 358 \ - --c_out 358 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --learning_rate 0.001 \ - --itr 1 \ No newline at end of file diff --git a/scripts/multivariate_forecasting/PEMS/iTransformer_04.sh b/scripts/multivariate_forecasting/PEMS/iTransformer_04.sh deleted file mode 100644 index be9851af..00000000 --- a/scripts/multivariate_forecasting/PEMS/iTransformer_04.sh +++ /dev/null @@ -1,87 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS04.npz \ - --model_id PEMS04_96_12 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 12 \ - --e_layers 4 \ - --enc_in 307 \ - --dec_in 307 \ - --c_out 307 \ - --des 'Exp' \ - --d_model 1024 \ - --d_ff 1024 \ - --learning_rate 0.0005 \ - --itr 1 \ - --use_norm 0 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS04.npz \ - --model_id PEMS04_96_24 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 24 \ - --e_layers 4 \ - --enc_in 307 \ - --dec_in 307 \ - --c_out 307 \ - --des 'Exp' \ - --d_model 1024 \ - --d_ff 1024 \ - --learning_rate 0.0005 \ - --itr 1 \ - --use_norm 0 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS04.npz \ - --model_id PEMS04_96_48 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 48 \ - --e_layers 4 \ - --enc_in 307 \ - --dec_in 307 \ - --c_out 307 \ - --des 'Exp' \ - --d_model 1024 \ - --d_ff 1024 \ - --learning_rate 0.0005 \ - --itr 1 \ - --use_norm 0 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS04.npz \ - --model_id PEMS04_96_96 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 307 \ - --dec_in 307 \ - --c_out 307 \ - --des 'Exp' \ - --d_model 1024 \ - --d_ff 1024 \ - --learning_rate 0.0005 \ - --itr 1 \ - --use_norm 0 diff --git a/scripts/multivariate_forecasting/PEMS/iTransformer_07.sh b/scripts/multivariate_forecasting/PEMS/iTransformer_07.sh deleted file mode 100644 index 352f3a3b..00000000 --- a/scripts/multivariate_forecasting/PEMS/iTransformer_07.sh +++ /dev/null @@ -1,89 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS07.npz \ - --model_id PEMS07_96_12 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 12 \ - --e_layers 2 \ - --enc_in 883 \ - --dec_in 883 \ - --c_out 883 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --learning_rate 0.001 \ - --itr 1 \ - --use_norm 0 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS07.npz \ - --model_id PEMS07_96_24 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 24 \ - --e_layers 2 \ - --enc_in 883 \ - --dec_in 883 \ - --c_out 883 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --learning_rate 0.001 \ - --itr 1 \ - --use_norm 0 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS07.npz \ - --model_id PEMS07_96_48 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 48 \ - --e_layers 4 \ - --enc_in 883 \ - --dec_in 883 \ - --c_out 883 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16\ - --learning_rate 0.001 \ - --itr 1 \ - --use_norm 0 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS07.npz \ - --model_id PEMS07_96_96 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 883 \ - --dec_in 883 \ - --c_out 883 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16\ - --learning_rate 0.001 \ - --itr 1 \ - --use_norm 0 diff --git a/scripts/multivariate_forecasting/PEMS/iTransformer_08.sh b/scripts/multivariate_forecasting/PEMS/iTransformer_08.sh deleted file mode 100644 index 99010775..00000000 --- a/scripts/multivariate_forecasting/PEMS/iTransformer_08.sh +++ /dev/null @@ -1,87 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS08.npz \ - --model_id PEMS08_96_12 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 12 \ - --e_layers 2 \ - --enc_in 170 \ - --dec_in 170 \ - --c_out 170 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --itr 1 \ - --use_norm 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS08.npz \ - --model_id PEMS08_96_24 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 24 \ - --e_layers 2 \ - --enc_in 170 \ - --dec_in 170 \ - --c_out 170 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --itr 1 \ - --use_norm 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS08.npz \ - --model_id PEMS08_96_48 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 48 \ - --e_layers 4 \ - --enc_in 170 \ - --dec_in 170 \ - --c_out 170 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16\ - --learning_rate 0.001 \ - --itr 1 \ - --use_norm 0 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/PEMS/ \ - --data_path PEMS08.npz \ - --model_id PEMS08_96_96 \ - --model $model_name \ - --data PEMS \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 170 \ - --dec_in 170 \ - --c_out 170 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16\ - --learning_rate 0.001 \ - --itr 1 \ - --use_norm 0 diff --git a/scripts/multivariate_forecasting/README.md b/scripts/multivariate_forecasting/README.md deleted file mode 100644 index 92d08795..00000000 --- a/scripts/multivariate_forecasting/README.md +++ /dev/null @@ -1,57 +0,0 @@ -# iTransformer for Multivariate Time Series Forecasting - -This folder contains the reproductions of the iTransformers for Multivariate Time Series Forecasting (MTSF). - -## Dataset - -Extensive challenging multivariate forecasting tasks are evaluated as the benchmark. We provide the download links: [Google Drive](https://drive.google.com/file/d/1l51QsKvQPcqILT3DwfjCgx8Dsg2rpjot/view?usp=drive_link) or [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/2ea5ca3d621e4e5ba36a/). - -

- -

- -## Scripts - -In each folder named after the dataset, we provide the iTransformer experiments under four different prediction lengths as shown in the table above. - -``` -# iTransformer on the Traffic Dataset - -bash ./scripts/multivariate_forecasting/Traffic/iTransformer.sh -``` - -To evaluate the model under other input/prediction lengths, feel free to change the ```seq_len``` and ```pred_len``` arguments: - -``` -# iTransformer on the Electricity Dataset, where 180 time steps are inputted as the observations, and the task is to predict the future 60 steps - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_180_60 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 180 \ - --pred_len 60 \ - --e_layers 3 \ - --enc_in 321 \ - --dec_in 321 \ - --c_out 321 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.0005 \ - --itr 1 -``` - - -## Training on Custom Dataset - -To train with your own time series dataset, you can try out the following steps: - -1. Read through the ```Dataset_Custom``` class under the ```data_provider/data_loader``` folder, which provides the functionality to load and process time series files. -2. The file should be ```csv``` format with the first column containing the timestamp and the following columns containing the variates of time series. -3. Set ```data=custom``` and modify the ```enc_in```, ```dec_in```, ```c_out``` arguments according to your number of variates in the training script. diff --git a/scripts/multivariate_forecasting/SolarEnergy/iTransformer.sh b/scripts/multivariate_forecasting/SolarEnergy/iTransformer.sh deleted file mode 100644 index 606c1b92..00000000 --- a/scripts/multivariate_forecasting/SolarEnergy/iTransformer.sh +++ /dev/null @@ -1,83 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_96 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 2 \ - --enc_in 137 \ - --dec_in 137 \ - --c_out 137 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_192 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 2 \ - --enc_in 137 \ - --dec_in 137 \ - --c_out 137 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_336 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 2 \ - --enc_in 137 \ - --dec_in 137 \ - --c_out 137 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --learning_rate 0.0005 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_720 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 2 \ - --enc_in 137 \ - --dec_in 137 \ - --c_out 137 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --learning_rate 0.0005 \ - --itr 1 diff --git a/scripts/multivariate_forecasting/Traffic/iTransformer.sh b/scripts/multivariate_forecasting/Traffic/iTransformer.sh deleted file mode 100644 index 07387742..00000000 --- a/scripts/multivariate_forecasting/Traffic/iTransformer.sh +++ /dev/null @@ -1,87 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512\ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512\ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001 \ - --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 4 \ - --enc_in 862 \ - --dec_in 862 \ - --c_out 862 \ - --des 'Exp' \ - --d_model 512 \ - --d_ff 512 \ - --batch_size 16 \ - --learning_rate 0.001\ - --itr 1 \ No newline at end of file diff --git a/scripts/multivariate_forecasting/Weather/iTransformer.sh b/scripts/multivariate_forecasting/Weather/iTransformer.sh deleted file mode 100644 index 1499e34b..00000000 --- a/scripts/multivariate_forecasting/Weather/iTransformer.sh +++ /dev/null @@ -1,82 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -model_name=iTransformer - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 96 \ - --e_layers 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --d_model 512\ - --d_ff 512\ - --itr 1 - - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_192 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 192 \ - --e_layers 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --d_model 512\ - --d_ff 512\ - --itr 1 - - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_336 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 336 \ - --e_layers 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --d_model 512\ - --d_ff 512\ - --itr 1 - - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/weather/ \ - --data_path weather.csv \ - --model_id weather_96_720 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --pred_len 720 \ - --e_layers 3 \ - --enc_in 21 \ - --dec_in 21 \ - --c_out 21 \ - --des 'Exp' \ - --d_model 512\ - --d_ff 512\ - --itr 1 \ No newline at end of file diff --git a/scripts/variate_generalization/ECL/iFlowformer.sh b/scripts/variate_generalization/ECL/iFlowformer.sh deleted file mode 100644 index bd3aad57..00000000 --- a/scripts/variate_generalization/ECL/iFlowformer.sh +++ /dev/null @@ -1,91 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -model_name=Flowformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/electricity/ \ -# --data_path electricity.csv \ -# --model_id ECL_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 321 \ -# --dec_in 321 \ -# --c_out 321 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 64 \ - --dec_in 64 \ - --c_out 64 \ - --des 'Exp' \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 8 \ - --d_model 32 \ - --d_ff 64 \ - --itr 1 - -model_name=iFlowformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/electricity/ \ -# --data_path electricity.csv \ -# --model_id ECL_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 321 \ -# --dec_in 321 \ -# --c_out 321 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 64 \ - --dec_in 64 \ - --c_out 64 \ - --des 'Exp' \ - --exp_name partial_train \ - --itr 1 diff --git a/scripts/variate_generalization/ECL/iInformer.sh b/scripts/variate_generalization/ECL/iInformer.sh deleted file mode 100644 index b22046da..00000000 --- a/scripts/variate_generalization/ECL/iInformer.sh +++ /dev/null @@ -1,91 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -model_name=Informer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/electricity/ \ -# --data_path electricity.csv \ -# --model_id ECL_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 321 \ -# --dec_in 321 \ -# --c_out 321 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 64 \ - --dec_in 64 \ - --c_out 64 \ - --des 'Exp' \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 8 \ - --d_model 32 \ - --d_ff 64 \ - --itr 1 - -model_name=iInformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/electricity/ \ -# --data_path electricity.csv \ -# --model_id ECL_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 321 \ -# --dec_in 321 \ -# --c_out 321 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 64 \ - --dec_in 64 \ - --c_out 64 \ - --des 'Exp' \ - --exp_name partial_train \ - --itr 1 \ No newline at end of file diff --git a/scripts/variate_generalization/ECL/iReformer.sh b/scripts/variate_generalization/ECL/iReformer.sh deleted file mode 100644 index 89bd3186..00000000 --- a/scripts/variate_generalization/ECL/iReformer.sh +++ /dev/null @@ -1,91 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=Reformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/electricity/ \ -# --data_path electricity.csv \ -# --model_id ECL_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 321 \ -# --dec_in 321 \ -# --c_out 321 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 64 \ - --dec_in 64 \ - --c_out 64 \ - --des 'Exp' \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 8 \ - --d_model 32 \ - --d_ff 64 \ - --itr 1 - -model_name=iReformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/electricity/ \ -# --data_path electricity.csv \ -# --model_id ECL_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 321 \ -# --dec_in 321 \ -# --c_out 321 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 64 \ - --dec_in 64 \ - --c_out 64 \ - --des 'Exp' \ - --exp_name partial_train \ - --itr 1 \ No newline at end of file diff --git a/scripts/variate_generalization/ECL/iTransformer.sh b/scripts/variate_generalization/ECL/iTransformer.sh deleted file mode 100644 index 47e15138..00000000 --- a/scripts/variate_generalization/ECL/iTransformer.sh +++ /dev/null @@ -1,92 +0,0 @@ -export CUDA_VISIBLE_DEVICES=3 - -model_name=Transformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/electricity/ \ -# --data_path electricity.csv \ -# --model_id ECL_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 321 \ -# --dec_in 321 \ -# --c_out 321 \ -# --des 'Exp' \ -# --itr 1 - -# 20% partial variates, enc_in: 64 = 321 // 5 -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 64 \ - --dec_in 64 \ - --c_out 64 \ - --des 'Exp' \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 8 \ - --d_model 32 \ - --d_ff 64 \ - --itr 1 - -model_name=iTransformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/electricity/ \ -# --data_path electricity.csv \ -# --model_id ECL_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 321 \ -# --dec_in 321 \ -# --c_out 321 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/electricity/ \ - --data_path electricity.csv \ - --model_id ECL_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 64 \ - --dec_in 64 \ - --c_out 64 \ - --des 'Exp' \ - --exp_name partial_train \ - --itr 1 \ No newline at end of file diff --git a/scripts/variate_generalization/README.md b/scripts/variate_generalization/README.md deleted file mode 100644 index 28d113bd..00000000 --- a/scripts/variate_generalization/README.md +++ /dev/null @@ -1,37 +0,0 @@ -# iTransformer for Variate Generalization - -This folder contains the implementation of the iTransformer to generalize on unseen variates. If you are new to this repo, we recommend you to read this [README](../multivariate_forecasting/README.md) first. - -By inverting vanilla Transformers, the model is empowered with the generalization capability on unseen variates. Firstly, benefiting from the flexibility of the number of input tokens, the amount of variate channels is no longer restricted and thus feasible to vary from training and inference. Second, feed-forward networks are identically applied on independent variate tokens to learn transferable representations of time series. - -## Scripts - -``` -# Train models with only 20% of variates from Traffic and test the model on all variates without finetuning - -bash ./scripts/variate_generalization/Traffic/iTransformer.sh -``` - -> During Training -

- -

- -> During Inference -

- -

- -In each folder named after the dataset, we provide two strategies to enable Transformers to generalize on unseen variate. - -* **CI-Transformers**: Channel Independence regards each variate of time series as independent channels, and uses a shared backbone to forecast all variates. Therefore, the model can predict variates one by one, but the training and inference procedure can be time-consuming. - -* **iTransformers**: benefiting from the flexibility of attention that the number of input tokens can be dynamically changeable, the amount of variates as tokens is no longer restricted, and can even allow the model to be trained on arbitrary variables. - -## Results - -

- -

- - iTransformers can be naturally trained with 20% variates and accomplish forecast on all variates with the ability to learn transferable representations. \ No newline at end of file diff --git a/scripts/variate_generalization/SolarEnergy/iFlowformer.sh b/scripts/variate_generalization/SolarEnergy/iFlowformer.sh deleted file mode 100644 index 503e0d44..00000000 --- a/scripts/variate_generalization/SolarEnergy/iFlowformer.sh +++ /dev/null @@ -1,95 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -model_name=Flowformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/Solar/ \ -# --data_path solar_AL.txt \ -# --model_id solar_96_96 \ -# --model $model_name \ -# --data Solar \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 137 \ -# --dec_in 137 \ -# --c_out 137 \ -# --des 'Exp' \ -# --learning_rate 0.0005 \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_96 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 27 \ - --dec_in 27 \ - --c_out 27 \ - --des 'Exp' \ - --d_model 32 \ - --d_ff 64 \ - --learning_rate 0.0005 \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 8 \ - --itr 1 - -model_name=iFlowformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/Solar/ \ -# --data_path solar_AL.txt \ -# --model_id solar_96_96 \ -# --model $model_name \ -# --data Solar \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 137 \ -# --dec_in 137 \ -# --c_out 137 \ -# --des 'Exp' \ -# --learning_rate 0.0005 \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_96 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 27 \ - --dec_in 27 \ - --c_out 27 \ - --des 'Exp' \ - --learning_rate 0.0005 \ - --exp_name partial_train \ - --itr 1 \ No newline at end of file diff --git a/scripts/variate_generalization/SolarEnergy/iInformer.sh b/scripts/variate_generalization/SolarEnergy/iInformer.sh deleted file mode 100644 index 86dfc192..00000000 --- a/scripts/variate_generalization/SolarEnergy/iInformer.sh +++ /dev/null @@ -1,95 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -model_name=Informer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/Solar/ \ -# --data_path solar_AL.txt \ -# --model_id solar_96_96 \ -# --model $model_name \ -# --data Solar \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 137 \ -# --dec_in 137 \ -# --c_out 137 \ -# --des 'Exp' \ -# --learning_rate 0.0005 \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_96 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 27 \ - --dec_in 27 \ - --c_out 27 \ - --des 'Exp' \ - --d_model 32 \ - --d_ff 64 \ - --learning_rate 0.0005 \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 8 \ - --itr 1 - -model_name=iInformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/Solar/ \ -# --data_path solar_AL.txt \ -# --model_id solar_96_96 \ -# --model $model_name \ -# --data Solar \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 137 \ -# --dec_in 137 \ -# --c_out 137 \ -# --des 'Exp' \ -# --learning_rate 0.0005 \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_96 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 27 \ - --dec_in 27 \ - --c_out 27 \ - --des 'Exp' \ - --learning_rate 0.0005 \ - --exp_name partial_train \ - --itr 1 \ No newline at end of file diff --git a/scripts/variate_generalization/SolarEnergy/iReformer.sh b/scripts/variate_generalization/SolarEnergy/iReformer.sh deleted file mode 100644 index ce1f4b35..00000000 --- a/scripts/variate_generalization/SolarEnergy/iReformer.sh +++ /dev/null @@ -1,95 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=Reformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/Solar/ \ -# --data_path solar_AL.txt \ -# --model_id solar_96_96 \ -# --model $model_name \ -# --data Solar \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 137 \ -# --dec_in 137 \ -# --c_out 137 \ -# --des 'Exp' \ -# --learning_rate 0.0005 \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_96 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 27 \ - --dec_in 27 \ - --c_out 27 \ - --des 'Exp' \ - --d_model 32 \ - --d_ff 64 \ - --learning_rate 0.0005 \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 8 \ - --itr 1 - -model_name=iReformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/Solar/ \ -# --data_path solar_AL.txt \ -# --model_id solar_96_96 \ -# --model $model_name \ -# --data Solar \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 137 \ -# --dec_in 137 \ -# --c_out 137 \ -# --des 'Exp' \ -# --learning_rate 0.0005 \ -# --itr - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_96 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 27 \ - --dec_in 27 \ - --c_out 27 \ - --des 'Exp' \ - --learning_rate 0.0005 \ - --exp_name partial_train \ - --itr 1 \ No newline at end of file diff --git a/scripts/variate_generalization/SolarEnergy/iTransformer.sh b/scripts/variate_generalization/SolarEnergy/iTransformer.sh deleted file mode 100644 index 99084c40..00000000 --- a/scripts/variate_generalization/SolarEnergy/iTransformer.sh +++ /dev/null @@ -1,96 +0,0 @@ -export CUDA_VISIBLE_DEVICES=3 - -model_name=Transformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/Solar/ \ -# --data_path solar_AL.txt \ -# --model_id solar_96_96 \ -# --model $model_name \ -# --data Solar \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 137 \ -# --dec_in 137 \ -# --c_out 137 \ -# --des 'Exp' \ -# --learning_rate 0.0005 \ -# --itr 1 - -# 20% partial variates: 27 = 137 // 5 -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_96 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 27 \ - --dec_in 27 \ - --c_out 27 \ - --des 'Exp' \ - --d_model 32 \ - --d_ff 64 \ - --learning_rate 0.0005 \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 8 \ - --itr 1 - -model_name=iTransformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/Solar/ \ -# --data_path solar_AL.txt \ -# --model_id solar_96_96 \ -# --model $model_name \ -# --data Solar \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 137 \ -# --dec_in 137 \ -# --c_out 137 \ -# --des 'Exp' \ -# --learning_rate 0.0005 \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/Solar/ \ - --data_path solar_AL.txt \ - --model_id solar_96_96 \ - --model $model_name \ - --data Solar \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 27 \ - --dec_in 27 \ - --c_out 27 \ - --des 'Exp' \ - --learning_rate 0.0005 \ - --exp_name partial_train \ - --itr 1 \ No newline at end of file diff --git a/scripts/variate_generalization/Traffic/iFlowformer.sh b/scripts/variate_generalization/Traffic/iFlowformer.sh deleted file mode 100644 index 5e02cbc5..00000000 --- a/scripts/variate_generalization/Traffic/iFlowformer.sh +++ /dev/null @@ -1,91 +0,0 @@ -export CUDA_VISIBLE_DEVICES=0 - -model_name=Flowformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/traffic/ \ -# --data_path traffic.csv \ -# --model_id traffic_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 862 \ -# --dec_in 862 \ -# --c_out 862 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 172 \ - --dec_in 172 \ - --c_out 172 \ - --des 'Exp' \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 4 \ - --d_model 32 \ - --d_ff 64 \ - --itr 1 - -model_name=iFlowformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/traffic/ \ -# --data_path traffic.csv \ -# --model_id traffic_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 862 \ -# --dec_in 862 \ -# --c_out 862 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 172 \ - --dec_in 172 \ - --c_out 172 \ - --des 'Exp' \ - --exp_name partial_train \ - --itr 1 diff --git a/scripts/variate_generalization/Traffic/iInformer.sh b/scripts/variate_generalization/Traffic/iInformer.sh deleted file mode 100644 index c9b70fad..00000000 --- a/scripts/variate_generalization/Traffic/iInformer.sh +++ /dev/null @@ -1,91 +0,0 @@ -export CUDA_VISIBLE_DEVICES=1 - -model_name=Informer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/traffic/ \ -# --data_path traffic.csv \ -# --model_id traffic_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 862 \ -# --dec_in 862 \ -# --c_out 862 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 172 \ - --dec_in 172 \ - --c_out 172 \ - --des 'Exp' \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 4 \ - --d_model 32 \ - --d_ff 64 \ - --itr 1 - -model_name=iInformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/traffic/ \ -# --data_path traffic.csv \ -# --model_id traffic_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 862 \ -# --dec_in 862 \ -# --c_out 862 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 172 \ - --dec_in 172 \ - --c_out 172 \ - --des 'Exp' \ - --exp_name partial_train \ - --itr 1 diff --git a/scripts/variate_generalization/Traffic/iReformer.sh b/scripts/variate_generalization/Traffic/iReformer.sh deleted file mode 100644 index 161a9dc5..00000000 --- a/scripts/variate_generalization/Traffic/iReformer.sh +++ /dev/null @@ -1,91 +0,0 @@ -export CUDA_VISIBLE_DEVICES=2 - -model_name=Reformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/traffic/ \ -# --data_path traffic.csv \ -# --model_id traffic_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 862 \ -# --dec_in 862 \ -# --c_out 862 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 172 \ - --dec_in 172 \ - --c_out 172 \ - --des 'Exp' \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 4 \ - --d_model 32 \ - --d_ff 64 \ - --itr 1 - -model_name=iReformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/traffic/ \ -# --data_path traffic.csv \ -# --model_id traffic_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 862 \ -# --dec_in 862 \ -# --c_out 862 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 172 \ - --dec_in 172 \ - --c_out 172 \ - --des 'Exp' \ - --exp_name partial_train \ - --itr 1 \ No newline at end of file diff --git a/scripts/variate_generalization/Traffic/iTransformer.sh b/scripts/variate_generalization/Traffic/iTransformer.sh deleted file mode 100644 index 94eaccdb..00000000 --- a/scripts/variate_generalization/Traffic/iTransformer.sh +++ /dev/null @@ -1,92 +0,0 @@ -export CUDA_VISIBLE_DEVICES=3 - -model_name=Transformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/traffic/ \ -# --data_path traffic.csv \ -# --model_id traffic_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 862 \ -# --dec_in 862 \ -# --c_out 862 \ -# --des 'Exp' \ -# --itr 1 - -# 20% partial variates, enc_in: 172 = 862 // 5 -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 172 \ - --dec_in 172 \ - --c_out 172 \ - --des 'Exp' \ - --channel_independence true \ - --exp_name partial_train \ - --batch_size 8 \ - --d_model 32 \ - --d_ff 64 \ - --itr 1 - -model_name=iTransformer - -#python -u run.py \ -## --is_training 1 \ -# --root_path ./dataset/traffic/ \ -# --data_path traffic.csv \ -# --model_id traffic_96_96 \ -# --model $model_name \ -# --data custom \ -# --features M \ -# --seq_len 96 \ -# --label_len 48 \ -# --pred_len 96 \ -# --e_layers 2 \ -# --d_layers 1 \ -# --factor 3 \ -# --enc_in 862 \ -# --dec_in 862 \ -# --c_out 862 \ -# --des 'Exp' \ -# --itr 1 - -python -u run.py \ - --is_training 1 \ - --root_path ./dataset/traffic/ \ - --data_path traffic.csv \ - --model_id traffic_96_96 \ - --model $model_name \ - --data custom \ - --features M \ - --seq_len 96 \ - --label_len 48 \ - --pred_len 96 \ - --e_layers 2 \ - --d_layers 1 \ - --factor 3 \ - --enc_in 172 \ - --dec_in 172 \ - --c_out 172 \ - --des 'Exp' \ - --exp_name partial_train \ - --itr 1 From 3fdcca73474ad42dc2494f6f70a89d0985717cf1 Mon Sep 17 00:00:00 2001 From: HirokiKaneko1124 Date: Mon, 10 Mar 2025 11:03:11 +0900 Subject: [PATCH 14/14] torch and requirements --- requirements.txt | 6 ++++++ torch.txt | 4 ++++ 2 files changed, 10 insertions(+) create mode 100644 requirements.txt create mode 100644 torch.txt diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 00000000..5d3ac625 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,6 @@ +pandas==1.5.3 +scikit-learn==1.2.2 +numpy==1.23.5 +matplotlib==3.7.0 +reformer-pytorch==1.4.4 +datetime==5.5 \ No newline at end of file diff --git a/torch.txt b/torch.txt new file mode 100644 index 00000000..a6b1024d --- /dev/null +++ b/torch.txt @@ -0,0 +1,4 @@ +--index-url https://download.pytorch.org/whl/cu118 +torch +torchvision +torchaudio \ No newline at end of file