Update readme

Author: habibirani

README.md

@@ -1,10 +1,11 @@
 <!-- TITLE -->
-# Positional Encoding Benchmark for Time Series Classification
+## Positional Encoding Benchmark for Time Series Classification
+
 
-[![arXiv](https://img.shields.io/badge/arXiv-2502.12370-b31b1b.svg)](https://arxiv.org/abs/2502.12370)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)](https://www.python.org/downloads/release/python-3100/)
-[![PyTorch](https://img.shields.io/badge/PyTorch-2.4.1-ee4c2c.svg)](https://pytorch.org/)
+[![PyTorch](https://img.shields.io/badge/PyTorch-2.0+-ee4c2c?logo=pytorch&logoColor=white)](https://pytorch.org/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![arXiv](https://img.shields.io/badge/arXiv-2502.12370-b31b1b.svg)](https://arxiv.org/abs/2502.12370)
 
 This repository provides a comprehensive evaluation framework for positional encoding methods in transformer-based time series models, along with implementations and benchmarking results.
 
@@ -70,9 +71,34 @@ python examples/run_benchmark.py
 python examples/run_benchmark.py --config path/to/custom_config.yaml
 ```
 
+## Usage 
+```python
+from encodings.positional_encodings import PE_Name
+from models.transformer import TimeSeriesTransformer
+
+# Use in transformer
+model = TimeSeriesTransformer(
+        input_timesteps= SEQ_LENGTH,         # Sequence length
+        in_channels= INPUT_CHANNELS,         # Number of input channels
+        patch_size=PATCH_SIZE,               # Patch size for embedding
+        embedding_dim=EMBED_DIM.             # Embedding dimension
+        num_transformer_layers=NUM_LAYERS,   # Number of transformer layers (4, 8, etc.)
+        num_heads=N_HEADS,                   # Number of attention heads
+        num_layers=NUM_LAYERS,               # Number of transformer layers
+        dim_feedforward=DIM_FF,              # Feedforward dimension
+        dropout=DROPOUT,                     # Dropout rate (0.1, 0.2, etc.)
+        num_classes= NUM_CLASSES             # Number of output classes
+        pos_encoding='PE_Name',              # Positional encoding type
+    )
+
+# Forward pass
+x = torch.randn(BATCH_SIZE, SEQ_LENGTH, INPUT_CHANNELS)  # (batch, sequence, features)
+output = model(x)
+```
+
 ## Results
 
-Our experimental evaluation encompasses eight distinct positional encoding methods tested across eleven diverse time series datasets using two transformer architectures.
+Our experimental evaluation encompasses ten distinct positional encoding methods tested across eleven diverse time series datasets using two transformer architectures.
 
 ### Key Findings
 
@@ -86,9 +112,9 @@ Our experimental evaluation encompasses eight distinct positional encoding metho
 - **Patch Embedding**: More balanced performance among top methods
 
 #### 🏆 Average Rankings
-- **SPE**: 1.727 (batch norm), 2.090 (patch embed)
-- **TUPE**: 1.909 (batch norm), 2.272 (patch embed)
-- **T-PE**: 2.636 (batch norm), 2.363 (patch embed)
+- **SPE**: 1.727 (TST), 2.090 (patch embed)
+- **TUPE**: 1.909 (TST), 2.272 (patch embed)
+- **T-PE**: 2.636 (TST), 2.363 (patch embed)
 
 ### Performance Analysis
 

src/encodings/positional_encodings.py

@@ -45,6 +45,82 @@ def __init__(self, d_model, dropout=0.1, max_len=1024, scale_factor=1.0):
     def forward(self, x):
         x = x + self.pe[:, :x.size(1)]
         return self.dropout(x)
+    
+class RotaryPositionalEncoding(nn.Module):
+    """Rotary Position Embedding (RoPE) - used in models like LLaMA"""
+    def __init__(self, d_model, dropout=0.1, max_len=5000):
+        super(RotaryPositionalEncoding, self).__init__()
+        self.dropout = nn.Dropout(p=dropout)
+        self.d_model = d_model
+
+        # Create frequency matrix
+        inv_freq = 1.0 / (10000 ** (torch.arange(0, d_model, 2).float() / d_model))
+        self.register_buffer('inv_freq', inv_freq)
+
+    def forward(self, x):
+        seq_len = x.shape[1]
+        device = x.device
+
+        # Generate position indices
+        position = torch.arange(seq_len, device=device).float()
+
+        # Create frequency matrix for all positions
+        freqs = torch.outer(position, self.inv_freq)
+        freqs = torch.cat([freqs, freqs], dim=-1)
+
+        # Apply rotary embedding
+        cos_freqs = freqs.cos()
+        sin_freqs = freqs.sin()
+
+        # Reshape for broadcasting
+        cos_freqs = cos_freqs.unsqueeze(0).expand(x.shape[0], -1, -1)
+        sin_freqs = sin_freqs.unsqueeze(0).expand(x.shape[0], -1, -1)
+
+        # Apply rotation
+        x_rotated = self.apply_rotary_pos_emb(x, cos_freqs, sin_freqs)
+        return self.dropout(x_rotated)
+
+    def apply_rotary_pos_emb(self, x, cos, sin):
+        # Split the last dimension in half
+        x1, x2 = x[..., ::2], x[..., 1::2]
+
+        # Apply rotation
+        rotated = torch.zeros_like(x)
+        rotated[..., ::2] = x1 * cos[..., ::2] - x2 * sin[..., ::2]
+        rotated[..., 1::2] = x1 * sin[..., 1::2] + x2 * cos[..., 1::2]
+
+        return rotated
+
+class RelativePositionalEncoding(nn.Module):
+    """Relative Positional Encoding - focuses on relative distances between tokens"""
+    def __init__(self, d_model, dropout=0.1, max_len=5000):
+        super(RelativePositionalEncoding, self).__init__()
+        self.dropout = nn.Dropout(p=dropout)
+        self.d_model = d_model
+        self.max_len = max_len
+
+        # Learnable relative position embeddings
+        self.relative_positions = nn.Parameter(
+            torch.randn(2 * max_len - 1, d_model) * 0.02
+        )
+
+    def forward(self, x):
+        batch_size, seq_len, d_model = x.shape
+
+        # Create relative position matrix
+        positions = torch.arange(seq_len, device=x.device)
+        relative_positions = positions[:, None] - positions[None, :]
+        relative_positions += self.max_len - 1  # Shift to positive indices
+
+        # Get relative position embeddings
+        rel_pos_emb = self.relative_positions[relative_positions]
+
+        # Average the relative position embeddings for each position
+        pos_encoding = rel_pos_emb.mean(dim=1).unsqueeze(0).expand(batch_size, -1, -1)
+
+        x = x + pos_encoding
+        return self.dropout(x)
+    
 
 class AbsolutePositionalEncoding(nn.Module):
     def __init__(self, d_model, dropout=0.1, max_len=1024, scale_factor=1.0):