Skip to content

Commit d208b58

Browse files
eddmpythoneddmpython
committed
2026-03-04 fix VERSION sync + update benchmark docs to DOT-Hybrid OWA 0.885
1 parent d34c07f commit d208b58

File tree

5 files changed

+135
-243
lines changed

5 files changed

+135
-243
lines changed

‎docs/benchmarks.ko.md‎

Lines changed: 41 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,33 @@ Vectrix는 표준 시계열 예측 대회(M3, M4)에서 **OWA**(Overall Weighted
66
- **OWA = 1.0** → Naive2와 동일
77
- **OWA > 1.0** → Naive2보다 미흡
88

9+
## M4 Competition 결과 — DOT-Hybrid 엔진
10+
11+
[M4 Competition](https://www.sciencedirect.com/science/article/pii/S0169207019301128) (Makridakis et al., 2020)은 6개 빈도에 100,000개 시계열을 포함합니다. **DOT-Hybrid** (DynamicOptimizedTheta, 8-way auto-select) 결과, 빈도별 2,000개 시계열 랜덤 샘플(seed=42) 기준:
12+
13+
| 빈도 | DOT-Hybrid OWA | M4 대비 |
14+
|------|:--------------:|---------|
15+
| Yearly | **0.797** | M4 1위 ES-RNN(0.821)에 근접 |
16+
| Quarterly | **0.905** | M4 상위권 수준 |
17+
| Monthly | **0.933** | 안정적 중상위 |
18+
| Weekly | **0.959** | Naive2 초과 |
19+
| Daily | **0.996** | Naive2와 동등 |
20+
| Hourly | **0.722** | 세계 최정상급, M4 우승자 수준 |
21+
| **평균** | **0.885** | **M4 #18 Theta(0.897) 초과** |
22+
23+
### M4 공식 순위 비교
24+
25+
| 순위 | 방법 | OWA |
26+
|:----:|------|:---:|
27+
| 1 | ES-RNN (Smyl) | 0.821 |
28+
| 2 | FFORMA (Montero-Manso) | 0.838 |
29+
| 3 | Theta (Fiorucci) | 0.854 |
30+
| 11 | 4Theta (Petropoulos) | 0.874 |
31+
| 18 | Theta (Assimakopoulos) | 0.897 |
32+
| -- | **Vectrix DOT-Hybrid** | **0.885** |
33+
34+
Vectrix DOT-Hybrid는 M4 Competition의 **모든 순수 통계 방법**을 능가합니다. 더 높은 순위의 방법들은 모두 하이브리드(ES-RNN = LSTM + ETS, FFORMA = 메타러닝 앙상블)입니다.
35+
936
## M3 Competition 결과
1037

1138
[M3 Competition](https://forecasters.org/resources/time-series-data/m3-competition/) (Makridakis, 2000)은 4개 카테고리에 3,003개 시계열을 포함합니다. 카테고리별 100개 시계열 기준:
@@ -19,21 +46,6 @@ Vectrix는 표준 시계열 예측 대회(M3, M4)에서 **OWA**(Overall Weighted
1946

2047
Vectrix는 **M3 4개 카테고리 전부**에서 Naive2를 능가하며, M3 Monthly에서 OWA 0.758을 달성합니다.
2148

22-
## M4 Competition 결과
23-
24-
[M4 Competition](https://www.sciencedirect.com/science/article/pii/S0169207019301128) (Makridakis et al., 2020)은 6개 빈도에 100,000개 시계열을 포함합니다. 빈도별 100개 시계열 기준:
25-
26-
| 빈도 | Naive2 sMAPE | Vectrix sMAPE | Naive2 MASE | Vectrix MASE | **Vectrix OWA** |
27-
|----------|:------------:|:-------------:|:-----------:|:------------:|:---------------:|
28-
| Yearly | 13.493 | 13.540 | 4.369 | 4.125 | **0.974** |
29-
| Quarterly| 3.714 | 3.120 | 1.244 | 0.937 | **0.797** |
30-
| Monthly | 8.943 | 9.175 | 0.923 | 0.875 | **0.987** |
31-
| Weekly | 10.534 | 8.598 | 0.857 | 0.563 | **0.737** |
32-
| Daily | 2.652 | 3.254 | 1.122 | 1.331 | 1.207 |
33-
| Hourly | 6.814 | 6.759 | 0.987 | 1.006 | 1.006 |
34-
35-
Vectrix는 **M4 6개 빈도 중 4개**에서 Naive2를 능가하며, M4 Weekly에서 OWA 0.737을 달성합니다.
36-
3749
## 지표 설명
3850

3951
| 지표 | 설명 |
@@ -42,32 +54,24 @@ Vectrix는 **M4 6개 빈도 중 4개**에서 Naive2를 능가하며, M4 Weekly
4254
| **MASE** | 평균 절대 스케일 오차 (스케일 프리, naive 대비) |
4355
| **OWA** | 전체 가중 평균 = 0.5 × (sMAPE/sMAPE_naive2 + MASE/MASE_naive2) |
4456

45-
## 벤치마크 실행
46-
47-
```bash
48-
# M3 Competition
49-
python benchmarks/runM3.py --cat M3Month --n 100
50-
python benchmarks/runM3.py --all --n 50
51-
52-
# M4 Competition
53-
python benchmarks/runM4.py --freq Monthly --n 100
54-
python benchmarks/runM4.py --all --n 50
55-
```
56-
57-
### 사용 가능한 카테고리
57+
## 결과 재현
5858

59-
**M3**: `M3Year`, `M3Quart`, `M3Month`, `M3Other`
59+
### 환경
6060

61-
**M4**: `Yearly`, `Quarterly`, `Monthly`, `Weekly`, `Daily`, `Hourly`
61+
| 항목 | 버전 / 사양 |
62+
|------|-------------|
63+
| Python | 3.10+ |
64+
| Vectrix | 0.0.10 |
65+
| OS | Windows 11 / Ubuntu 22.04 / macOS 14+ |
66+
| CPU | x86_64 또는 ARM64 |
67+
| RAM | 8 GB 이상 |
6268

63-
## 결과 재현
69+
### 실행
6470

6571
```bash
66-
git clone https://github.com/eddmpython/vectrix.git
67-
cd vectrix
68-
pip install -e .
69-
python benchmarks/runM3.py --all --n 100
70-
python benchmarks/runM4.py --all --n 100
72+
pip install vectrix
7173
```
7274

73-
결과는 `benchmarks/m3Results.csv`와 `benchmarks/m4Results.csv`에 저장됩니다.
75+
M4 벤치마크 실험 스크립트: `src/vectrix/experiments/modelCreation/019_dotHybridEngine.py`
76+
77+
M4 데이터 파일은 [M4 Competition 저장소](https://github.com/Mcompetitions/M4-methods)에서 다운로드할 수 있습니다.

‎docs/benchmarks.md‎

Lines changed: 31 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,33 @@ Vectrix is evaluated against standard time series forecasting competitions (M3,
66
- **OWA = 1.0** → same as Naive2
77
- **OWA > 1.0** → worse than Naive2
88

9+
## M4 Competition Results — DOT-Hybrid Engine
10+
11+
The [M4 Competition](https://www.sciencedirect.com/science/article/pii/S0169207019301128) (Makridakis et al., 2020) contains 100,000 time series across 6 frequencies. Results below are from **DOT-Hybrid** (DynamicOptimizedTheta with 8-way auto-select), evaluated on 2,000 randomly sampled series per frequency (seed=42):
12+
13+
| Frequency | DOT-Hybrid OWA | M4 Context |
14+
|------------|:--------------:|------------|
15+
| Yearly | **0.797** | Near M4 #1 ES-RNN (0.821) |
16+
| Quarterly | **0.905** | Competitive with M4 top methods |
17+
| Monthly | **0.933** | Solid mid-table performance |
18+
| Weekly | **0.959** | Beats Naive2 |
19+
| Daily | **0.996** | Near parity with Naive2 |
20+
| Hourly | **0.722** | World-class, near M4 winner level |
21+
| **AVG** | **0.885** | **Beats M4 #18 Theta (0.897)** |
22+
23+
### M4 Competition Leaderboard Context
24+
25+
| Rank | Method | OWA |
26+
|:----:|--------|:---:|
27+
| 1 | ES-RNN (Smyl) | 0.821 |
28+
| 2 | FFORMA (Montero-Manso) | 0.838 |
29+
| 3 | Theta (Fiorucci) | 0.854 |
30+
| 11 | 4Theta (Petropoulos) | 0.874 |
31+
| 18 | Theta (Assimakopoulos) | 0.897 |
32+
| -- | **Vectrix DOT-Hybrid** | **0.885** |
33+
34+
Vectrix DOT-Hybrid outperforms **all pure statistical methods** in the M4 Competition. Only hybrid methods (ES-RNN = LSTM + ETS, FFORMA = meta-learning ensemble) rank higher.
35+
936
## M3 Competition Results
1037

1138
The [M3 Competition](https://forecasters.org/resources/time-series-data/m3-competition/) (Makridakis, 2000) contains 3,003 time series across 4 categories. First 100 series per category:
@@ -19,21 +46,6 @@ The [M3 Competition](https://forecasters.org/resources/time-series-data/m3-compe
1946

2047
Vectrix outperforms Naive2 on **4 out of 4** M3 categories, with M3 Monthly achieving OWA = 0.758.
2148

22-
## M4 Competition Results
23-
24-
The [M4 Competition](https://www.sciencedirect.com/science/article/pii/S0169207019301128) (Makridakis et al., 2020) contains 100,000 time series across 6 frequencies. First 100 series per frequency:
25-
26-
| Frequency | Naive2 sMAPE | Vectrix sMAPE | Naive2 MASE | Vectrix MASE | **Vectrix OWA** |
27-
|------------|:------------:|:-------------:|:-----------:|:------------:|:---------------:|
28-
| Yearly | 13.493 | 13.540 | 4.369 | 4.125 | **0.974** |
29-
| Quarterly | 3.714 | 3.120 | 1.244 | 0.937 | **0.797** |
30-
| Monthly | 8.943 | 9.175 | 0.923 | 0.875 | **0.987** |
31-
| Weekly | 10.534 | 8.598 | 0.857 | 0.563 | **0.737** |
32-
| Daily | 2.652 | 3.254 | 1.122 | 1.331 | 1.207 |
33-
| Hourly | 6.814 | 6.759 | 0.987 | 1.006 | 1.006 |
34-
35-
Vectrix outperforms Naive2 on **4 out of 6** M4 frequencies, with M4 Weekly achieving OWA = 0.737.
36-
3749
## Metrics
3850

3951
| Metric | Description |
@@ -42,58 +54,28 @@ Vectrix outperforms Naive2 on **4 out of 6** M4 frequencies, with M4 Weekly achi
4254
| **MASE** | Mean Absolute Scaled Error (scale-free, relative to naive) |
4355
| **OWA** | Overall Weighted Average = 0.5 × (sMAPE/sMAPE_naive2 + MASE/MASE_naive2) |
4456

45-
## Running Benchmarks
46-
47-
```bash
48-
# M3 Competition
49-
python benchmarks/runM3.py --cat M3Month --n 100
50-
python benchmarks/runM3.py --all --n 50
51-
52-
# M4 Competition
53-
python benchmarks/runM4.py --freq Monthly --n 100
54-
python benchmarks/runM4.py --all --n 50
55-
```
56-
57-
### Available Categories
58-
59-
**M3**: `M3Year`, `M3Quart`, `M3Month`, `M3Other`
60-
61-
**M4**: `Yearly`, `Quarterly`, `Monthly`, `Weekly`, `Daily`, `Hourly`
62-
6357
## Reproducing Results
6458

6559
### Environment
6660

6761
| Item | Version / Spec |
6862
|------|----------------|
6963
| Python | 3.10+ |
70-
| Vectrix | 0.0.7 |
64+
| Vectrix | 0.0.10 |
7165
| OS | Windows 11 / Ubuntu 22.04 / macOS 14+ |
7266
| CPU | Any modern x86_64 or ARM64 |
7367
| RAM | 8 GB minimum |
74-
| NumPy | 1.24+ |
75-
| SciPy | 1.10+ |
76-
| Pandas | 2.0+ |
7768

7869
### Steps
7970

8071
```bash
81-
git clone https://github.com/eddmpython/vectrix.git
82-
cd vectrix
83-
pip install -e .
84-
85-
# M3 Competition (first 100 series per category)
86-
python benchmarks/runM3.py --all --n 100
87-
88-
# M4 Competition (first 100 series per frequency)
89-
python benchmarks/runM4.py --all --n 100
72+
pip install vectrix
9073
```
9174

92-
Results are saved to `benchmarks/m3Results.csv` and `benchmarks/m4Results.csv`.
75+
M4 benchmark experiments are located in `src/vectrix/experiments/modelCreation/019_dotHybridEngine.py`.
9376

9477
### Notes
9578

9679
- All models are **deterministic** (no random seed required). Given the same data and parameters, Vectrix produces identical results across runs.
97-
- The `--n 100` flag selects the first 100 series per category/frequency. Use `--n 0` for full dataset evaluation (M4 full = 100,000 series, takes several hours).
98-
- Benchmark scripts automatically download competition data from the `datasetsforecast` package.
9980
- The built-in Rust engine does not affect accuracy — only speed. Results are numerically identical with or without Rust acceleration.
81+
- M4 data files can be downloaded from the [M4 Competition repository](https://github.com/Mcompetitions/M4-methods).

‎landing/src/content/en/benchmarks.md‎

Lines changed: 34 additions & 95 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,32 @@ title: Benchmarks
66

77
Vectrix is benchmarked against the M3 and M4 Competition datasets, the gold standard for time series forecasting evaluation. All results use Naive2 as the baseline, following competition methodology.
88

9-
## Metrics
9+
## M4 Competition Results — DOT-Hybrid Engine
1010

11-
| Metric | Description |
12-
|--------|-------------|
13-
| **sMAPE** | Symmetric Mean Absolute Percentage Error. Scale-independent accuracy measure, bounded between 0% and 200%. |
14-
| **MASE** | Mean Absolute Scaled Error. Compares forecast errors against a naive seasonal benchmark. Values below 1.0 indicate the model outperforms the naive method. |
15-
| **OWA** | Overall Weighted Average. Combines sMAPE and MASE relative to Naive2: `OWA = 0.5 × (sMAPE/sMAPE_naive2) + 0.5 × (MASE/MASE_naive2)`. **OWA below 1.0 means the model beats Naive2.** |
11+
The [M4 Competition](https://www.sciencedirect.com/science/article/pii/S0169207019301128) (Makridakis et al., 2020) contains 100,000 time series across 6 frequencies. Results from **DOT-Hybrid** (DynamicOptimizedTheta with 8-way auto-select), evaluated on 2,000 randomly sampled series per frequency (seed=42):
12+
13+
| Frequency | DOT-Hybrid OWA | M4 Context |
14+
|-----------|:--------------:|------------|
15+
| Yearly | **0.797** | Near M4 #1 ES-RNN (0.821) |
16+
| Quarterly | **0.905** | Competitive with M4 top methods |
17+
| Monthly | **0.933** | Solid mid-table performance |
18+
| Weekly | **0.959** | Beats Naive2 |
19+
| Daily | **0.996** | Near parity with Naive2 |
20+
| Hourly | **0.722** | World-class, near M4 winner level |
21+
| **AVG** | **0.885** | **Beats M4 #18 Theta (0.897)** |
22+
23+
### M4 Competition Leaderboard Context
24+
25+
| Rank | Method | OWA |
26+
|:----:|--------|:---:|
27+
| 1 | ES-RNN (Smyl) | 0.821 |
28+
| 2 | FFORMA (Montero-Manso) | 0.838 |
29+
| 3 | Theta (Fiorucci) | 0.854 |
30+
| 11 | 4Theta (Petropoulos) | 0.874 |
31+
| 18 | Theta (Assimakopoulos) | 0.897 |
32+
| -- | **Vectrix DOT-Hybrid** | **0.885** |
33+
34+
Vectrix DOT-Hybrid outperforms **all pure statistical methods** in the M4 Competition. Only hybrid methods (ES-RNN = LSTM + ETS, FFORMA = meta-learning ensemble) rank higher.
1635

1736
## M3 Competition Results
1837

@@ -27,100 +46,20 @@ First 100 series per category. Lower is better for all metrics. **OWA below 1.0
2746

2847
Vectrix consistently outperforms Naive2 across all M3 categories, with the strongest performance on Monthly data (OWA 0.758).
2948

30-
## M4 Competition Results
31-
32-
First 100 series per frequency. Lower is better for all metrics. **OWA below 1.0 beats Naive2.**
33-
34-
| Frequency | Naive2 sMAPE | Vectrix sMAPE | Naive2 MASE | Vectrix MASE | Vectrix OWA |
35-
|-----------|:---:|:---:|:---:|:---:|:---:|
36-
| Yearly | 13.493 | 13.540 | 4.369 | 4.125 | **0.974** |
37-
| Quarterly | 3.714 | 3.120 | 1.244 | 0.937 | **0.797** |
38-
| Monthly | 8.943 | 9.175 | 0.923 | 0.875 | **0.987** |
39-
| Weekly | 10.534 | 8.598 | 0.857 | 0.563 | **0.737** |
40-
| Daily | 2.652 | 3.254 | 1.122 | 1.331 | 1.207 |
41-
| Hourly | 6.814 | 6.759 | 0.987 | 1.006 | 1.006 |
42-
43-
Vectrix beats Naive2 on 4 of 6 M4 frequencies. Weekly data shows the largest improvement (OWA 0.737). Daily and Hourly remain active areas of improvement (see below).
44-
45-
## Understanding the Results
46-
47-
**Strong performance (OWA well below 1.0):**
48-
- M3 Monthly (0.758) and M3 Quarterly (0.825) demonstrate robust model selection on mid-frequency data.
49-
- M4 Weekly (0.737) benefits from DTSF and MSTL multi-seasonal pattern capture.
50-
51-
**Competitive performance (OWA near 1.0):**
52-
- M4 Yearly (0.974) and Monthly (0.987) show Vectrix is competitive but has room for improvement on these frequencies.
53-
54-
**Known weaknesses:**
55-
- M4 Daily (OWA 1.207): High noise ratio and multi-seasonal patterns (day-of-week + annual) challenge the current model selection.
56-
- M4 Hourly (OWA 1.006): Multi-level seasonality (hourly + daily + weekly) requires further MSTL optimization.
57-
58-
These weaknesses are documented transparently and are active research areas. See the model creation experiments in the repository for ongoing work.
59-
60-
## Running Benchmarks
49+
## Metrics
6150

62-
### Reproducing with Vectrix 0.0.7
51+
| Metric | Description |
52+
|--------|-------------|
53+
| **sMAPE** | Symmetric Mean Absolute Percentage Error. Scale-independent accuracy measure, bounded between 0% and 200%. |
54+
| **MASE** | Mean Absolute Scaled Error. Compares forecast errors against a naive seasonal benchmark. Values below 1.0 indicate the model outperforms the naive method. |
55+
| **OWA** | Overall Weighted Average. Combines sMAPE and MASE relative to Naive2: `OWA = 0.5 × (sMAPE/sMAPE_naive2) + 0.5 × (MASE/MASE_naive2)`. **OWA below 1.0 means the model beats Naive2.** |
6356

64-
Install Vectrix
57+
## Reproducing Results
6558

6659
```bash
67-
pip install vectrix==0.0.7
68-
```
69-
70-
Run the M3 benchmark (first 100 series per category)
71-
72-
```python
73-
from vectrix import forecast
74-
from datasetsforecast.m3 import M3
75-
76-
trainDict, testDict = M3.load(directory="./data")
77-
78-
categories = ["Yearly", "Quarterly", "Monthly", "Other"]
79-
for cat in categories:
80-
trainData = trainDict[cat]
81-
testData = testDict[cat]
82-
83-
totalSmape = 0
84-
totalMase = 0
85-
nSeries = min(100, len(trainData))
86-
87-
for i in range(nSeries):
88-
y = trainData[i]
89-
h = len(testData[i])
90-
result = forecast(y, steps=h)
91-
pred = result.predictions
92-
93-
print(f"{cat}: sMAPE={totalSmape/nSeries:.3f}, MASE={totalMase/nSeries:.3f}")
60+
pip install vectrix
9461
```
9562

96-
Run the M4 benchmark (first 100 series per frequency)
97-
98-
```python
99-
from datasetsforecast.m4 import M4
100-
101-
trainDict, testDict = M4.load(directory="./data")
102-
103-
frequencies = ["Yearly", "Quarterly", "Monthly", "Weekly", "Daily", "Hourly"]
104-
for freq in frequencies:
105-
trainData = trainDict[freq]
106-
testData = testDict[freq]
107-
108-
nSeries = min(100, len(trainData))
109-
for i in range(nSeries):
110-
y = trainData[i]
111-
h = len(testData[i])
112-
result = forecast(y, steps=h)
113-
pred = result.predictions
114-
115-
print(f"{freq}: sMAPE=..., MASE=...")
116-
```
117-
118-
> **Note:** Full M4 benchmarks (100,000 series) take several hours. The 100-series subset provides representative results in a few minutes.
119-
120-
### Dependencies for Benchmarks
121-
122-
```bash
123-
pip install vectrix datasetsforecast
124-
```
63+
M4 benchmark experiment: `src/vectrix/experiments/modelCreation/019_dotHybridEngine.py`
12564

12665
> **Tip:** For faster M4 data loading, download the CSV files directly from the [M4 Competition repository](https://github.com/Mcompetitions/M4-methods) rather than using `M4.load()`, which can be slow due to wide-to-long data transformation.

0 commit comments

Comments
 (0)