3.3 Add Fusion model

[hughmancoder]

Follow paper A Deep-Learning Framework with Multi-Feature Fusion and Attention Mechanism for Classification of Chinese Traditional Instruments

[hughmancoder]

Adapt method in
A Deep-Learning Framework with Multi-Feature Fusion and Attention Mechanism for Classification of Chinese Traditional Instruments”:

• Download CTIS dataset (China Conservatory of Music) — 216 instruments grouped into 4 families (Wind, Percussion, Plucked String, Bowed String).
• Split data into Train : Validation : Test = 8 : 1 : 1 by total audio duration (~100 039 s train, ~12 505 s val/test).

Preprocessing suggestions

• Clip or pad all recordings to 5 s duration — pad shorter clips, trim longer ones.
• Normalize volume (e.g., peak normalization).
• Optionally augment using pitch shift or time stretch to increase robustness.

Feature Extraction

Using Librosa:

MFCC

Extract 13 coefficients per frame, 50% overlap.

Apply pre-emphasis → framing → Hamming window → FFT → Mel filterbank → log → DCT.

CQT

Use 84 bins to cover the instrument frequency range (20 Hz – 8 kHz).

Maintain logarithmic frequency spacing.

Chroma

Compute 12-band chroma features using STFT and triangular filter bank.

Normalise each feature (Min-Max scaling).

Align frequency scales for fusion.

stacking

Stacking (proposed): Create 3-channel “RGB-like” spectrograms with (3 × 128 × 128) tensors per sample

Channel 1: MFCC

Channel 2: CQT

Channel 3: Chroma

model architecture

Baseline CNN (inspired by VGGNet):
BatchNorm → 2×2 MaxPool.

Channel sizes: 32 → 64 → 128.

Global Average Pooling.

Dropout (p ≈ 0.5).

2 Fully Connected layers → LogSoftmax output (4 classes).

embed attention between layers

Embed between conv layers:

Channel Attention:

Global pooling

Adding Spatial Attention

2D convs on aggregated features, element-wise multiply with feature map.

Combine both (channel + spatial) sequentially

test on metrics

Accuracy

Precision

Recall

F1 Score

Confusion Matrix