model not converging the sparse categorical accuracy stays same for the whole epochs.

[subalbeura]

while comparing with the results provided in https://keras.io/examples/timeseries/timeseries_classification_transformer/, the dense layer is different.

dense_2 (Dense)             (None, 128)                  256       ['global_average_pooling1d_2[0
                                                                    ][0]']                        
                                                                                                  
 dropout_25 (Dropout)        (None, 128)                  0         ['dense_2[0][0]']  

how to fix it?

[jentstemmerman]

Same experience, the model training stops because of the EarlyStopping callback.
Setting callbacks = None results in 150 EPOCHS of zero progress in accuracy.

Also reproducible in Colab

[jentstemmerman]

I think I might have found the solution. The GlobalAveragePooling1D just before the dense layer returns the wrong shape.
It returns (None, 1) with input (None, 500, 1), but the result in the example page is (None, 500)
Changing the parameter data_format into channels_first like below fixes that:
x = layers.GlobalAveragePooling1D(data_format="channels_first", name="TooSmallOutput")(x)

With that change, the model does converge.

DISCLAIMER: I am new to transformers, only played with LSTM's so far.
From the tensorflow spec:
"channels_last" corresponds to inputs with shape (batch, steps, features) while "channels_first" corresponds to inputs with shape (batch, features, steps)

If 500 is the number of steps ( the sequence length?) and there is only one feature in the dataset, channels_last makes sense, so I don't understand why channel_first works and channel_last does not.

[subalbeura]

Without " name="TooSmallOutput" it is also working.
Regarding TNN, I am also new to the machine learning and am currently trying some examples.

[jentstemmerman]

Ah I forgot to remove that.
I added the "name="TooSmallOutput" to debug it.
It just prints out the name in the model summary.

[subalbeura]

Thanks for the reply

[subalbeura]

For multiclass classification using time series data, the accuracy comes nearly 0.009. What are the factors that can improve accuracy?

[sonali-kumari1]

Hi @subalbeura -
Thanks for reporting this issue. Here is a similar issue #20627 and a PR #2006 was raised to revert back to data_format="channels_first to resolve the accuracy and parameter issues which was introduced by commit a52a1c5.

For improving accuracy in multiclass classification using time series data, consider adding more layers to your model, performing a hyperparameter search, optimizing the learning rate schedule, or trying a different optimizer. Additionally, data augmentation and experimenting with different model architectures can help improve performance.

[github-actions]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed because it has been inactive for 28 days. Please reopen if you'd like to work on this further.