Possible parameter error in time series Transformer model example

[stallam-unb]

The the example for time series classification with transformer, the the function build_model() is defined as:

def build_model(
    input_shape,
    head_size,
    num_heads,
    ff_dim,
    num_transformer_blocks,
    mlp_units,
    dropout=0,
    mlp_dropout=0,
):
    inputs = keras.Input(shape=input_shape)
    x = inputs
    for _ in range(num_transformer_blocks):
        x = transformer_encoder(x, head_size, num_heads, ff_dim, dropout)

    x = layers.GlobalAveragePooling1D(data_format="channels_first")(x) #<- This line appears to be wrong.
    for dim in mlp_units:
        x = layers.Dense(dim, activation="relu")(x)
        x = layers.Dropout(mlp_dropout)(x)
    outputs = layers.Dense(n_classes, activation="softmax")(x)
    return keras.Model(inputs, outputs)

The pooling layer is initialised as x = layers.GlobalAveragePooling1D(data_format="channels_first")(x). Isn't the data format however channels_last?

[SuryanarayanaY]

Hi @stallam-unb ,

The tutorial time series classification with transformer used FordA dataset which as per my understanding is actually time series data collected in 500 timestamps for one feature (i.e Noise). So the dataset which is initially of shape (x,500) converted into (x,500,1) and theoritically it should be of form (batch_size, steps, features).

Even in the Tutorial page under model summary it is mentioned like below.

Our model processes a tensor of shape (batch size, sequence length, features), where sequence length is the number of time steps and features is each input timeseries.

But as per documentation of GlobalAveragePooling1D, data_format="channels_first" means the data should be of shape (batch_size, features, steps)

As per the data we are passing it should be data_format="channels_last".

I may raise a PR for same.

[sonali-kumari1]

Hi @stallam-unb -

Thedata_format="channels_last"change was introduced in PR #1733 for the example Timeseries classification with a Transformer model.

However, after this change, the model no longer learns effectively and it also no longer produces the same number of parameters as shown in the example output in the documentation. To address this issue, PR #2006 was raised to revert back to data_format="channels_first. Thanks!