How to determine the range of num_hidden_unit given the input feature dimension.

[AlbertXTang]

Thank you for developing such a powerful tool! I'm currently experimenting with grid search and wondering if there are any fundamental principles for determining the number of hidden units given the input data dimensions?
Specifically, can the number of hidden units exceed the input data dimensions? For example, when the input data is 70, is it reasonable to consider setting 1.5 to 2 times the number of hidden units to capture more information?

[stes]

Hi @AlbertXTang , thanks for asking.

Specifically, can the number of hidden units exceed the input data dimensions? For example, when the input data is 70, is it reasonable to consider setting 1.5 to 2 times the number of hidden units to capture more information?

Yes, this is possible and in fact recommended. The number of hidden units (alongside the model architecture) determines the capacity/#parameters of the model. With more capacity and very little data, it becomes more likely that the network overfits. With less capacity, you might get underfitting. The factor (1.5, 2, ...) mainly depends on how much data you have available (=samples/timesteps). If you have enough data to avoid overfitting, it is certainly possible to e.g. train a model with hidden dim 256 or even higher.

Projection to a lower dim space is done in the last layer, and the number of output features, is indeed much lower than the input dimensionality.

The best practices notebook has some examples how to practically tune the model!

How large is your dataset?

[AlbertXTang]

Thank you for your response!
My dataset contains 70 features and approximately 40,000 samples, and the sample size seems sufficient to accommodate more hidden units. I've also discovered that using more than 70 hidden units does indeed yield clearer embeddings!

[stes]

Yes, in that range (without knowing any additional details about data complexity), it is def. safe to go to hidden dim of 256-512.