LSTM and additional non-permutational-invariant features

[nadwir]

Hi,

I have two questions:

1. is it possible to have additional features besides the permutational invariant?
2. is it possible to use LSTM-Layers since those require themselves 3D inputs?

[offchan42]

1. Yes. There are many ways to do it. This depends on the nature of your problem. Do you have no clue how to solve your problem and want the features to be used as early in the network as possible? Then the easiest way is to treat additional features as extra properties. You can duplicate them and concatenate them to the properties of each object. I use the vocab written in the docstring of this file: https://github.com/off99555/superkeras/blob/master/permutational_layer.py

For example, if your object consists of 5 properties, and you have 2 extra features, then you could treat your object as having 7 properties, where the last 2 properties are the same for all objects. The extra features are called global features. Intuitively, your object could be a moving ball with properties like position, velocity, and acceleration. If you want to add extra features like the friction of the ground, you can think of it as a global feature that every ball shares. So if the friction is 0.8, then all the balls should have friction=0.8 as its extra feature.
I discovered this global feature duplication trick from PointNet architecture.

Another way is to include the extra features later in the network. After you merge the objects into one vector, then you can concatenate the extra features to this vector. If your network does not have merging, then the global features duplication trick is needed. You just need to decide when to concatenate the global features. You can choose to concatenate as early as possible or you can concatenate after 2 permutational layers, it's up to you.

2. Maybe it's possible to use but you will have to rethink the architecture. The code was not designed for that kind of usage. The paper also did not prepare for that use case. So I'm not sure if it will work. You have to tinker with the idea yourself, come up with architecture ideas first.

[offchan42]

By the way, I suggest using PointNet architecture for permutation invariant instead of PermutationalLayer if you have many objects as input e.g. more than 5. Because running many comparisons using PermutationalLayer is expensive. If you have 5 objects there will be 5**2=25 comparisons using PairwiseModel.