Backend framework comparison

We want to compare TensorFlow, PyTorch, JAX and maybe other similar frameworks here.

We do not want to compare higher-level frameworks like Keras here.

Side node: The classification low-level, mid-level and high-level framework comes from our tutorial (video, slides). Maybe what we call "backend framework" here corresponds most closely to "mid-level framework". But the distinction is not always clear. Here we mean all frameworks which could potentially be used as backend for RETURNN.

This comparison is specifically about their properties specifically when used as a backend framework for RETURNN, using the RETURNN principles and core features.

Theano

• Our initial backend.
• The most widely used backend framework around ~2016-2017 maybe?

TensorFlow

• Our second backend.
• The most widely used backend framework around ~2017-2019 maybe?

PyTorch

• Probably the most widely used framework since around ~2019?

JAX

• Recently gained lots of interest in the community.