use torchx for manual many replica (20+) tests

[d4l3k]

This adds a torchx component that can be used to run many replica tests locally. This is intended to make it easier to validate E2E behavior since it's now 2 commands to run an e2e job with train_ddp.py.

TorchX is a job launcher and the component we create will launch one role per replica group with torchelastic managing the workers within each group. Elastic uses a port range 29600 + replica_id.

It also includes some small tweaks to logging, static types and the Lighthouse dashboard for monitoring.

Test plan:

torchft_lighthouse --min_replicas 2 --join_timeout_ms 10000 &

torchx run -- --replicas 20

Increasing the join timeout is required to avoid split brain issues when some replicas are recovering.

From experimentation, join_timeout_ms must be longer than the recovery time otherwise workers will never recover.

pyre

[H-Huang]

looks good!