todo: run the EFA transport on EFA hardware and report the numbers

[thxCode]

What is left undone

KVCacheBackend.spec.transport.protocol admits EFA and mirrored-mooncake compiles the EFA
transport in — but that is measured as compiled into a published artifact, the bar the API
comment sets. It has never moved a byte: no member has come up on efa, no transfer has crossed an
EFA link.

This issue is the hardware run that settles it, for EFA what the cluster e2e cases were for the
leader HA and the local-disk tier.

What to run

On two EFA-capable instances (p5/p5e/p5en/p6 for the headline numbers, or a cheaper EFA family such
as c7i/m7i/r7i for a functional pass), with the AWS EFA driver installed (fi_info -p efa answers):

1. Member comes up on efa. A KVCacheBackend with transport.protocol: EFA; the member Pod
   mounts /opt/amazon/efa and reports the EFA device lines
   (EFA device (libfabric): rdmap…, provider: efa) in its startup log;
   status.members[].protocol reports what the leader observed.
2. A transfer crosses the fabric. Write/read keys from a client on the peer node and confirm
   the bytes moved over EFA (per-device counters / fi_info), not over a fallback.
3. CPU-to-CPU is the relevant path here. This operator's medium is DRAM; upstream's
   transfer_engine_bench --protocol=efa --use_vram=false numbers (block=1MB) are the reference
   point — DRAM bandwidth is the expected ceiling, not the NICs.
4. FailedMount honesty. A node without the EFA driver must stop the member at FailedMount
   (the /opt/amazon/efa mount is Directory), never start against an empty stand-in.

What to watch for

• Instance family matters: i7ie is not EFA-capable (EfaSupported: false, measured via
  describe-instance-types) — the storage-optimized family the KV-cache e2e clusters used cannot
  run this. Pick the instance first.
• The operator mounts the host's /opt/amazon/efa and puts its lib/ first through
  LD_LIBRARY_PATH; the image's distro libfabric is only a load-time fallback. A version mismatch
  between container libfabric and host EFA driver surfaces as
  fi_ep_bind (av) failed: Function not implemented — that is the mount/LD_LIBRARY_PATH not
  taking effect, not a transport bug.
• Same-host loopback on DRAM buffers is safe (SHM memcpy path); the documented segfault
  (prov/efa: SHM intra-node path segfaults in host memcpy when the buffer is FI_HMEM_CUDA device memory ofiwg/libfabric#12328) applies to GPU buffers only, which this scope does not mount.

References

• Upstream design doc: docs/source/design/transfer-engine/efa_transport.md in kvcache-ai/Mooncake
  (CPU-to-CPU benchmark tables included).
• Follow-up: engine Pods currently get no fabric access on any host-fabric path — tracked
  separately.