fix allreduce usage (meta-pytorch#279)

Summary:

pass full allreduce options to the pg allreduce to avoid the watchdog abort from getting triggered

Differential Revision: D84101243

Author: tushar00jain

torchft/manager.py

@@ -423,7 +423,9 @@ def allreduce(
                     torch.accelerator.current_stream(),
                 )
             else:
-                work = self._pg.allreduce([tensor], reduce_op)
+                opts = AllreduceOptions()
+                opts.reduceOp = reduce_op
+                work = self._pg.allreduce([tensor], opts)
 
             # schedule grad normalization as a continuation
             # on the Future