Distributed inference on M4 Macs using TensorFlow on thunderbolt 5 40-80 Gbps networking

[obriensystems]

Inference across multiple machines using thunderbolt 5 40-80 Gbps networking
Ran initial LLM discussion to get link coverage..

202250209 start with https://www.tensorflow.org/guide/distributed_training

[obriensystems]

20250208: o1 pro generated

write a report on how to design a system to train neural networks across multiple apple mini computers using m4 chips. i would like to use the unified memory across multiple machines via tensorflow or pytorch for example. describe network setup that could decrease latency using thunderbolt 5 networking as a bridge between the machine. Detail also if using kubeflow via kubernetes across the cluster is an optimum method

Designing Multi-Machine Neural Network Training
Reasoned for 2m 6s

Mapping out future scenarios

I’m exploring the design of a neural network training system on multiple upcoming Mac Minis, using Thunderbolt 5 and conventional networking, and pondering shared memory feasibility.

Focusing on network and tools

I’m generating a report on training neural networks on M4 Mac Minis, utilizing unified memory with TF or PyTorch, optimizing latency with Thunderbolt 5, and evaluating Kubernetes for cluster management.

Delving into distributed training

I’m mapping out the limitations of Apple's HPC and examining PyTorch or TensorFlow for distributed training. Also, I’m detailing cluster setup with Kubeflow on Kubernetes and overhead considerations.

Navigating unified memory

Hmm, that's interesting—exploring the practical limits of unified memory across multiple Mac Minis, I’ll present distributed training via data parallel or model parallel approaches, optimizing connectivity with Thunderbolt and Ethernet.

Mapping software solutions

I’m outlining software for distributed training, like Horovod and MPI, and comparing Kubeflow to HPC schedulers. Kubeflow excels in container-based multi-node training and full pipeline automation.

Below is a high-level report outlining considerations, potential architectures, and best practices for building a multi-Mac Mini (with future M4 chips) cluster that can train neural networks in a distributed fashion. We will discuss unified memory, networking approaches (including Thunderbolt 5), and cluster orchestration with Kubernetes/Kubeflow. While Apple’s hardware ecosystem has unique benefits (such as the M-series SoC with Unified Memory), it also imposes constraints that affect how distributed training can be implemented.

---

1. Understanding Apple Silicon’s Unified Memory

1.1 Unified Memory on a Single Mac

• Shared Pool: On an M-series system (e.g., M1, M2, or in the future M4), the CPU, GPU, and Neural Engine share a unified memory pool. This eliminates data copying overheads between CPU and GPU within the same machine.
• High Bandwidth: Apple Silicon typically has very high internal bandwidth, so intra-device data transfers are quite efficient.

1.2 Unified Memory Across Machines?

• Physical Limitations: Although each Mac Mini has unified memory (CPU/GPU/Neural Engine), these pools do not unify across separate physical machines. Each system has its own memory, and currently no standard Apple technology allows you to treat the combined memory of multiple Macs as a single address space in a seamless way.
• Distributed Training: Instead, the common approach is data-parallel or model-parallel training across the cluster. Each node operates on its own set of data or segment of the model and synchronizes parameters over the network (rather than directly pooling memory).

---

2. Distributed Training Architecture

2.1 Data Parallel vs. Model Parallel

• Data Parallel: The most common strategy for multi-node training. Each Mac Mini gets a replica of the model; each processes different training samples (a shard of the dataset). Periodically, they synchronize gradients and update the model parameters.
• Model Parallel: Splits the model layers/tensors across devices. This can be useful for extremely large models but requires careful partitioning logic.
• Hybrid Approaches: In some large-scale scenarios, a mixture of data parallelism and model parallelism is used.

2.2 Framework Support

• PyTorch: Offers native Distributed Data Parallel (DDP) and also supports TensorPipe, Gloo, and MPI backends.
• TensorFlow: Has tf.distribute and also integrates with MPI or other backend libraries for multi-node support (e.g., Horovod).
• Horovod: A well-known library from Uber that works on top of both PyTorch and TensorFlow, simplifying the multi-GPU/multi-node setup.

---

3. Networking Considerations for Low Latency

3.1 Thunderbolt 5 Networking

• Next-Generation Throughput: Thunderbolt 5 (TB5) is expected to deliver up to 80 Gbps or more of bandwidth, theoretically doubling TB4. This can be used to directly connect two Mac Minis or link them in a daisy chain or via a high-speed switch (if or when that becomes available).
• Practical Setup: As of now, multi-system bridging using Thunderbolt can be somewhat tricky. You might rely on Thunderbolt-to-ethernet adapters (like 10GbE or 40GbE if supported) or direct peer-to-peer TB connections.
• Latency and Software: Even with high raw bandwidth, the distributed training performance depends on the framework’s communication backend (MPI, Gloo, etc.). Ensuring low latency means optimizing not only hardware connections but also software network layers.

3.2 Alternative Network Stacks

• 10/25/40 GbE: If Thunderbolt bridging is complex, using a 10/25/40Gb Ethernet setup with high-quality NICs and a compatible switch can also provide robust throughput for gradient synchronization.
• Infiniband: Rarely used in Mac environments, but theoretically an option with external adapters. Likely overkill for a Mac Mini cluster and not officially supported by Apple.

---

4. Cluster Orchestration: Kubernetes & Kubeflow

4.1 Kubernetes on macOS

• Containerization: Typically, running Kubernetes workloads for ML relies on Linux containers. macOS is not natively a container host in the same way Linux is, so you would likely use virtualized or cross-platform tooling.
• K3s or microK8s: Lightweight Kubernetes distributions may be more practical on macOS. Alternatively, you might deploy a small Linux cluster for orchestration and use the macOS machines as worker nodes (though this can introduce complexities).

4.2 Kubeflow

• ML Pipelines: Kubeflow is designed to manage end-to-end ML workflows, from data ingestion to training, hyperparameter tuning, and model serving.
• Advantages:
  1. Scalability: Manage multiple training jobs, schedule them across available nodes.
  2. Experiment Tracking: Orchestrate hyperparameter sweeps, data transformations, and more.
  3. Reproducibility: Container-based approach ensures consistent environments.
• Challenges:
  1. Multi-OS Environment: Mac Minis typically run macOS, whereas most containers are Linux-based.
  2. M-Series GPU/Neural Engine: The GPU stack on Apple Silicon is not as straightforward as standard CUDA on Linux. You may need specialized TensorFlow/PyTorch builds or Apple’s Metal Performance Shaders backend.
  3. Networking Overhead: Each container in a Kubernetes cluster adds overhead for distributed communication.

4.3 Is Kubeflow Optimal for Mac Mini Clusters?

• Pros:
  • Centralized job management.
  • A familiar, portable environment for teams used to Kubernetes in production.
  • Good for large-scale workflows that move beyond mere training, e.g., pipeline automation, model versioning, and experiment management.
• Cons:
  • Overhead of Kubernetes on a macOS-based cluster can be non-trivial, both in terms of compute and complexity.
  • If your primary aim is just raw distributed training, simpler solutions (e.g., using torchrun or Horovod directly across SSH) might suffice and give better performance.

---

5. Example Setup

Below is a conceptual reference architecture for a small cluster of Mac Minis (M4 or future M-series) for distributed training:

1. Hardware

   • 4–8 Mac Minis (M4) each with 32–64 GB of unified memory (configurable).
   • Thunderbolt 5 cables or TB5-to-10/25/40Gb Ethernet adapters.
   • A dedicated high-speed Ethernet switch (if more than two machines).

2. Networking

   • If Thunderbolt 5 bridging is supported in a stable, multi-node way, connect Mac Minis in a daisy chain or star topology.
   • Otherwise, each Mac Mini uses a TB5-to-40GbE (or 10GbE) adapter to a central switch. Aim for symmetrical bandwidth across all nodes.

3. Software Stack

   • Operating System: macOS with developer tools (Xcode CLI, Python, etc.).
   • Python Environment: Use Conda or a virtual environment.
   • Framework:
     • PyTorch with Metal Performance Shaders (MPS) backend on Apple Silicon, or
     • TensorFlow (Apple Silicon build) if it supports M4 in the future.
   • Distributed Communication:
     • PyTorch DDP or Horovod (if you want a single solution for TF/PyTorch).
     • MPI or Gloo backends (whichever offers better performance).
   • Orchestration (optional):
     • Kubernetes (K3s/microK8s) + Kubeflow for large scale workflow.
     • Alternatively, a simpler HPC cluster approach with SSH-based coordination or Slurm-like job scheduling.

4. Training Workflow

   • Data Preparation: Place data in a shared NAS or on each node. If in a shared location, each node should have fast read speeds.
   • Launching Jobs:
     • If using PyTorch DDP: torchrun --nproc_per_node=1 --nnodes=<num_nodes> --node_rank=<rank> --master_addr=<master_ip> train.py
     • If using Kubeflow: Create a TFJob/PyTorchJob manifest that defines the number of replicas and resources.
   • Monitoring & Logging: TensorBoard, Weights & Biases, or another platform, ensuring each node logs to a central location.

---

6. Performance and Optimization Tips

1. Minimize Inter-Node Communication

   • Use larger batch sizes on each node to reduce frequency of gradient synchronization.
   • Overlap computation and communication with asynchronous backends if possible.

2. Profile Network and CPU/GPU Usage

   • Tools like PyTorch’s profiler or TensorFlow’s profiler can help identify whether communication or GPU compute is the bottleneck.

3. Optimize Apple Silicon Backend

   • Ensure the frameworks are compiled or installed with the proper Apple Silicon (Metal/MPS) acceleration.
   • Investigate whether Apple’s Core ML Tools or custom tooling might further speed up parts of the workflow (especially for inference or advanced features of the Neural Engine).

4. Use HPC-Style Job Scheduling (if not using Kubernetes)

   • Tools like Slurm can manage multi-node jobs with a lower overhead than a containerized approach on macOS.
   • This can be simpler if your cluster is dedicated to training and you do not need advanced ML pipeline features.

5. Scale-Out Considerations

   • As you increase the number of machines, the overhead of gradient synchronization can become substantial.
   • Consider network topology (e.g., ring allreduce vs. hierarchical allreduce) to mitigate overhead.

---

7. Summary of Key Points

1. True Unified Memory is Local: Apple’s unified memory is per machine; across machines, you’ll rely on distributed frameworks.
2. Networking is Critical: To reduce latency, leverage Thunderbolt 5 or high-speed Ethernet. Verify that your distributed training framework can fully utilize that bandwidth.
3. Framework Choice: PyTorch and TensorFlow both support distributed training, but each requires configuration for Apple Silicon (MPS or Apple-specific builds).
4. Kubeflow Pros/Cons: While powerful for production ML pipelines, it can introduce overhead in a macOS-based cluster. Simpler HPC job schedulers (or direct SSH-based methods) may be more straightforward for purely training-focused workflows.
5. Optimization: Use standard distributed ML best practices—overlap communication/computation, tune batch sizes, and monitor resource usage to identify bottlenecks.

---

Final Thoughts

Designing a multi-Mac Mini cluster for distributed training involves balancing the benefits of Apple Silicon’s high-efficiency unified memory on each node with the realities of cross-node communication. While you cannot pool memory across machines at a hardware level, you can still achieve scalable training throughput using data-parallel or model-parallel strategies. High-speed interconnects—preferably Thunderbolt 5 or 10/25/40Gb Ethernet—are essential to keep communication overhead manageable.

For orchestration and large-scale pipeline automation, Kubeflow on Kubernetes can be a viable approach, but it might be more complex than necessary if the goal is simply to run distributed training jobs. Ultimately, your setup will hinge on how extensively you want to automate and manage ML workflows (data processing, training, serving) versus simply scaling out training on Apple Silicon.

---

References & Further Reading

• [Apple Developer: Metal Performance Shaders](https://developer.apple.com/metal/metal-performance-shaders/)
• [PyTorch Distributed Overview](https://pytorch.org/tutorials/intermediate/dist_tuto.html)
• [TensorFlow Distributed Overview](https://www.tensorflow.org/guide/distributed_training)
• [Horovod GitHub](https://github.com/horovod/horovod)
• [Kubeflow Website](https://www.kubeflow.org/)

This should provide a solid starting point for your design considerations on a multi-Apple Silicon (M4) Mac Mini cluster, networking specifics, and whether or not to employ Kubeflow/Kubernetes for orchestration.

The cluster size is 3 so that we can use thunderbolt 5 networking. I don’t need a storage solution as each Mac has 512g of duplicated storage. No budget as equipment is already purchased. I need model and data parallelism but model parallelism is the focus. I can use Linux on apple silicon via VMware virtual machines but would prefer a native macOS. My use case is any deep learning architecture. I don’t have a preference on Tensorflow or PyTorch. I am more familiar with Tensorflow.

Multi-Machine Neural Network Training Setup
Reasoned for 1m 9s

Below is a focused design outline for building and running a 3-node Mac Mini cluster (future M4 chips) with Thunderbolt 5 as the high-speed interconnect. The emphasis is on model parallelism—though you can also incorporate data parallelism for flexible scaling. You have local storage on each machine (512 GB each) and prefer native macOS over Linux VMs. While you have no strict preference between frameworks, we will lean toward TensorFlow, given your familiarity.

---

1. Cluster Overview

1.1 Hardware Setup

1. Nodes: 3× Mac Mini (M4)

   • Each with (for example) 32–64 GB unified memory (or more)
   • 512 GB SSD (no shared storage required)
   • macOS running natively

2. Networking: Thunderbolt 5 interconnect

   • The main advantage is high throughput (80 Gbps or more) and relatively low latency.
   • Topology:
     • Star topology (if TB5 hubs/switches are available): Each Mac connects to a central TB5 hub.
     • Chain topology (if star is unavailable): One Mac connected to the next. You’ll need to confirm stable multi-node bridging.
   • Fallback: If direct TB5 bridging proves unreliable, use TB5-to-Ethernet adapters (10/25/40GbE) and a small, high-speed Ethernet switch.

1.2 Software Environment

1. Operating System: macOS (native)
2. Virtualization: Prefer not to use VMs unless certain drivers/frameworks are only Linux-compatible. (Performance overhead is likely smaller on bare-metal macOS for Apple Silicon ML libraries.)
3. TensorFlow: Use an Apple Silicon-optimized build (with Metal backend).
   • Make sure you install a version that supports the M-series GPU/Neural Engine via Metal Performance Shaders.
4. Python Env: Conda environment or venv with the relevant packages:
   • tensorflow-macos
   • tensorflow-metal
   • MPI (if needed for multi-node communication)
   • Additional libraries: h5py, numpy, etc.

---

2. Distributed Strategy: Model Parallelism Focus

2.1 Why Model Parallelism on Apple Silicon?

• Large Models: When your model is too big to fit into a single device’s unified memory (e.g., large language models or generative vision models), you can distribute its layers or tensor shards across multiple devices.
• Leverage All Nodes’ Compute: While Apple’s unified memory is powerful per node, multi-node model parallelism effectively scales total memory and GPU/Neural Engine compute across the cluster.

2.2 TensorFlow Support for Model Parallelism

While TensorFlow has extensive support for data parallelism (e.g., tf.distribute.MirroredStrategy or MultiWorkerMirroredStrategy), model parallelism often requires either:

1. Manual Partitioning of the computational graph (e.g., splitting layers across devices).
2. Pipeline Parallelism: Splitting layers into sequential pipeline stages on each node, especially for large transformer-based models.
3. Third-Party Add-Ons: Some libraries or custom code provide more automated approaches. For instance, [Mesh TensorFlow (older Google library)](https://github.com/tensorflow/mesh) or custom pipeline parallel frameworks.

Note: As of this writing, pure model parallel frameworks for TensorFlow on Apple Silicon are less mature than on CUDA-based systems. You may need to implement custom sharding or pipeline logic.

2.3 Hybrid Strategy (Model + Data Parallel)

• Recommended for 3 Nodes:
  • Stage 1 (Node 1): Some fraction of your layers
  • Stage 2 (Node 2): Next block of layers
  • Stage 3 (Node 3): Final layers
  • Data Parallel within Each Node: If each node has multiple GPU-like resources (e.g., the M4’s GPU plus Neural Engine) you might combine local data parallel with pipeline parallel across nodes.
• Batch Splitting: Each pipeline stage processes microbatches in a pipeline fashion to keep all devices busy.

---

3. Communication Backend & Configuration

3.1 Native TensorFlow Multi-Worker Communication

• MultiWorkerMirroredStrategy in TensorFlow uses the CollectiveOps (i.e., ring allreduce or hierarchical allreduce) over TCP/IP.
• On macOS, you rely on standard network sockets over Thunderbolt. As long as each node sees the others’ IP addresses (via TB5 or TB5–Ethernet bridging), the synchronization can occur.
• For model parallel logic, you still have to manage how the layers are split. The built-in multi-worker strategy automatically handles data parallel training, but not layer-level partitioning.

3.2 MPI/Horovod for Model Parallel

• Horovod (by Uber) can unify data and model parallel patterns in some advanced setups. It can run on macOS, but be sure to compile/install with Apple Silicon compatibility.
• MPI: If you prefer an HPC approach, you can install an MPI library (e.g., OpenMPI) on macOS. Then build TensorFlow with MPI support or run Horovod.
  • This might require building from source or using conda-forge packages that have Apple Silicon builds.

---

4. Practical Setup Steps

1. Network Configuration

   1. Connect the three Mac Minis via Thunderbolt 5 cables.
   2. Ensure each Mac obtains an IP address on the TB5 interface (manually or via DHCP on a small router/hub).
   3. Verify low-latency connectivity (e.g., ping or iperf3 tests).

2. Install Apple Silicon TensorFlow

   • [TensorFlow for macOS (Apple Silicon)](https://developer.apple.com/metal/tensorflow-plugin/).
   • Verify GPU acceleration works locally:

     import tensorflow as tf
     print(tf.config.list_physical_devices('GPU'))

3. Enable Multi-Node

   • Each Mac needs the same environment: same Python version, same TF version, same scripts.
   • A typical setup might rely on environment variables to define the cluster addresses and ports. For example, TF_CONFIG environment variable for MultiWorkerMirroredStrategy or custom code for pipeline parallel.

4. Implement Model Parallel Logic

   • If you’re building a custom pipeline approach, you might manually define subgraphs/layers on each node. For example, in pseudo-code:

     # Node 1
     input = tf.keras.layers.Input(...)
     x = SomeLayers()(input)  # partial model
     # Then send x to Node 2
     
     # Node 2
     # receives x from Node 1
     x = SomeMoreLayers()(x)
     # Then pass x to Node 3
     
     # Node 3
     # receives x from Node 2
     output = FinalLayers()(x)
     model = tf.keras.Model(inputs=input, outputs=output)

   • The tricky part: shipping intermediate tensors between nodes. This typically involves network ops or custom TensorFlow distributed ops.
   • Alternatively, explore existing libraries (like older Mesh TensorFlow) for a more seamless distribution of large tensor dimensions across devices. But MPS (Apple GPU backend) might not be fully tested there.

5. Validate End-to-End Training

   • Start with small synthetic data.
   • Incrementally verify that each pipeline stage runs on the appropriate node.
   • Monitor performance and ensure you are saturating the local GPU/Neural Engine.

6. Performance Tweaks

   • Use microbatching for pipeline parallelism so that node 1 can start the next microbatch while node 2 processes the previous batch’s partial outputs.
   • Increase batch size (and microbatch count) to maximize throughput.
   • If using data parallel as well, ensure your MirroredStrategy or MultiWorkerMirroredStrategy is tuned for the best allreduce approach.
   • Profilers (e.g., tf.profiler) can reveal if communication is a bottleneck.

---

5. Operational Considerations

5.1 Resource Management Without Kubernetes

• With just 3 nodes, a full-fledged Kubernetes + Kubeflow stack may be overkill.
• Simpler HPC or SSH-based orchestration is often easier for a small cluster. For example:

  # Start training on Node 1
  python train_model.py --this_node_rank=0 --cluster_spec='node0_ip,node1_ip,node2_ip'
  
  # Similarly on Node 2:
  python train_model.py --this_node_rank=1 ...

• Alternatively, you could run microk8s or k3s if you want containerization, but be aware that macOS is not the typical host OS for Kubernetes.

5.2 Monitoring & Logging

• Central Logging: If you want consolidated logs, you can set up a small solution on one node (e.g., use a remote logging service or shared aggregator).
• TensorBoard:
  • Run TensorBoard on one node and point it to logs produced by all three nodes.
  • Or each node writes logs to a network-accessible path (e.g., an NFS share or an SSHFS mount if desired).

5.3 Future Expandability

• With only 3 nodes, you might saturate the maximum performance for your experiment. If you ever add more nodes (4–8 or beyond), you’ll want to ensure the Thunderbolt network or TB-to-ethernet solution can handle additional concurrency.
• If your experiments outgrow the cluster’s memory or compute capacity, you might consider mixing in dedicated GPU servers (with Linux + NVIDIA GPUs) or cloud-based TPU/GPU nodes for specialized tasks.

---

6. Example Workflow (Conceptual)

Below is a simplified conceptual workflow with pipeline parallelism across 3 Macs using custom code in TensorFlow. Note: This is not drop-in ready but rather a high-level illustration:

1. Node 0:

   import tensorflow as tf
   from communications import send_to_node_1
   
   # Layers for sub-model 0
   inputs = tf.keras.Input(shape=(...))
   x = tf.keras.layers.Conv2D(...)(inputs)
   x = tf.keras.layers.ReLU()(x)
   # ...
   # "outputs" is the pipeline output from Node 0
   # send intermediate results to Node 1
   outputs = x
   
   while True:
       batch_data = get_next_batch()
       out = model_0(batch_data)
       send_to_node_1(out)

2. Node 1:

   import tensorflow as tf
   from communications import recv_from_node_0, send_to_node_2
   
   # Layers for sub-model 1
   x_in = tf.keras.Input(shape=(...))
   x = tf.keras.layers.Dense(...)(x_in)
   x = tf.keras.layers.ReLU()(x)
   outputs = x
   
   while True:
       in_data = recv_from_node_0()
       out = model_1(in_data)
       send_to_node_2(out)

3. Node 2:

   import tensorflow as tf
   from communications import recv_from_node_1
   
   # Layers for sub-model 2
   x_in = tf.keras.Input(shape=(...))
   x = tf.keras.layers.Dense(...)(x_in)
   final_output = tf.keras.layers.Softmax()(x)
   
   # Compute final loss, do backward pass, etc.
   while True:
       partial = recv_from_node_1()
       preds = model_2(partial)
       loss = compute_loss(preds, labels)
       # Backprop through entire pipeline requires gradient passing
       # backward_through_pipeline()
   
       # Send gradients backward or run an allreduce, etc.

• Data Flow: Node 0 processes the first chunk of layers, sends the intermediate activation to Node 1, which processes next chunk, sends output to Node 2.
• Gradient Flow: You need a strategy to propagate gradients back upstream (Node 2 → Node 1 → Node 0). Typically, frameworks with built-in pipeline parallel or model parallel solutions handle this logic automatically. In a custom approach, you must code it or rely on a library that organizes these steps.

---

7. Final Recommendations

1. Stay on Native macOS

   • Avoid virtualization if possible. Apple’s TensorFlow + Metal plugin are optimized for Apple Silicon on macOS directly.

2. Leverage TensorFlow for Data Parallel

   • For simpler scaling (especially for smaller models or a hybrid approach), you can rely on tf.distribute.MultiWorkerMirroredStrategy across the 3 nodes. This is well-documented, though it’s data parallel, not strictly model parallel.

3. Custom or Library-Based Model Parallel

   • Because model parallel features are less “out-of-the-box” in TensorFlow for Apple Silicon, consider looking at pipeline parallel or older libraries like Mesh TensorFlow. Alternatively, implement custom partitioning if you need maximum control.

4. Start with Prototypes

   • Build small, proof-of-concept pipeline parallel prototypes. Verify you can pass activations and gradients across nodes using your Thunderbolt 5 network.

5. Keep the Setup Simple

   • With only 3 nodes, a manual or SSH-based approach to launching training scripts is simpler than setting up a full Kubernetes + Kubeflow stack.
   • If you expand or need comprehensive workflow automation, you can revisit Kubeflow.

---

Conclusion

With three Apple Silicon (M4) Mac Minis connected by Thunderbolt 5, you can implement both model and data parallelism. Model parallelism will require either a specialized library or custom logic, especially on macOS, as the out-of-the-box distributed strategies in TensorFlow focus primarily on data parallel approaches. The recommended path is to use a pipeline parallel approach or consider a small-scale library that supports layer/tensor partitioning. Keep your environment consistent across all nodes, ensure the TB5 interconnect is functioning at high speed, and carefully design how intermediate activations and gradients flow among the three machines. This setup can handle large deep learning architectures that exceed the memory capacity of a single Mac Mini, while leveraging the powerful GPU/Neural Engine acceleration of Apple Silicon on each node.

Request for o1-pro
Reasoned for 2m 50s

Mapping out future scenarios

I’m exploring the design of a neural network training system on multiple upcoming Mac Minis, using Thunderbolt 5 and conventional networking, and pondering shared memory feasibility.

Focusing on network and tools

I’m generating a report on training neural networks on M4 Mac Minis, utilizing unified memory with TF or PyTorch, optimizing latency with Thunderbolt 5, and evaluating Kubernetes for cluster management.

Delving into distributed training

I’m mapping out the limitations of Apple's HPC and examining PyTorch or TensorFlow for distributed training. Also, I’m detailing cluster setup with Kubeflow on Kubernetes and overhead considerations.

Navigating unified memory

Hmm, that's interesting—exploring the practical limits of unified memory across multiple Mac Minis, I’ll present distributed training via data parallel or model parallel approaches, optimizing connectivity with Thunderbolt and Ethernet.

Mapping software solutions

I’m outlining software for distributed training, like Horovod and MPI, and comparing Kubeflow to HPC schedulers. Kubeflow excels in container-based multi-node training and full pipeline automation.

Below is a preliminary outline of how to set up a two-node (2× Mac Mini M4) cluster for model parallel (and optionally data parallel) training, connected via Thunderbolt 5. This condensed report focuses on the essentials, offering a simpler path than a larger, more complex setup.

---

1. Hardware and Network Setup

1.1 Nodes

• 2× Mac Mini M4
  • Recommended: 32 GB or 64 GB of unified memory on each (depending on the size of the models you plan to train).
  • Each with 512 GB of local SSD for storage (no shared filesystem needed).

1.2 Thunderbolt 5 Networking

• Direct Connection: With only two Macs, you can connect them directly via a Thunderbolt 5 cable.
• Throughput & Latency: TB5 provides up to 80 Gbps (or more) bandwidth, ideal for fast gradient and activation transfers.
• IP Addressing: Configure each Mac’s Thunderbolt interface so both can see each other (e.g., static IPs or DHCP if you have a TB5 dock/router).

1.3 Operating System & Software

• Native macOS: Run macOS with Apple Silicon-optimized builds (no Linux VMs unless absolutely necessary).
• TensorFlow:
  • Install the Apple Silicon build (tensorflow-macos + tensorflow-metal plugin).
  • Confirm GPU acceleration is detected by running tf.config.list_physical_devices('GPU').
• Python Environment:
  • Use a venv or Conda environment across both machines, with the same package versions.

---

2. Distributed Strategy: Model Parallel (Focus)

2.1 Rationale for Model Parallelism

• Large Models: If your model’s parameter count or memory footprint is too large for a single device’s unified memory, you can split layers or parameters across the two Mac Minis.
• More Efficient Resource Use: Each Mac’s GPU/Neural Engine is used to handle separate parts of the model simultaneously.

2.2 Pipeline Parallelism (Two Stages)

1. Node A (first stage): Input layers, early blocks of the network.
2. Node B (second stage): Remaining blocks, output layer, and final loss computation.

Note: You’ll also need to handle gradient flow back from Node B to Node A. Frameworks like PyTorch or specialized libraries can sometimes automate pipeline parallel. For TensorFlow on macOS, you may need custom logic or specialized libraries (e.g., older [Mesh TensorFlow](https://github.com/tensorflow/mesh), pipeline scripts, or manual code).

2.3 Hybrid (Model + Data Parallel)

Even with two nodes, you may want a small data parallel component. For instance, if each node has multiple GPU-like resources internally (M4 GPU + Neural Engine), you can replicate certain blocks or microbatch across them.

---

3. Communication and Framework Options

3.1 Native TensorFlow or Horovod

• Native TensorFlow (MultiWorkerMirroredStrategy / CollectiveOps):
  • Primarily geared toward data parallel training.
  • Model parallel requires manual partitioning or pipeline definitions.
• Horovod:
  • Can unify data parallel or model parallel, but you may need to build from source for Apple Silicon.
  • Over MPI or Gloo. On macOS, ensure your MPI is Apple Silicon–compatible (e.g., via conda-forge).

3.2 Custom Communication

• For pipeline parallel, you can manually pass intermediate tensors over a Python TCP connection (or with built-in TensorFlow ops if you implement them). The logic, however, is more complex compared to built-in data parallel.
• Given you only have two nodes, a point-to-point pipeline can be more straightforward (Node A → Node B, and Node B → Node A for gradients).

---

4. Example Workflow (Two-Stage Pipeline)

Below is a conceptual pipeline approach in TensorFlow. Note that this is not a turnkey solution, but a high-level demonstration:

Node A: First half of the network.

# Node A Pseudocode
import tensorflow as tf
from communication import send_to_B, recv_from_B  # (Custom or library-based)

# Define sub-model (Stage 1)
inputs = tf.keras.Input(...)
x = tf.keras.layers.Conv2D(...)(inputs)
x = tf.keras.layers.ReLU()(x)
# ... more layers ...
stage1_model = tf.keras.Model(inputs, x)

# Training loop
for batch_data, batch_labels in dataset:
    # Forward
    partial_outputs = stage1_model(batch_data)
    # Send partial outputs to Node B for next stage
    send_to_B(partial_outputs, batch_labels)

    # (Optionally) receive partial gradients from Node B
    # grads_for_stage1 = recv_from_B()
    # stage1_optimizer.apply_gradients(zip(grads_for_stage1, stage1_model.trainable_variables))

Node B: Second half of the network.

# Node B Pseudocode
import tensorflow as tf
from communication import recv_from_A, send_to_A

# Define sub-model (Stage 2)
x_in = tf.keras.Input(...)
x = tf.keras.layers.Dense(...)(x_in)
x = tf.keras.layers.Softmax()(x)
stage2_model = tf.keras.Model(x_in, x)

# Training loop
for _ in range(num_steps):
    partial_outputs, labels = recv_from_A()
    # Forward
    preds = stage2_model(partial_outputs)
    loss = custom_loss_fn(preds, labels)

    # Backprop (assuming manual gradient pass)
    with tf.GradientTape() as tape:
        ...
    grads = tape.gradient(loss, stage2_model.trainable_variables)
    stage2_optimizer.apply_gradients(zip(grads, stage2_model.trainable_variables))

    # Optionally send partial gradients back to Node A for Stage 1
    # send_to_A(grads_for_stage1)

• This naive example sends intermediate activations from Node A to Node B, then does the final forward pass and partial backward pass on Node B. A thorough pipeline parallel approach would pass partial gradients back to Node A so it can update its parameters.
• In practice, you might implement a fully automated pipeline approach or use an existing library if available.

---

5. Deployment and Orchestration

5.1 Simpler SSH-Based Approach

• With only two nodes, you can simply SSH into each machine to start the Python scripts. Use environment variables or a JSON config to tell each script which node it is (A or B).
• A typical command might look like:

  # On Node A:
  python train_pipeline.py --node_rank=0 --other_node_ip=<IP_of_B>
  
  # On Node B:
  python train_pipeline.py --node_rank=1 --other_node_ip=<IP_of_A>

5.2 Kubernetes/Kubeflow?

• Likely overkill for just two machines unless you specifically need container-based workflows, CI/CD pipelines, or advanced experiment tracking.
• If you do want some scheduling or pipeline automation, you could explore microk8s or k3s on each Mac. But for a small cluster, simpler HPC or direct control is usually easier.

---

6. Performance Tips

1. Microbatching: For pipeline parallel, feed data in smaller microbatches so Node B starts work on the first microbatch while Node A processes the next.
2. Network Profiling: Test your Thunderbolt 5 speed (e.g., iperf3) to ensure stable, high throughput.
3. Batch Size: Increase batch size if you have enough memory on each node to reduce how often you synchronize.
4. Local GPU/Neural Engine: Confirm that each stage actually runs on the Apple GPU or Neural Engine, not just the CPU. Check device placement logs or tf.config.set_visible_devices(...).

---

7. Summary and Recommendations

1. Two-Node Model Parallel:

   • Straightforward to connect the Macs via TB5.
   • You can split your network (e.g., first half on Node A, second half on Node B).
   • Implement or adapt a pipeline parallel approach to move activations and gradients across the network.

2. Simplify Deployment:

   • For only two nodes, avoid heavy orchestration frameworks.
   • Use SSH-based scripts, environment variables, or a small Python-based job launcher.

3. Check Apple Silicon–Specific Libraries:

   • Apple’s tensorflow-macos + tensorflow-metal plugin for GPU acceleration.
   • Keep an eye on any emerging libraries that better automate model parallel or pipeline parallel on Apple Silicon.

4. Validate with Small Tests:

   • Start with a toy model and confirm the pipeline logic is correct.
   • Scale up to bigger models once you trust the process.

By carefully dividing layers between the two Macs, you can create a robust pipeline parallel training environment—ideal if you’re focusing on model parallelism to handle architectures exceeding a single Mac’s memory capacity. The high-speed Thunderbolt 5 link helps reduce communication overhead, making this a feasible small-scale distributed solution for advanced deep learning tasks.

[obriensystems]

o1 pro "deep research"

write a report on how to design a system to train neural networks across mulitple apple mini computers using m4 chips. i would like to use the unified memory across multiple machines via tensorflow or pytorch for example. describe network setup that could decrease latency like using 2.5Gb/sec usbc dongles or thunderbolt 5 networking

To provide a detailed and useful report, could you clarify the following:

• Software Stack: Are you planning to use TensorFlow, PyTorch, or both? Would you prefer any specific distributed training framework like Horovod or PyTorch's DDP?
• Operating System: Will the Apple Mac Minis run macOS, or are you considering Linux via Asahi Linux for better CUDA support in the future?
• Networking Hardware: Do you have a preference for specific networking technologies, such as Thunderbolt 4/5 vs. USB-C 2.5GbE dongles, or are you open to any high-speed networking recommendations?
• Data Storage: Will each Mac Mini have its own local storage for training data, or do you want a shared storage solution such as a NAS or networked SSDs?
• Cluster Size: How many Mac Minis do you plan to use, and is scalability a key concern?
• Use Case: Are you targeting deep learning tasks like computer vision, NLP, or reinforcement learning, as different tasks may require different optimizations?

Let me know so I can tailor the report to your needs.

a distributed framwork would be better, i have a preference for thunderbolt 5. local storage or network storage is fine. for now 3 mac minis - 2 m4 pro and one m4 (optional) but concentrate on 2 m4 pros. i am targeting reinforcement learning as well as computer vision. i am sticking to apple silicon and can reformat the osx system with ubuntu if needed.

Great, I’ll put together a report on designing a distributed training system using multiple Apple Mac Mini computers with M4 Pro chips. The focus will be on:

• Setting up a distributed framework using TensorFlow or PyTorch
• Leveraging unified memory across multiple machines
• Optimizing networking performance with Thunderbolt 5
• Recommendations for local vs. network storage
• Best practices for reinforcement learning and computer vision workloads
• Configuring macOS or Ubuntu for optimal AI performance

I’ll let you know when the report is ready.

Distributed Neural Network Training on an Apple M4 Pro Mac Mini Cluster

Introduction

Building a distributed training system with multiple Apple Mac Mini computers (M4 Pro chips) can accelerate deep learning workloads for reinforcement learning (RL) and computer vision. By clustering these Apple Silicon machines, we can leverage their combined compute and memory resources to train models faster than on a single Mac. This design outlines how to set up such a cluster, covering the choice of distributed framework (TensorFlow or PyTorch), high-speed networking with Thunderbolt 5, unified memory considerations, storage strategies, OS configuration, potential bottlenecks, and a step-by-step setup guide. The goal is to maximize performance while taking advantage of the M4 Pro’s unified memory and efficient hardware for RL and vision tasks.

Distributed Framework Choices (TensorFlow vs. PyTorch)

TensorFlow MultiWorkerMirroredStrategy: TensorFlow’s tf.distribute.MultiWorkerMirroredStrategy enables synchronous training across multiple machines (workers) in parallel ([Distributed training with TensorFlow  |  TensorFlow Core](https://www.tensorflow.org/guide/distributed_training#:~:text=tf,each%20device%20across%20all%20workers)). Each worker keeps a copy of the model variables, and gradients are averaged (all-reduced) across workers each step to keep models in sync. In essence, it functions like MirroredStrategy (multi-GPU) but over networked machines ([Distributed training with TensorFlow  |  TensorFlow Core](https://www.tensorflow.org/guide/distributed_training#:~:text=tf,each%20device%20across%20all%20workers)). TensorFlow will handle communication under the hood (using RPC-based cross-device ops by default, or NCCL if GPU support is available) ([Distributed training with TensorFlow  |  TensorFlow Core](https://www.tensorflow.org/guide/distributed_training#:~:text=,them%20in%20the%20following%20way)). This strategy works on CPUs or GPUs, so it can leverage the M4 Pro’s 12-core CPU and its Metal-supported GPU. With MultiWorkerMirroredStrategy, you can use high-level Keras APIs (Model.fit) to train on your Mac Mini cluster with minimal code changes. You will need to configure the TF_CONFIG environment variable on each Mac to tell TensorFlow which machines are in the cluster and their roles (e.g. chief, workers) ([Distributed training with TensorFlow  |  TensorFlow Core](https://www.tensorflow.org/guide/distributed_training#:~:text=One%20of%20the%20key%20differences,TF_CONFIG%20section%20of%20this%20document)). TensorFlow’s built-in fault tolerance can handle worker restarts, and one “chief” node can coordinate tasks like checkpointing.

PyTorch Distributed Data Parallel (DDP): PyTorch’s DistributedDataParallel module provides a similar data-parallel approach. With DDP, you launch one training process per machine (or per device) and each process has its own copy of the model ([Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.6.0+cu124 documentation](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#:~:text=DistributedDataParallel%20,instance%20of%20DDP%20per%20process)). During training, PyTorch synchronizes gradients across processes at each iteration so that all model copies stay in sync ([Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.6.0+cu124 documentation](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#:~:text=DistributedDataParallel%20,instance%20of%20DDP%20per%20process)). This is typically done via an efficient all-reduce operation over the network. On Apple Silicon, PyTorch can utilize the GPU through the MPS backend (Metal Performance Shaders) for training ([Accelerated PyTorch Training on Mac](https://huggingface.co/docs/accelerate/en/usage_guides/mps#:~:text=With%20PyTorch%20v1,on%20Mac%20and%20MPS%20BACKEND)) ([Accelerated PyTorch Training on Mac](https://huggingface.co/docs/accelerate/en/usage_guides/mps#:~:text=1,need%20for%20additional%20local%20GPUs)). Each Mac Mini’s integrated GPU (accessible as an "mps" device in PyTorch) can be treated as one worker. PyTorch DDP will use the Gloo or MPI backend for inter-process communication on macOS (since NCCL is NVIDIA-specific). This requires initializing the process group with the IP addresses of the machines and a designated master node for coordination. DDP is powerful and fairly low-level; you might use the torchrun launch utility to easily spawn processes across the cluster. The key benefit of DDP is that it’s proven and can work across machine boundaries, effectively treating multiple Macs as one large training system ([Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.6.0+cu124 documentation](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#:~:text=DistributedDataParallel%20,instance%20of%20DDP%20per%20process)).

Framework Recommendation: Both frameworks can work, but consider your team’s familiarity. If you already use Keras/TF, MultiWorkerMirroredStrategy integrates nicely into that ecosystem. If you prefer PyTorch, DDP gives fine-grained control and is well-supported. Either way, ensure the version you install supports Apple Silicon GPU acceleration (TensorFlow requires the Apple silicon build or tensorflow-metal plugin, and PyTorch 1.12+ supports MPS ([Accelerated PyTorch Training on Mac](https://huggingface.co/docs/accelerate/en/usage_guides/mps#:~:text=With%20PyTorch%20v1,on%20Mac%20and%20MPS%20BACKEND))). Also note that Apple has an experimental distributed training project called MLX which uses MPI and Thunderbolt networking for Apple Silicon clusters ([Xgrid lives — a new project resurrects the promise of Apple's dead clustering software | AppleInsider](https://appleinsider.com/articles/24/06/17/xgrid-lives----a-new-project-resurrects-the-promise-of-apples-dead-clustering-software#:~:text=However%2C%20a%20new%20project%20called,standard%20MPI%20distributed%20computing%20methodology)), but here we focus on standard TensorFlow/PyTorch solutions for broader compatibility.

High-Speed Networking with Thunderbolt 5

Thunderbolt 5 Interconnect: To minimize communication bottlenecks, connect the Mac Minis using Thunderbolt 5 networking. Thunderbolt networking (using a Thunderbolt Bridge interface) allows direct IP communication between Macs at extremely high speeds. In fact, Thunderbolt 4 already supports up to 40 Gb/s, and Thunderbolt 5 doubles that up to 80 Gb/s bi-directional on M4 Pro/Max Macs ([M4 Mac minis in a cluster is cool, but not massively effective](https://appleinsider.com/articles/24/11/25/m4-mac-minis-in-a-computing-cluster-is-an-incredibly-cool-project-but-not-hugely-effective#:~:text=Ethernet%20can%20run%20at%201Gb%2Fs,directionally)). This far exceeds standard 1 Gb Ethernet and even 10 Gb Ethernet throughput ([M4 Mac minis in a cluster is cool, but not massively effective](https://appleinsider.com/articles/24/11/25/m4-mac-minis-in-a-computing-cluster-is-an-incredibly-cool-project-but-not-hugely-effective#:~:text=packets%20of%20data%20to%20be,sent%2C%20saving%20on%20processing%20performance)). By using Thunderbolt 5 cables between the machines, you create a low-latency, high-bandwidth cluster network specifically for training data and gradient exchange. In macOS, simply connecting two Macs with a Thunderbolt cable and enabling the Thunderbolt Bridge in Network settings will assign them IP addresses for direct communication ([M4 Mac minis in a cluster is cool, but not massively effective](https://appleinsider.com/articles/24/11/25/m4-mac-minis-in-a-computing-cluster-is-an-incredibly-cool-project-but-not-hugely-effective#:~:text=While%20typical%20home%20cluster%20computing,sent%2C%20saving%20on%20processing%20performance)). For multiple nodes, you can create a daisy chain or star topology: for example, one Mac Mini could act as a hub if it has multiple Thunderbolt ports, or you can use Thunderbolt hubs to interconnect several machines. Each Mac Mini M4 Pro typically has at least two Thunderbolt/USB4 ports; with Thunderbolt 5, you might connect up to 5-6 machines in a chain or via a host with multiple ports ([Xgrid lives — a new project resurrects the promise of Apple's dead clustering software | AppleInsider](https://appleinsider.com/articles/24/06/17/xgrid-lives----a-new-project-resurrects-the-promise-of-apples-dead-clustering-software#:~:text=has%20one%20master%20machine%2C%20and,host%20machine%20and%20the%20workers)) ([Xgrid lives — a new project resurrects the promise of Apple's dead clustering software | AppleInsider](https://appleinsider.com/articles/24/06/17/xgrid-lives----a-new-project-resurrects-the-promise-of-apples-dead-clustering-software#:~:text=And%2C%20it%20looks%20easy%20enough,if%20you%20were%20so%20inclined)).

Why Thunderbolt Networking: The higher bandwidth reduces the time spent on syncing gradients and exchanging model parameters between machines. This is crucial for synchronous training where each iteration is only as fast as the slowest network communication. Moreover, Thunderbolt’s bandwidth helps when sharing large batches of images or replay buffer data in RL across nodes. It also offers relatively low latency (Thunderbolt essentially extends the PCIe bus between machines), which can improve all-reduce efficiency. As Alex Ziskind demonstrated in a Mac Mini cluster, using Thunderbolt Bridge instead of Ethernet significantly sped up node-to-node data transfers, enabling larger packets and faster processing ([M4 Mac minis in a cluster is cool, but not massively effective](https://appleinsider.com/articles/24/11/25/m4-mac-minis-in-a-computing-cluster-is-an-incredibly-cool-project-but-not-hugely-effective#:~:text=While%20typical%20home%20cluster%20computing,sent%2C%20saving%20on%20processing%20performance)). In summary, Thunderbolt 5 provides a network on par with specialized cluster interconnects, helping the Mac Minis communicate almost as if their memory buses were connected.

Networking Setup Tips: In practice, after cabling the Macs via Thunderbolt, you should:

• Enable Thunderbolt Bridge on each macOS machine (in System Settings > Network) and ensure they have IPs on the same subnet. This might be automatic via self-assigned link-local IP, or you can set manual IPs for reliability ([Use IP over Thunderbolt to connect Mac computers - Apple Support](https://support.apple.com/guide/mac-help/ip-thunderbolt-connect-mac-computers-mchld53dd2f5/mac#:~:text=When%20you%20connect%20computers%20using,enter%20an%20IPv4%20address%20manually)) ([Use IP over Thunderbolt to connect Mac computers - Apple Support](https://support.apple.com/guide/mac-help/ip-thunderbolt-connect-mac-computers-mchld53dd2f5/mac#:~:text=3,service%2C%20then%20click%20Details)).
• Verify connectivity by pinging between the Macs over the Thunderbolt Bridge IP.
• In your distributed training code, make sure to use these Thunderbolt IP addresses (e.g., in TF_CONFIG or PyTorch’s init method) so the traffic goes over the Thunderbolt interface, not Wi-Fi or Ethernet. For example, set PyTorch’s MASTER_ADDR to the Thunderbolt IP of rank 0 node.
• Optionally, disable other network interfaces for training processes to avoid any inadvertent use of slower links.
• If Thunderbolt 5 cables or ports are limited, a 10 Gigabit Ethernet network is the next best option (Mac Mini can be configured with 10GbE); however, 10Gb is an order of magnitude slower than TB5 (10 Gb/s vs 80 Gb/s) ([M4 Mac minis in a cluster is cool, but not massively effective](https://appleinsider.com/articles/24/11/25/m4-mac-minis-in-a-computing-cluster-is-an-incredibly-cool-project-but-not-hugely-effective#:~:text=Ethernet%20can%20run%20at%201Gb%2Fs,directionally)), so expect more communication overhead in that case.

Unified Memory and Multi-Machine Considerations

Apple Unified Memory Architecture: Apple Silicon’s unified memory means the CPU, GPU, and Neural Engine on each Mac share a single memory pool. This design gives the GPU direct access to the full system memory without needing separate VRAM ([M4 Mac minis in a cluster is cool, but not massively effective](https://appleinsider.com/articles/24/11/25/m4-mac-minis-in-a-computing-cluster-is-an-incredibly-cool-project-but-not-hugely-effective#:~:text=Apple%27s%20use%20of%20Unified%20memory,configurations%20with%2032GB%20or%20more)). For example, an M4 Pro Mac with 32GB or 64GB unified memory allows the GPU to utilize nearly all of it for large tensors, which is beneficial for deep learning ([M4 Mac minis in a cluster is cool, but not massively effective](https://appleinsider.com/articles/24/11/25/m4-mac-minis-in-a-computing-cluster-is-an-incredibly-cool-project-but-not-hugely-effective#:~:text=Apple%27s%20use%20of%20Unified%20memory,configurations%20with%2032GB%20or%20more)). In training, this translates to less overhead copying data between CPU and GPU – the model and data reside in one coherent memory space. PyTorch on MPS notes that unified memory can enable training with larger batch sizes or models than a typical GPU with limited VRAM, and it reduces data transfer latency by letting the GPU access the full memory store directly ([Accelerated PyTorch Training on Mac](https://huggingface.co/docs/accelerate/en/usage_guides/mps#:~:text=1,need%20for%20additional%20local%20GPUs)).

Across Multiple Machines – Feasibility of “Unified” Memory: A common question is whether we can extend this unified memory across machines in the cluster, essentially pooling the RAM of all Mac Minis. In practice, true hardware-level unified memory does not extend beyond the boundaries of a single machine. Each Mac Mini’s memory is physically separate. There is currently no Apple Silicon equivalent of a distributed shared memory system that would let one Mac’s GPU directly address memory on another Mac. All inter-machine data sharing must happen via messaging over the network (e.g. sending tensors over Thunderbolt). Therefore, your cluster will behave in a distributed memory model – each node’s process works on local data, and explicit communication is needed to share results.

That said, a cluster can still effectively harness aggregate memory across nodes for larger workloads. For instance, if you have five Mac Minis each with 32GB RAM, the cluster has 160GB total memory. This can be useful if you partition a model or dataset that wouldn’t fit on one machine. In some scenarios (like extremely large models), you could split the model across machines (model parallelism) so that each machine holds part of the model parameters – leveraging the total memory pool of the cluster ([M4 Mac minis in a cluster is cool, but not massively effective](https://appleinsider.com/articles/24/11/25/m4-mac-minis-in-a-computing-cluster-is-an-incredibly-cool-project-but-not-hugely-effective#:~:text=using%20multiple%20high)). However, implementing model parallel training is complex and beyond standard frameworks’ out-of-the-box features. The more common data-parallel approach (via DDP or MultiWorker) assumes each model replica fits in one node’s memory. As long as your model fits in a single Mac’s RAM (which it typically will for most RL and CV models), you can use data parallelism and still benefit from unified memory on each machine independently.

Memory Sharing and Communication: Because memory isn’t physically shared, optimized communication is crucial. Frameworks like TensorFlow and PyTorch use all-reduce algorithms to combine gradients efficiently across machines ([Distributed training with TensorFlow  |  TensorFlow Core](https://www.tensorflow.org/guide/distributed_training#:~:text=,them%20in%20the%20following%20way)). These algorithms try to overlap computation with communication and use the network optimally. Over Thunderbolt 5, the high bandwidth helps mimic a quasi-shared memory environment, but some overhead remains. In reinforcement learning, besides gradients, you might also share episode experiences or parameters between actor and learner processes on different machines. For such cases, consider using efficient serialization (e.g. sending NumPy arrays or tensors directly rather than Python objects) and possibly an RPC framework or Redis queue for experience replay data. Keep an eye on memory usage on each node – unified memory means if your training uses e.g. 20GB for the model and activations, the system has less headroom for other processes or caching. It’s wise to monitor using powermetrics or Activity Monitor to ensure you’re not swapping.

In summary, unified memory is a big advantage within each Mac, allowing GPU tasks to use large memory and simplifying programming (no manual CPU-GPU data copy). Across Macs, you cannot automatically unify the memory, but with fast networking the cluster can still cooperate closely. Design your training to send only necessary data across the network (e.g. gradient vectors, model weights, or occasional validation metrics) and let each node handle local computations on its own memory to maximize throughput.

Storage Solutions: Local vs. Networked Datasets

Efficient data access is essential for high-performance training, especially for computer vision workloads that read large image datasets. There are two main approaches to storing and accessing the training data in a multi-Mac setup:

• Local Storage on Each Machine: Copy the dataset to each Mac Mini’s local SSD storage. This way, training processes read data from a fast local disk, avoiding network I/O during each training epoch. Apple Mac Minis come with fast NVMe-based SSDs which can read data at several GB/s. By having “all the data on all the nodes,” you drastically reduce latency and avoid creating a bottleneck at a single storage server (Filesystem setup for distributed multi-node LLM training
  : r/HPC). This is often the simplest and highest-performance approach for clusters up to a moderate size. Before training, you can synchronize the dataset to all machines (via scp, rsync or a shared external drive). The downside is duplicate storage space usage and the need to update each copy if data changes, but for static datasets this is usually fine. Many practitioners with multi-node setups prefer this method for its reliability and speed.

• Networked Shared Storage: Use one storage location that all machines access over the network. This could be one Mac Mini designated as an NFS/SMB file server, an external NAS device, or even a Thunderbolt-attached RAID that is shared. The advantage is having a single source of truth for the dataset and easier management (no need to copy data to each node manually). However, the network and the server’s disk speed become critical. If using a single NFS server on one Mac Mini, that Mac’s storage and network will be heavily used by all training nodes, which can become a performance bottleneck (Filesystem setup for distributed multi-node LLM training
  : r/HPC). To mitigate this, ensure you have at least a 10 Gb or Thunderbolt link to the storage server. Even then, multiple nodes hammering one disk can saturate I/O. For sequential reads (like streaming through images), an NFS might work “just fine” if each node can cache data after an initial read (Filesystem setup for distributed multi-node LLM training
  : r/HPC). But if your training involves lots of small, random reads (high IOPS), a single networked storage can slow things down unless it’s a high-end solution (e.g. an SSD RAID over a very fast interface, or a parallel file system) (Filesystem setup for distributed multi-node LLM training
  : r/HPC).

Recommendation: If the dataset is not enormous, use local copies on each Mac for maximum throughput and no network dependency during training. For example, if each Mac has a 1TB SSD and the dataset is 200GB, copying that 200GB to each machine yields the best performance. This approach was suggested in an HPC context as the best way to cut latency – if it fits on one node, put it on all nodes (Filesystem setup for distributed multi-node LLM training
: r/HPC). On the other hand, if your dataset is massive (say many terabytes, exceeding local disk capacity) or frequently updated, you may need a network storage solution. In that case, consider a fast NAS or even an Apple Xserve-style solution (though Apple doesn’t make Xserve anymore, a Mac Studio or Mac Mini with a Thunderbolt RAID could serve). Mount the network drive similarly on all Macs (using NFS over the Thunderbolt or Ethernet network). Also, leverage caching: macOS will cache recently read files in memory, and you can also manually warm up caches (e.g., have each node sequentially scan through the dataset once at the start). For RL workloads, if using a replay buffer or logging experiences, you might not need shared storage at all – those can be kept in memory or local disk on the main learner machine, since they are generated during training.

Checkpoints and Outputs: You should also decide where to save model checkpoints and training logs. With TensorFlow MultiWorkerMirroredStrategy, typically one worker is designated to write checkpoints to avoid conflicts. If using a shared filesystem, all workers can write to the same location. Otherwise, configure only the chief node to save checkpoints to its local disk and later collect them. If using PyTorch DDP, each process could save its own copy of the model (not ideal) or you can also just save on rank 0 process. It might be convenient to have a shared folder (via NFS/SMB or even an iCloud/Dropbox synced folder) for storing final outputs accessible from any machine, but during training, try to minimize writes to network storage to avoid slowdowns.

Optimizing for Apple Silicon (macOS vs. Ubuntu)

Operating System Choices: Your Mac Mini cluster can run macOS (the default) or a Linux distribution like Ubuntu via projects such as Asahi Linux. Each has pros and cons:

• macOS (native): This is the straightforward choice and is well-supported by Apple’s software stack. Apple’s Accelerate and Metal libraries are optimized for the hardware. TensorFlow can be installed on macOS (use the Apple Silicon build which leverages Metal GPU through the tensorflow-metal plugin). PyTorch on macOS uses the MPS backend, which Apple and the PyTorch team have optimized for M-series chips ([Accelerated PyTorch Training on Mac](https://huggingface.co/docs/accelerate/en/usage_guides/mps#:~:text=With%20PyTorch%20v1,on%20Mac%20and%20MPS%20BACKEND)). By running macOS, you also retain use of the 16-core Neural Engine on M4 Pro, though frameworks like PyTorch/TF do not yet automatically use the Neural Engine for training (it’s more accessible via Core ML tools for inference). macOS allows easy configuration of Thunderbolt networking and file sharing, and you can use Homebrew to install open-source tools (including OpenMPI if needed for MPI-based solutions). The downside might be that some Linux-first tools or NVIDIA-specific libraries won’t run, but for TensorFlow/PyTorch this is not an issue. Apple’s ML Compute framework (used by TensorFlow on Mac) and MPS backend will handle GPU acceleration. If using any simulation environments for RL, ensure they are Mac-compatible or use containers/virtualization if not.

• Ubuntu/Linux (Asahi Linux): Running Linux on Apple Silicon is possible through the Asahi Linux project, which by 2025 supports many features of the M-series Macs. The advantage of Linux is compatibility with a wide range of ML tools, Docker containers, and easier integration into typical server workflows. You could install Ubuntu or Arch (via Asahi) on the Mac Minis and then use standard CUDA-compatible workflows – however, note that NVIDIA CUDA is not applicable to Apple GPUs, and the open-source GPU driver for Apple Silicon is still maturing. As of now, the GPU acceleration on Asahi Linux exists but may not be as optimized as Apple’s Metal drivers on macOS. In practice, if you run Linux on M4 Pro, you might end up using CPU for training (which would be much slower) unless the GPU driver has full support for compute. Therefore, macOS is generally recommended to fully utilize the hardware. If you have a strong reason to use Linux (say for certain distributed orchestration tools or to integrate with an existing Linux cluster), be prepared for a more experimental setup and possibly limited GPU usage.

Apple Silicon Optimizations: No matter the OS, take advantage of Apple Silicon’s strengths:

• Metal Performance Shaders (MPS): Ensure your ML frameworks use the GPU. For PyTorch, check that torch.device("mps") is available and that your code moves models/tensors to MPS. The MPS backend will offload as many operations as possible to the GPU, which can significantly speed up convnets and tensor ops. The unified memory means even large image batches can be held in memory for GPU processing ([Accelerated PyTorch Training on Mac](https://huggingface.co/docs/accelerate/en/usage_guides/mps#:~:text=1,need%20for%20additional%20local%20GPUs)).
• Accelerate/BLAS: Apple’s Accelerate library provides fast linear algebra on CPU. TensorFlow and PyTorch on Mac should automatically use these for any CPU ops. Just make sure you install the frameworks’ pip wheels that are optimized for Apple Silicon (usually they are labeled as supports macOS/arm64).
• Parallelism: The M4 Pro has performance and efficiency cores. For data loading (e.g., tf.data.Dataset pipelines or PyTorch DataLoader), utilize multiple worker threads/processes to saturate the SSD and feed the GPU. macOS allows this, though keep an eye on the GIL in Python – using multiprocessing for data loading is recommended over multithreading in PyTorch DataLoader.
• Affinity: macOS doesn’t easily expose CPU pinning, but ensure that heavy background processes are minimized. You might consider disabling “App Nap” or other energy saving for these Macs when using them as servers. On Linux, you could set CPU affinity for processes if needed.

In summary, running on macOS is simpler and leverages Apple’s full software optimization for the M4 Pro. Linux on Apple Silicon could be considered for advanced users, but expect to mostly use macOS for best performance unless Linux GPU support has caught up.

Key Bottlenecks and Performance Tips

Even with a powerful setup, distributed training can face bottlenecks. Here are potential issues and how to address them:

• Inter-Node Communication Latency: Although Thunderbolt 5 provides high bandwidth, latency is still higher than within one machine’s memory. All-reduce operations (summing gradients across nodes) will incur some overhead each step. To mitigate this, use larger batch sizes per node so that you do more computation relative to communication. If using PyTorch DDP, you can overlap communication with computation (PyTorch does this automatically for gradients of different layers). Ensure the communication backend is optimized: on macOS, Gloo (which uses TCP) might be the default; if OpenMPI is installed, you could experiment with the MPI backend which may offer better performance over TB5 ([Xgrid lives — a new project resurrects the promise of Apple's dead clustering software | AppleInsider](https://appleinsider.com/articles/24/06/17/xgrid-lives----a-new-project-resurrects-the-promise-of-apples-dead-clustering-software#:~:text=However%2C%20a%20new%20project%20called,standard%20MPI%20distributed%20computing%20methodology)). Additionally, set environment variables to optimize TCP window sizes if needed for the 80Gb link, though this is an advanced tuning.

• Synchronization Overhead: Synchronous training means the slowest worker sets the pace. If one Mac is significantly slower (or handling extra tasks), it will slow down the whole cluster. It’s best if all Mac Minis in the cluster have identical specs (same M4 Pro chip, memory size, etc.). Also, monitor that each node’s GPU is being utilized fully. If one node spends extra time on data loading or is swapping to disk, that becomes a bottleneck. The training frameworks typically wait at barriers between iterations to synchronize. Using profiling tools (TensorFlow’s timeline or PyTorch’s autograd profiler with distributed profiling) can help identify if the processes are waiting on communications a lot. If so, you might try gradient compression techniques or less frequent sync (e.g., asynchronous updates, though that deviates from pure MultiWorkerMirroredStrategy/DDP).

• Memory Bandwidth and Sharing: Within each Mac, CPU and GPU share memory bandwidth. If your code simultaneously uses CPU (e.g., for preprocessing) and GPU heavily, they might compete for memory throughput. It might be beneficial to offload as much as possible to the GPU (since GPU can access all unified memory) and keep the CPU mostly for coordination and minor tasks. Alternatively, stagger workloads such that while GPU is busy computing, CPU is reading the next batch from disk, to balance usage. The unified memory eliminates copies, but total bandwidth is finite.

• Reinforcement Learning Specifics: In RL training, often there is an “actor-learner” architecture where some processes generate experiences (by running simulation or environment steps) and a central learner updates the model. In a multi-machine setup, you could dedicate one or two Mac Minis as simulators (running game environments or physics engines, possibly without heavy neural network work) and one or two as learners that perform the neural network updates. The simulators can send experience data (states, actions, rewards) to the learner machines over the network. If using such a design, ensure the data transfer of experiences is efficient (batched, compressed if possible) so it doesn’t flood the network. Alternatively, each machine could run both environment and training in parallel (for example, each Mac collecting its own data and training on it, with occasional cross-machine weight synchronization). This starts to resemble asynchronous RL algorithms (like A3C/A2C or IMPALA). The key is to prevent the environment simulation from starving the training loop or vice versa. Profile where your RL pipeline spends time (environment stepping vs model inference vs training update) and allocate resources accordingly across the cluster.

• Computer Vision Specifics: CV models (CNNs, vision transformers) tend to be data-intensive. Ensure your image decoding pipeline is optimized. It might be useful to use TensorFlow’s tf.data with prefetch and perhaps cache if dataset is small, or in PyTorch use prefetch_factor and pin_memory=False (on macOS, pin_memory is typically for CUDA, not relevant to MPS). If images are large, consider compressing them in a format that’s quick to decode (JPEG is fine since Apple hardware can accelerate JPEG decoding via frameworks like Accelerate or even the M-series media engines). If you see CPU decode becoming a bottleneck, you could offload image preprocessing to the GPU using libraries or perform simpler augmentations.

• Scalability and Diminishing Returns: Adding more nodes doesn’t always yield linear speed-ups. There is an initial benefit (for example, one experiment showed three Mac Studios in a cluster achieved ~2.9x speedup over a single Mac, not the full 3x, due to overhead ([Xgrid lives — a new project resurrects the promise of Apple's dead clustering software | AppleInsider](https://appleinsider.com/articles/24/06/17/xgrid-lives----a-new-project-resurrects-the-promise-of-apples-dead-clustering-software#:~:text=As%20with%20any%20massively%20parallel,than%20a%20single%20Mac%20Studio))). Beyond a certain number of machines, the overhead of syncing may outweigh the gains, especially if the model is small or if each node isn’t doing enough work. For an M4 Pro cluster, using 2-4 machines can significantly cut training time for large models. Going to, say, 8 machines might show smaller incremental gains unless the model and batch size are very large. Always measure and if needed, adjust the strategy (for instance, you could switch to an asynchronous parameter-server approach if scaling further, or use mixed data/model parallelism for extremely large models).

• Software Bottlenecks: Watch out for any ops that might not be optimized on Apple Silicon. While core ops are fast, some fringe operations or certain TensorFlow ops might run on CPU if not supported by Metal, which could slow that node. It’s wise to test your training script on a single Mac first and use Apple’s profiling tools (like perfetto or even Activity Monitor GPU usage) to ensure it’s utilizing the GPU as expected. If certain layers are not supported on GPU (check TensorFlow’s log or PyTorch warnings), you may want to adjust the model or update the framework if a newer version improves support.

By anticipating these bottlenecks and planning around them, you can ensure your distributed training job makes the most of the Mac Mini cluster’s capabilities. The combination of unified memory per node, fast Thunderbolt 5 networking, and optimized frameworks should handle most RL and CV tasks efficiently if configured well.

Step-by-Step Configuration Guide

Follow these steps to set up and launch a distributed training job on your Mac Mini (M4 Pro) cluster:

1. Hardware Setup and Network Configuration

• Position and Connect Macs: Place the Mac Minis in the same location (to keep Thunderbolt cables short). Connect the machines using Thunderbolt 5 cables. For two machines, a single cable between them is sufficient. For more than two, you can chain them or connect all to a central Mac if it has multiple TB ports. Ensure each connection appears as a Thunderbolt Bridge interface on macOS.
• Enable Thunderbolt Networking: On each Mac, go to System Settings > Network. If “Thunderbolt Bridge” is listed, click it; if not, add a new network service for Thunderbolt Bridge. Configure IPv4 addresses manually if the default self-assigned IPs are not in the same range. For example, set one Mac to 10.0.0.1 and the second to 10.0.0.2 with subnet mask 255.255.255.0 (no router needed for direct link). Do this for all nodes, using unique IPs.
• Test Connectivity: Open Terminal on each machine and ping the others (ping 10.0.0.2, etc.) to verify the Thunderbolt network is working. The latency should be very low (sub-millisecond or a few microseconds).
• (Optional) Internet Access: If these Macs also need internet (for installing packages), you may connect them via Wi-Fi or a secondary Ethernet, but ensure the training traffic stays on Thunderbolt. You can note the Thunderbolt interface name (e.g., thunderbolt0) and maybe force your framework to use it (PyTorch’s init_process_group allows specifying an interface for communication, or just use the TB IP addresses which achieves the same).

2. Software Installation

• Install Python and Dependencies: On each Mac, install Python 3 (if not already, though macOS comes with Python, it’s better to install a newer one via Homebrew or Miniforge for Apple Silicon). Install needed packages on each node, preferably the same versions. For TensorFlow, install tensorflow-macos and tensorflow-metal packages which provide GPU acceleration on M-series chips. For PyTorch, install the wheel with MPS support: e.g., pip install torch torchvision (version 2.0+ which includes MPS by default for macOS). Also install any other libraries (OpenAI Gym or other RL libs, OpenCV for vision, etc.) as needed.
• Install Distributed Framework Support: If using TensorFlow MultiWorkerMirroredStrategy, no extra install is needed (it uses grpc/RPC internally). If using PyTorch DDP, the torch.distributed backend is included. PyTorch’s Gloo backend works out-of-the-box on macOS. Should you prefer MPI, you can install OpenMPI via Homebrew (brew install open-mpi) and ensure PyTorch was built with MPI support (this might require a source build or a special build). For most cases, Gloo will suffice for CPU/GPU training over TCP/IP.
• Environment Setup: It’s often useful to use the same user account name on all Macs for simplicity and perhaps set up SSH keys so you can ssh between machines without password (to launch training remotely). Ensure each Mac has the same Python environment (you might create a Conda environment or a Python venv and copy it). This avoids version mismatches.

3. Data Preparation

• Dataset Distribution: Decide on local vs network storage as discussed. If local, copy your dataset to an identical path on each Mac (e.g., /Users/Shared/data/my_dataset or ~/datasets/my_dataset). Using the same path means your training script can be identical on all machines. You can use rsync -av to sync the data folder from a master node to all others. This might take time initially, but it’s a one-time cost. If using network storage, set up the share now (e.g., enable File Sharing on the main Mac and put the dataset in the shared folder, or mount your NAS to the same mount point on all machines). Test that each node can read the data. For large CV datasets, also consider converting data to a format like TFRecords or LMDB which can improve throughput.
• Checkpoint/Output Location: Choose where you will save model checkpoints. If using a shared directory (like an NFS mount or an external drive attached to one Mac and shared), ensure all nodes have write access. If not shared, plan to have only the rank 0/chief save and later collect the files manually. Set any path variables in your code accordingly.

4. Configure Training Code for Distribution

• TensorFlow MultiWorker Setup: Incorporate MultiWorkerMirroredStrategy in your training code. For example:

  strategy = tf.distribute.MultiWorkerMirroredStrategy()
  with strategy.scope():
      model = build_model(...)
      model.compile(...)
  model.fit(dataset, epochs=..., callbacks=...)

  Before running, set the TF_CONFIG environment variable on each machine to define the cluster. For instance, on machine 0 (chief) you might export:

  export TF_CONFIG='{
    "cluster": {
      "worker": ["10.0.0.1:12345", "10.0.0.2:23456", "10.0.0.3:34567"]
    },
    "task": {"type": "worker", "index": 0}
  }'

  and similarly on the others with index 1, 2, etc (using their IP and a port). TensorFlow uses this to know about all workers ([Distributed training with TensorFlow  |  TensorFlow Core](https://www.tensorflow.org/guide/distributed_training#:~:text=One%20of%20the%20key%20differences,TF_CONFIG%20section%20of%20this%20document)). Alternatively, use tf.distribute.RunConfig in TensorFlow Estimator to set cluster config. Ensure each process is launched (via ssh or a cluster manager) roughly at the same time. They will find each other and begin synced training.

• PyTorch DDP Setup: Modify your training script to use DDP. This typically means:

  • Initializing distributed: torch.distributed.init_process_group(backend='gloo', init_method='tcp://10.0.0.1:29500', world_size=N, rank=R) where N is total nodes and R is the rank of this node (0 to N-1). You can choose one node’s TB IP and an open port as the init_method.
  • Move model to MPS: model.to(torch.device("mps")).
  • Wrap model: ddp_model = torch.nn.parallel.DistributedDataParallel(model, device_ids=None) (device_ids is None for single MPS device per process).
  • Adjust your DataLoader to do distributed sampling: use torch.utils.data.distributed.DistributedSampler so each node gets a unique portion of the dataset each epoch.
  • Launch the script on each machine. You can use the torchrun launcher for convenience. For example, on the first node (rank 0) run:

    torchrun --nproc_per_node=1 --nnodes=3 --node_rank=0 --master_addr=10.0.0.1 --master_port=29500 train.py

    and similarly on the others with --node_rank=1, etc. This will handle spawning the processes and initializing the process group.
  • The script will then run in parallel and DDP will average the gradients across the three Macs each backward pass. Remember to call torch.distributed.barrier() at the end if you want to synchronize before program exit (useful for clean termination).

• Verify Device Usage: In either framework, print out the devices being used for model variables. TensorFlow will likely show something like using /device:GPU:0 (which on Mac is actually the ANE or GPU via Metal plugin) or it might list CPU if it couldn’t find GPU – if the latter, make sure the Metal plugin is installed correctly. PyTorch, you can print torch.cuda.is_available() (should be False on Mac, but torch.backends.mps.is_available() should be True) and perhaps a test tensor on mps. This step ensures that each process is indeed using the Apple GPU for training, which is important for performance.

5. Optimize and Tune

• Batch Size and Learning Rate: With N machines, you can typically increase the total batch size by roughly N if scaling linearly. For example, if one Mac handled batch 32, three Macs might handle batch 96 (32 each). You might need to adjust learning rate accordingly (linear scaling rule as a starting point). Ensure each node’s DataLoader is feeding the correct portion of data (in PyTorch, DistributedSampler does this; in TF, the dataset shard can be split by worker index using tf.data.Dataset.shard if needed).
• Monitor Performance: Start the training and monitor resource usage. On each Mac, use Activity Monitor or top to see CPU usage, and the Metal Performance HUD (available in Xcode’s Instruments) or powermetrics to observe GPU utilization. Ideally, the GPU on each Mac should be near 100% busy during training, and network utilization on Thunderbolt should also be significant but not saturated constantly. If you see GPUs frequently idling, the bottleneck might be data loading or communication. In that case, tweak the number of data loader workers, or check if the network link is maxed out (using netstat or Activity Monitor’s network tab).
• Debugging: If the processes hang or fail to find each other, double-check firewall settings (macOS might block incoming connections for Python – you can allow these in Security & Privacy settings). Ensure that the MASTER_ADDR and ports are consistent. For TensorFlow, misconfiguration of TF_CONFIG is a common issue – all nodes must have identical cluster spec and the correct index. Use simple tests (like a dummy all-reduce) to debug the distributed setup before launching a long training. Both TF and PyTorch have debugging flags (e.g., NCCL_DEBUG isn’t relevant here, but GLOO_SOCKET_IFNAME can force a network interface for PyTorch if needed).

6. Reinforcement Learning Configuration (if applicable)

• If your workload is RL with separate actor processes, you might run those as separate scripts. For example, you could have one Mac run environment simulators that send experience via network to the learner. Ensure those connections (could be a simple socket or using an RL library’s mechanism) are configured to use the Thunderbolt IPs as well. This might involve running a small server on the learner machine to collect experiences. The training steps would proceed as usual but now you have asynchronous data coming in. This setup can be complex, so attempt it after verifying the basic distributed training works. Alternatively, run self-play or environment and model on each node and periodically average model parameters across nodes (an approach some RL algorithms use) – this can be done by calling DDP-style communication every few updates instead of every step, to reduce overhead.

7. Start Training

• Launch the training job across all machines. If using SSH, you can script this so one command triggers the script on all nodes (for example, using a simple pdsh or parallel-ssh, or even an Apple Remote Desktop script). Make sure all processes start around the same time. Watch the logs on each machine. You should see indications that they are communicating (e.g., in TensorFlow, each worker will report joining the cluster; in PyTorch, you may see logs from each rank).
• Let it run for a short while and ensure that you get periodic training updates (like loss decreasing) and that it’s faster than using a single machine. If you have a baseline for single-machine throughput (say images per second or iters per second), compare against the multi-node to confirm you’re getting a speed boost. It might not be exactly N times faster for N nodes due to overhead, but it should be significantly faster than one node.

8. Post-Training

• Once training is complete (or if you stop it manually), gather the results. If checkpoints were saved on a single node’s disk, copy them to a central location if needed. If each node saved something (like different shards of data), consolidate as necessary. Also, note any improvements for the next run: e.g., maybe you found that increasing the batch size further gave better GPU utilization – you can apply that in future runs.
• Maintain the cluster setup for reuse: since you’ve configured networking and installed everything, subsequent training jobs will be easier. It might be useful to create a shared script or Makefile that sets environment vars and launches the distributed job on all machines, to simplify running new experiments.

By following this guide, you will have a robust distributed training environment spanning multiple Apple Silicon Mac Minis. Such a setup, with high-speed Thunderbolt 5 links and efficient use of unified memory on each node, is well-suited for cutting-edge reinforcement learning experiments and computer vision model training. It combines the ease-of-use of desktop Macs with near-HPC levels of networking performance, enabling practical at-home or lab clusters for deep learning research.

Conclusion and Recommendations

In designing this multi-Mac Mini training system, we addressed the critical aspects: choosing a suitable distributed training framework, networking the machines for maximum throughput, understanding the limits of unified memory sharing, managing data storage, configuring the software for Apple Silicon, and mitigating bottlenecks. Here are the key takeaways and best practices from the design:

• Use synchronous data parallel training (TensorFlow MultiWorkerMirroredStrategy or PyTorch DDP) to let each Mac train on a portion of data and keep models synchronized ([Distributed training with TensorFlow  |  TensorFlow Core](https://www.tensorflow.org/guide/distributed_training#:~:text=tf,each%20device%20across%20all%20workers)) ([Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.6.0+cu124 documentation](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#:~:text=DistributedDataParallel%20,instance%20of%20DDP%20per%20process)). This approach is effective for both RL and CV workloads when scaling out to multiple machines.
• Leverage Thunderbolt 5 networking to connect the Macs, as it offers up to 80 Gb/s communication – crucial for fast gradient exchanges ([M4 Mac minis in a cluster is cool, but not massively effective](https://appleinsider.com/articles/24/11/25/m4-mac-minis-in-a-computing-cluster-is-an-incredibly-cool-project-but-not-hugely-effective#:~:text=Ethernet%20can%20run%20at%201Gb%2Fs,directionally)). Configure Thunderbolt Bridge on macOS for a plug-and-play high-speed cluster network.
• Unified memory on Apple Silicon is a boon for single-machine performance ([Accelerated PyTorch Training on Mac](https://huggingface.co/docs/accelerate/en/usage_guides/mps#:~:text=1,need%20for%20additional%20local%20GPUs)), but it cannot be pooled across machines. Each node will use its own unified memory, so design the training to exchange necessary data over the network rather than expecting a global shared memory.
• Prefer local storage of datasets on each machine for speed, if storage capacity allows, to avoid I/O bottlenecks (Filesystem setup for distributed multi-node LLM training
  : r/HPC). If using network storage, invest in fast disks and interfaces, and possibly consider parallel file systems if scaling larger.
• Optimize for Apple Silicon by using macOS and the Metal/MPS backend. Ensure all training processes utilize the GPU cores of the M4 Pro. Keep software updated – Apple and community improvements (like further Metal optimizations, or new versions of ML frameworks) will continue to improve performance on this hardware.
• Tackle bottlenecks by tuning batch sizes, using efficient communication strategies, and balancing workload between CPU, GPU, and network. High-bandwidth networking and proper data pipeline design will help keep the cluster running at full capacity.
• Follow the step-by-step setup to configure networking, software, and launch training jobs. Consistency across the machines (same package versions, same paths, etc.) will save you headaches in a distributed setup.

With these considerations, a cluster of Apple M4 Pro Mac Minis can be a powerful platform for deep learning. It offers an energy-efficient and compact alternative to traditional GPU servers, while still achieving strong performance for parallelizable workloads. As demonstrated in recent experiments, such clusters are feasible and can speed up tasks like large model training or large-scale RL simulations, though their efficiency shines best when high-end interconnects (Thunderbolt) and optimized software are used ([Xgrid lives — a new project resurrects the promise of Apple's dead clustering software | AppleInsider](https://appleinsider.com/articles/24/06/17/xgrid-lives----a-new-project-resurrects-the-promise-of-apples-dead-clustering-software#:~:text=As%20with%20any%20massively%20parallel,than%20a%20single%20Mac%20Studio)) ([M4 Mac minis in a cluster is cool, but not massively effective](https://appleinsider.com/articles/24/11/25/m4-mac-minis-in-a-computing-cluster-is-an-incredibly-cool-project-but-not-hugely-effective#:~:text=Where%20a%20cluster%20like%20this,for%20the%20model%20to%20use)). By implementing the recommendations in this design, you’ll set up a capable distributed training environment ready to tackle complex reinforcement learning challenges and computer vision models, all by harnessing the combined power of Apple Silicon.

[obriensystems]

pip3 install -U transformers
pip3 install -U tensorflow
michaelobrien@mbp8 distributed_ml % python3 tensorflow_distributed.py

gpu missing
1058  brew install pyenv
 1059  pyenv install --list
 1060  pyenv install 3.9.6
 1062  pyenv global 3.9.6
 1065  eval "$(pyenv init --path)"
 1066  python --version
 1067  vi ~/.bash_profile

forgot
https://developer.apple.com/metal/tensorflow-plugin/
tensorflow-metal

python -m pip install tensorflow-metal
python -m pip install tensorflow

michaelobrien@mbp8 src % python tflow.py                 
2025-02-10 17:46:55.035029: I metal_plugin/src/device/metal_device.cc:1154] Metal device set to: Apple M4 Max
2025-02-10 17:46:55.035063: I metal_plugin/src/device/metal_device.cc:296] systemMemory: 48.00 GB
2025-02-10 17:46:55.035067: I metal_plugin/src/device/metal_device.cc:313] maxCacheSize: 18.00 GB
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1739227615.035114 16017457 pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
I0000 00:00:1739227615.035165 16017457 pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2025-02-10 17:46:56.683039: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:117] Plugin optimizer for device_type GPU is enabled.
Epoch 1/25

[obriensystems]

full install on clean M4pro

   48  brew install pyenv
   49  pyenv install --list
   50  pyenv install 3.9.6
   51  vi ~/.bash_profile
   52  source ~/.bash_profile
   53  pip install -U transformers
   54  python -m pip install -U transformers
   55  python --version
   56  pyenv global 3.9.6
   57  python -m pip install -U transformers
   58  python -m pip install -U torch
   59  python -m pip install -U accelerate
   60  python -m pip install -U tensorflow
   61  python -m pip install -U tensorflow-metal
   62  python tensorflow_distributed.py

# thunderbolt networking
export TF_CONFIG='{
  "cluster": {
    "worker": ["169.254.135.233:12345", "169.254.79.20:23456"]
  },
  "task": {"type": "worker", "index": 0}
}'
michaelobrien@mbp8 distributed_ml %

master

michaelobrien@mbp8 distributed_ml % export TF_CONFIG='{                    
  "cluster": {
    "worker": ["169.254.135.233:12345", "169.254.79.20:23456"]
  },
  "task": {"type": "worker", "index": 0}
}'
michaelobrien@mbp8 distributed_ml % python tensorflow_distributed.py
2025-02-10 18:46:14.292976: I metal_plugin/src/device/metal_device.cc:1154] Metal device set to: Apple M4 Max
2025-02-10 18:46:14.293001: I metal_plugin/src/device/metal_device.cc:296] systemMemory: 48.00 GB
2025-02-10 18:46:14.293005: I metal_plugin/src/device/metal_device.cc:313] maxCacheSize: 18.00 GB
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1739231174.293023 16092362 pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
I0000 00:00:1739231174.293047 16092362 pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
I0000 00:00:1739231174.293942 16092362 pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
I0000 00:00:1739231174.293951 16092362 pluggable_device_factory.cc:271] Created TensorFlow device (/job:worker/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2025-02-10 18:46:14.294930: I external/local_xla/xla/tsl/distributed_runtime/coordination/coordination_service.cc:447] Initializing CoordinationService
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1739231174.295181 16092362 grpc_server_lib.cc:464] Started server with target: grpc://169.254.135.233:12345
2025-02-10 18:46:14.297820: I external/local_xla/xla/tsl/distributed_runtime/coordination/coordination_service.cc:700] /job:worker/replica:0/task:0 has connected to coordination service. Incarnation: 12254344858638549811
2025-02-10 18:46:14.297834: I external/local_xla/xla/tsl/distributed_runtime/coordination/coordination_service.cc:653] Waiting for 1/2 tasks to connect.
2025-02-10 18:46:14.297837: I external/local_xla/xla/tsl/distributed_runtime/coordination/coordination_service.cc:656] Example stragglers:
/job:worker/replica:0/task:1
I0000 00:00:1739231174.297927 16092362 coordination_service_agent.cc:340] Coordination agent has successfully connected.
2025-02-10 18:46:21.432296: I external/local_xla/xla/tsl/distributed_runtime/coordination/coordination_service.cc:700] /job:worker/replica:0/task:1 has connected to coordination service. Incarnation: 14268250196240035111
2025-02-10 18:46:21.432485: I external/local_xla/xla/tsl/distributed_runtime/coordination/coordination_service.cc:653] Waiting for 0/2 tasks to connect.
mirrored_strategy:  <tensorflow.python.distribute.collective_all_reduce_strategy.CollectiveAllReduceStrategy object at 0x102863fd0>
2025-02-10 18:46:23.194642: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:117] Plugin optimizer for device_type GPU is enabled.
2025-02-10 18:46:23.435993: W tensorflow/core/framework/op_kernel.cc:1805] INTERNAL: Failed to build OpKernel for Add : No registered 'Add' OpKernel for 'GPU' devices compatible with node {{node Add}}
	 (OpKernel was found, but attributes didn't match) Requested Attributes: T=DT_INT64, _device="/job:worker/replica:0/task:0/device:GPU:0"
	.  Registered:  device='XLA_CPU_JIT'; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BFLOAT16, DT_COMPLEX128, DT_HALF]
  device='GPU'; T in [DT_FLOAT]
  device='GPU'; T in [DT_HALF]
  device='GPU'; T in [DT_BFLOAT16]
  device='DEFAULT'; T in [DT_INT32]
  device='CPU'; T in [DT_FLOAT]
  device='CPU'; T in [DT_HALF]
  device='CPU'; T in [DT_DOUBLE]
  device='CPU'; T in [DT_INT32]
  device='CPU'; T in [DT_INT64]
  device='CPU'; T in [DT_BFLOAT16]
  device='CPU'; T in [DT_INT8]
  device='CPU'; T in [DT_INT16]
  device='CPU'; T in [DT_COMPLEX64]
  device='CPU'; T in [DT_UINT8]
  device='CPU'; T in [DT_COMPLEX128]
  device='CPU'; T in [DT_STRING]

2025-02-10 18:46:23.436056: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: INTERNAL: Failed to build OpKernel for Add : No registered 'Add' OpKernel for 'GPU' devices compatible with node {{node Add}}
	 (OpKernel was found, but attributes didn't match) Requested Attributes: T=DT_INT64, _device="/job:worker/replica:0/task:0/device:GPU:0"
	.  Registered:  device='XLA_CPU_JIT'; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BFLOAT16, DT_COMPLEX128, DT_HALF]
  device='GPU'; T in [DT_FLOAT]
  device='GPU'; T in [DT_HALF]
  device='GPU'; T in [DT_BFLOAT16]
  device='DEFAULT'; T in [DT_INT32]
  device='CPU'; T in [DT_FLOAT]
  device='CPU'; T in [DT_HALF]
  device='CPU'; T in [DT_DOUBLE]
  device='CPU'; T in [DT_INT32]
  device='CPU'; T in [DT_INT64]
  device='CPU'; T in [DT_BFLOAT16]
  device='CPU'; T in [DT_INT8]
  device='CPU'; T in [DT_INT16]
  device='CPU'; T in [DT_COMPLEX64]
  device='CPU'; T in [DT_UINT8]
  device='CPU'; T in [DT_COMPLEX128]
  device='CPU'; T in [DT_STRING]

2025-02-10 18:46:23.436094: I tensorflow/core/framework/local_rendezvous.cc:428] Local rendezvous send item cancelled. Key hash: 1357961346763198188
2025-02-10 18:46:23.436099: I tensorflow/core/framework/local_rendezvous.cc:428] Local rendezvous send item cancelled. Key hash: 10823327942303459423
2025-02-10 18:46:23.436102: I tensorflow/core/framework/local_rendezvous.cc:428] Local rendezvous send item cancelled. Key hash: 15603959714220415293
Traceback (most recent call last):
  File "/Users/michaelobrien/wse_github/ObrienlabsDev/machine-learning/src/distributed_ml/tensorflow_distributed.py", line 86, in <module>
    parallel_model.fit(x_train, y_train, epochs=20, batch_size=2048)#5120)#7168)#7168)
  File "/Users/michaelobrien/.pyenv/versions/3.9.6/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py", line 122, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/Users/michaelobrien/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/michaelobrien/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/Users/michaelobrien/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Users/michaelobrien/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
tensorflow.python.framework.errors_impl.InternalError: {{function_node __wrapped__CollectiveReduceV2_Nordering_token_1_device_/job:worker/replica:0/task:0/device:GPU:0}} Failed to build OpKernel for Add : No registered 'Add' OpKernel for 'GPU' devices compatible with node {{node Add}}
	 (OpKernel was found, but attributes didn't match) Requested Attributes: T=DT_INT64, _device="/job:worker/replica:0/task:0/device:GPU:0"
	.  Registered:  device='XLA_CPU_JIT'; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BFLOAT16, DT_COMPLEX128, DT_HALF]
  device='GPU'; T in [DT_FLOAT]
  device='GPU'; T in [DT_HALF]
  device='GPU'; T in [DT_BFLOAT16]
  device='DEFAULT'; T in [DT_INT32]
  device='CPU'; T in [DT_FLOAT]
  device='CPU'; T in [DT_HALF]
  device='CPU'; T in [DT_DOUBLE]
  device='CPU'; T in [DT_INT32]
  device='CPU'; T in [DT_INT64]
  device='CPU'; T in [DT_BFLOAT16]
  device='CPU'; T in [DT_INT8]
  device='CPU'; T in [DT_INT16]
  device='CPU'; T in [DT_COMPLEX64]
  device='CPU'; T in [DT_UINT8]
  device='CPU'; T in [DT_COMPLEX128]
  device='CPU'; T in [DT_STRING]
 [Op:CollectiveReduceV2] name: 

slave

michaelobrien@mini10 distributed_ml % export TF_CONFIG='{             
  "cluster": {
    "worker": ["169.254.135.233:12345", "169.254.79.20:23456"]
  },
  "task": {"type": "worker", "index": 1}
}'
michaelobrien@mini10 distributed_ml % python tensorflow_distributed.py
2025-02-10 18:46:21.339973: I metal_plugin/src/device/metal_device.cc:1154] Metal device set to: Apple M4 Pro
2025-02-10 18:46:21.339999: I metal_plugin/src/device/metal_device.cc:296] systemMemory: 24.00 GB
2025-02-10 18:46:21.340004: I metal_plugin/src/device/metal_device.cc:313] maxCacheSize: 8.00 GB
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1739231181.340024 3728033 pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
I0000 00:00:1739231181.340050 3728033 pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
I0000 00:00:1739231181.340913 3728033 pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
I0000 00:00:1739231181.340923 3728033 pluggable_device_factory.cc:271] Created TensorFlow device (/job:worker/replica:0/task:1/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1739231181.342719 3728033 grpc_server_lib.cc:464] Started server with target: grpc://169.254.79.20:23456
I0000 00:00:1739231181.348721 3728033 coordination_service_agent.cc:340] Coordination agent has successfully connected.
mirrored_strategy:  <tensorflow.python.distribute.collective_all_reduce_strategy.CollectiveAllReduceStrategy object at 0x100bddfd0>
2025-02-10 18:46:23.111351: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:117] Plugin optimizer for device_type GPU is enabled.
2025-02-10 18:46:23.380221: W tensorflow/core/framework/op_kernel.cc:1805] INTERNAL: Failed to build OpKernel for Add : No registered 'Add' OpKernel for 'GPU' devices compatible with node {{node Add}}
	 (OpKernel was found, but attributes didn't match) Requested Attributes: T=DT_INT64, _device="/job:worker/replica:0/task:1/device:GPU:0"
	.  Registered:  device='XLA_CPU_JIT'; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BFLOAT16, DT_COMPLEX128, DT_HALF]
  device='GPU'; T in [DT_FLOAT]
  device='GPU'; T in [DT_HALF]
  device='GPU'; T in [DT_BFLOAT16]
  device='DEFAULT'; T in [DT_INT32]
  device='CPU'; T in [DT_FLOAT]
  device='CPU'; T in [DT_HALF]
  device='CPU'; T in [DT_DOUBLE]
  device='CPU'; T in [DT_INT32]
  device='CPU'; T in [DT_INT64]
  device='CPU'; T in [DT_BFLOAT16]
  device='CPU'; T in [DT_INT8]
  device='CPU'; T in [DT_INT16]
  device='CPU'; T in [DT_COMPLEX64]
  device='CPU'; T in [DT_UINT8]
  device='CPU'; T in [DT_COMPLEX128]
  device='CPU'; T in [DT_STRING]

2025-02-10 18:46:23.380281: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous is aborting with status: INTERNAL: Failed to build OpKernel for Add : No registered 'Add' OpKernel for 'GPU' devices compatible with node {{node Add}}
	 (OpKernel was found, but attributes didn't match) Requested Attributes: T=DT_INT64, _device="/job:worker/replica:0/task:1/device:GPU:0"
	.  Registered:  device='XLA_CPU_JIT'; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BFLOAT16, DT_COMPLEX128, DT_HALF]
  device='GPU'; T in [DT_FLOAT]
  device='GPU'; T in [DT_HALF]
  device='GPU'; T in [DT_BFLOAT16]
  device='DEFAULT'; T in [DT_INT32]
  device='CPU'; T in [DT_FLOAT]
  device='CPU'; T in [DT_HALF]
  device='CPU'; T in [DT_DOUBLE]
  device='CPU'; T in [DT_INT32]
  device='CPU'; T in [DT_INT64]
  device='CPU'; T in [DT_BFLOAT16]
  device='CPU'; T in [DT_INT8]
  device='CPU'; T in [DT_INT16]
  device='CPU'; T in [DT_COMPLEX64]
  device='CPU'; T in [DT_UINT8]
  device='CPU'; T in [DT_COMPLEX128]
  device='CPU'; T in [DT_STRING]

2025-02-10 18:46:23.380312: I tensorflow/core/framework/local_rendezvous.cc:428] Local rendezvous send item cancelled. Key hash: 13201734406611989596
2025-02-10 18:46:23.380316: I tensorflow/core/framework/local_rendezvous.cc:428] Local rendezvous send item cancelled. Key hash: 4886782031629155724
2025-02-10 18:46:23.380318: I tensorflow/core/framework/local_rendezvous.cc:428] Local rendezvous send item cancelled. Key hash: 3065626177187131079
Traceback (most recent call last):
  File "/Users/michaelobrien/wse_github/ObrienlabsDev/machine-learning/src/distributed_ml/tensorflow_distributed.py", line 86, in <module>
    parallel_model.fit(x_train, y_train, epochs=20, batch_size=2048)#5120)#7168)#7168)
  File "/Users/michaelobrien/.pyenv/versions/3.9.6/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py", line 122, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/Users/michaelobrien/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/michaelobrien/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/Users/michaelobrien/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Users/michaelobrien/.pyenv/versions/3.9.6/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
tensorflow.python.framework.errors_impl.InternalError: {{function_node __wrapped__CollectiveReduceV2_Nordering_token_1_device_/job:worker/replica:0/task:1/device:GPU:0}} Failed to build OpKernel for Add : No registered 'Add' OpKernel for 'GPU' devices compatible with node {{node Add}}
	 (OpKernel was found, but attributes didn't match) Requested Attributes: T=DT_INT64, _device="/job:worker/replica:0/task:1/device:GPU:0"
	.  Registered:  device='XLA_CPU_JIT'; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_COMPLEX64, DT_INT64, DT_BFLOAT16, DT_COMPLEX128, DT_HALF]
  device='GPU'; T in [DT_FLOAT]
  device='GPU'; T in [DT_HALF]
  device='GPU'; T in [DT_BFLOAT16]
  device='DEFAULT'; T in [DT_INT32]
  device='CPU'; T in [DT_FLOAT]
  device='CPU'; T in [DT_HALF]
  device='CPU'; T in [DT_DOUBLE]
  device='CPU'; T in [DT_INT32]
  device='CPU'; T in [DT_INT64]
  device='CPU'; T in [DT_BFLOAT16]
  device='CPU'; T in [DT_INT8]
  device='CPU'; T in [DT_INT16]
  device='CPU'; T in [DT_COMPLEX64]
  device='CPU'; T in [DT_UINT8]
  device='CPU'; T in [DT_COMPLEX128]
  device='CPU'; T in [DT_STRING]
 [Op:CollectiveReduceV2] name: 
michaelobrien@mini10 distributed_ml % 

[obriensystems]

see tensorflow/tensorflow#61314