fix dependecies

Author: Jackmin801

README.md

@@ -30,10 +30,28 @@ source .venv/bin/activate
 
 Install python dependencies:
 ```bash
+# Hivemind
+cd hivemind_source
+pip install .
+cp build/lib/hivemind/proto/* hivemind/proto/.
+pip install -e ".[all]"
+cd ..
+# Requirements
 pip install -r requirements.txt
+# Others
+pip install --pre torchdata --index-url https://download.pytorch.org/whl/nightly/cpu
+pip install -e ./pydantic_config
+# OpenDiLoCo
 pip install .
 ```
 
+Optionally, you can install flash-attn to use Flash Attention 2.
+This requires your system to have cuda compiler set up.
+```
+# (Optional) flash-attn
+pip install flash-attn==2.5.8
+```
+
 ## Docker container
 
 If you prefer to run your experiments in a reproduceable container, you can use our pre-built docker image containing the repository and pre-installed dependencies.
@@ -86,13 +104,13 @@ The [multiaddress](https://github.com/multiformats/multiaddr) strings listed aft
 
 ## Stopping hivemind runs
 
-The current implementation of hivemind doesnt handle Ctrl+C keyboard interrupt well. You can stop the runs using `pkill`:
+The current implementation of hivemind doesn't handle Ctrl+C keyboard interrupt well. You can stop the runs using `pkill`:
 ```bash
 pkill -f torchrun
 ```
 
 ## Resuming from checkpoint
-To resume from checkpoint, you can pass the `--resume-from-checkpoint` argment to the training script. e.g.
+To resume from checkpoint, you can pass the `--resume-from-checkpoint` argument to the training script. e.g.
 ```bash
 torchrun --nproc_per_node=8 \
     train_fsdp.py \

requirements.txt

@@ -4,4 +4,3 @@ wandb==0.16.4
 cyclopts>=2.6.1
 fsspec[gcs]>=2024.3.1
 torch==2.3.1
-flash-attn