Skip to content

Commit 362d113

Browse files
update image size
1 parent 5663536 commit 362d113

File tree

1 file changed

+1
-0
lines changed
  • cloud-infrastructure/ai-infra-gpu/ai-infrastructure/dstack

1 file changed

+1
-0
lines changed

cloud-infrastructure/ai-infra-gpu/ai-infrastructure/dstack/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,7 @@ fsdp_config:
122122
* `fsdp_config`: These set up how FSDP operates, such as how the model is sharded (`fsdp_sharding_strategy`) and whether parameters are offloaded to CPU (`fsdp_offload_params`).
123123

124124
![Hybrid shards](assets/images/image2.png "Hybrid shards")
125+
<img src="https://github.com/oracle-devrel/technology-engineering/blob/dstack-tutorial/cloud-infrastructure/ai-infra-gpu/ai-infrastructure/dstack/assets/images/image1.png" width="400">
125126

126127
With the `FSDP` of `distributed_type` and `FULL_SHARD` of `fsdp_config`’s `fsdp_sharding_strategy`, a model will be sharded across multiple GPUs in a single machine. When dealing with multiple compute nodes, each node will host an identical copy of the model, which is itself split across multiple GPUs within that node. This means each partitioned model instance on each node processes different sections or batches of your dataset. To distribute a single model across multiple GPUs spanning across multiple nodes, configure the parameter `fsdp_sharding_strategy` as `HYBRID_SHARD`.
127128

0 commit comments

Comments
 (0)