You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is splitted between differents gpus.
1077
+
Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is splitted between different gpus.
1078
1078
You can know which layers mapped to which device by checking the associated device_map: `model.hf_device_map`.
1079
1079
1080
1080
@@ -1267,7 +1267,7 @@ class SlidingWindowCache(StaticCache):
1267
1267
dtype (`torch.dtype`, *optional*, defaults to `torch.float32`):
1268
1268
The default `dtype` to use when initializing the layer.
Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is splitted between differents gpus.
1270
+
Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is splitted between different gpus.
1271
1271
You can know which layers mapped to which device by checking the associated device_map: `model.hf_device_map`.
1272
1272
1273
1273
Example:
@@ -1579,7 +1579,7 @@ class HybridCache(Cache):
1579
1579
dtype (torch.dtype, *optional*, defaults to `torch.float32`):
1580
1580
The default `dtype` to use when initializing the layer.
Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is splitted between differents gpus.
1582
+
Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is splitted between different gpus.
1583
1583
You can know which layers mapped to which device by checking the associated device_map: `model.hf_device_map`.
1584
1584
1585
1585
Example:
@@ -1929,7 +1929,7 @@ class OffloadedStaticCache(StaticCache):
1929
1929
offload_device (`Union[str, torch.device]`, *optional*, defaults to `cpu`):
Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is splitted between differents gpus.
1932
+
Mapping between the layers and its device. This is required when you are manually initializing the cache and the model is splitted between different gpus.
1933
1933
You can know which layers mapped to which device by checking the associated device_map: `model.hf_device_map`.
0 commit comments