Skip to content

Commit 46b432f

Browse files
authored
Add MoE FAQ about --clone-tensors option. (#296)
Add this answer from this issue to a FAQ about MoE: #279
1 parent 8d689e4 commit 46b432f

File tree

1 file changed

+9
-0
lines changed

1 file changed

+9
-0
lines changed

docs/moe.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,3 +113,12 @@ experts:
113113
- "<|im_start|>user\nWie geht es dir?<|im_end|>"
114114
- "Das ist ein Satz auf Deutsch."
115115
```
116+
117+
## FAQ
118+
119+
### What does the "Your model has duplicated tensors but the --clone-tensors flag is not set" warning mean?
120+
121+
Answer from [Charles O. Goddard (cg123)](https://github.com/cg123)
122+
(also see [this GitHub issue](https://github.com/arcee-ai/mergekit/issues/279#issuecomment-2081818104)):
123+
124+
> This is completely benign. This happens when a single tensor from a model is used in multiple places, like when doing sparse upcycling with the moe script or doing passthrough merges that repeat layers. Having `--clone-tensors` set can use slightly more memory, but having it unset will slow down saving and introduce small memory usage spikes in cases where this warning occurs. It's honestly a small enough difference that the warning could be removed entirely.

0 commit comments

Comments
 (0)