You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/xet/xorb.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -111,7 +111,7 @@ Note that a Xorb MAY contain chunks that utilize different compression schemes.
111
111
2.**Best Effort Prediction**
112
112
113
113
In `xet-core`, to predict if BG4 will be useful we maximum KL divergence between the distribution of per-byte pop-counts on a sample of each of the 4 groups that would be formed.
114
-
You can read more about it in [bg4_prediction.rs](./cas_object/src/byte_grouping/bg4_prediction.rs) and accompanying scripts.
114
+
You can read more about it in [bg4_prediction.rs](https://github.com/huggingface/xet-core/blob/main/cas_object/src/byte_grouping/bg4_prediction.rs) and accompanying scripts.
115
115
116
116
If the predictor does not show that BG4 will be better, we use Lz4 and in either case we will store the chunk as the uncompressed version if the compression scheme used does not show any benefit.
0 commit comments