Commit 971f245
authored
llama : recognize IBM Granite 3.3 FIM tokens (ggml-org#12988)
The Granite's FIM tokens are very similar to Qwen's; it's just that
they use underscore instead of a dash. So <fim_middle> for example
instead of <fim-middle>.
Opening up tokenizer_config.json in ibm-granite/granite-3.3-8b-base
shows:
```
"<fim_prefix>",
"<fim_middle>",
"<fim_suffix>",
"<fim_pad>",
...
"<reponame>",
```1 parent 12b1750 commit 971f245
1 file changed
+5
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1841 | 1841 | | |
1842 | 1842 | | |
1843 | 1843 | | |
| 1844 | + | |
1844 | 1845 | | |
1845 | 1846 | | |
1846 | 1847 | | |
| |||
1859 | 1860 | | |
1860 | 1861 | | |
1861 | 1862 | | |
| 1863 | + | |
1862 | 1864 | | |
1863 | 1865 | | |
1864 | 1866 | | |
| |||
1877 | 1879 | | |
1878 | 1880 | | |
1879 | 1881 | | |
| 1882 | + | |
1880 | 1883 | | |
1881 | 1884 | | |
1882 | 1885 | | |
| |||
1895 | 1898 | | |
1896 | 1899 | | |
1897 | 1900 | | |
| 1901 | + | |
1898 | 1902 | | |
1899 | 1903 | | |
1900 | 1904 | | |
| |||
1913 | 1917 | | |
1914 | 1918 | | |
1915 | 1919 | | |
| 1920 | + | |
1916 | 1921 | | |
1917 | 1922 | | |
1918 | 1923 | | |
| |||
0 commit comments