Expose seperator token

The seperator token is required in bert style rerankers, normally cls token is also required for rerankers but is currently deprecated in llama.cpp (as it is the same as the bos token for a model), and thus not necesary.

Author: emilnorsker

llama-cpp-2/src/model.rs

@@ -185,6 +185,13 @@ impl LlamaModel {
         LlamaToken(token)
     }
 
+    /// Get the separator token (SEP).
+    #[must_use]
+    pub fn token_sep(&self) -> LlamaToken {
+        let token = unsafe { llama_cpp_sys_2::llama_vocab_sep(self.vocab_ptr()) };
+        LlamaToken(token)
+    }
+
     /// Convert single token to a string.
     ///
     /// # Errors