llama_tokenizer Struct Reference

Public Member Functions
	llama_tokenizer (const llama_vocab &vocab)
void	tokenize (const std::string &text, std::vector< int32_t > &output)

---

The documentation for this struct was generated from the following file:

• llm/src/LLaMATokenizer.cc