Tokenizer Struct Reference

Public Attributes
char **	vocab
float *	vocab_scores
TokenIndex *	sorted_vocab
int	vocab_size
unsigned int	max_token_length
unsigned char	byte_pieces [512]

---

The documentation for this struct was generated from the following file:

• llm/src/nn_modules/non_cuda/LLaMA3Generate.cc