starcoder_vocab Struct Reference

Public Member Functions
void	add_special_token (const std::string &token)

Public Attributes
std::map< std::string, int32_t >	token_to_id
std::map< int32_t, std::string >	id_to_token
std::vector< std::string >	special_tokens

---

The documentation for this struct was generated from the following file:

• llm/include/GPTBigCodeTokenizer.h