Load the Limite tokenizer — limite_tokenizer • Rlimite

Load the Limite tokenizer

Usage

limite_tokenizer(
  identifier = "paradigma-inc/limite-1b-violetto",
  revision = "main",
  local_files_only = FALSE
)

Arguments

identifier: Model identifier or local path.
revision: Hugging Face revision.
local_files_only: If TRUE, do not download missing Hub files.

Value

A tok_tokenizer.