ML.Tokens¶
| Name | Mandatory | Description | Default | Type |
|---|---|---|---|---|
⬅️ Input |
The input of the shard, if any | String |
||
Output ➡️ |
The resulting output of the shard | [Int] |
||
Tokenizer |
No | The tokenizer to use. | none |
Var(Tokenizer) |
AddSpecialTokens |
No | If true, add special tokens. | true |
Bool |
AsTensor |
No | Outputs a tensor object instead of an int sequence. | false |
Bool |
Format |
No | The format of the output tensor. If As Tensor is true. | TensorType::U32 |
TensorType |
GPU |
No | If true, the output tensor will be on the GPU (if ). | false |
Bool |
Tokenizes text using a tokenizer.