tokenize
The process of converting input data into a discrete sequence of tokens that a model can read and predict.
Learn
When to use it
Use tokenization when raw input data, such as text or MIDI, cannot be directly processed by a model. Tokenization breaks down this input into manageable units, enabling models to understand and generate sequences like text or music notes.
Quick example
In the on-device AI model that autocompletes piano performances, MIDI data representing musical notes is tokenized into a sequence that the transformer model can process. This tokenization allows the model to predict and generate the next notes in real-time. Here, the tokenization process is crucial for transforming raw MIDI data into a format the model can understand and work with.
MIDI input → tokenize → model → predicted notes → stop
Ecosystem
Tokenization is a foundational step in preparing data for models, often preceding embedding and model inference.
raw data → tokenize → embedding → model
Misconceptions
| Misconception | Rebuttal |
|---|---|
| Tokenization is only for text | It applies to any sequential data, like MIDI notes |
| Tokens are always words | Tokens can be subwords, characters, or other units |
Trade-offs
- Model compatibility — requires preprocessing setup
- Data size reduction — can lose nuanced information
- Efficiency — may increase complexity of input processing