AI-900 · Question #254
What are three stages in a transformer model? Each correct answer presents a complete solution. NOTE: Each correct answer is worth one point.
The correct answer is B. embedding calculation C. tokenization D. next token prediction. What is tokenization in transformer? Preparing Text Data for Transformers: Tokenization, Mapping ... Tokenization is the process of dividing text into smaller units called tokens, which can be words, phrases, subwords, or characters. In the context of Transformer models, tokeniza
Question
Options
- Aobject detection
- Bembedding calculation
- Ctokenization
- Dnext token prediction
- Eanonymization
How the community answered
(30 responses)- A7% (2)
- B90% (27)
- E3% (1)
Explanation
What is tokenization in transformer? Preparing Text Data for Transformers: Tokenization, Mapping ... Tokenization is the process of dividing text into smaller units called tokens, which can be words, phrases, subwords, or characters. In the context of Transformer models, tokenization is a crucial step in preprocessing text data for use in natural language processing tasks. Input embeddings: The input sentence is first transformed into numerical representations called embeddings. These capture the semantic meaning of the tokens in the input sequence. For sequences of words, these embeddings can be learned during training or obtained from pre- trained word embeddings. Inference: After training, the model can be used for inference on new data. During inference, the input sequence is passed through the pre-trained model, and the model generates predictions or representations for the given task.
Topics
Community Discussion
No community discussion yet for this question.