nerdexam
Amazon

MLS-C01 · Question #235

A company is building an application that can predict spam email messages based on email text. The company can generate a few thousand human-labeled datasets that contain a list of email messages…

The correct answer is D. Initialize the model with pretrained weights in all layers. Replace the last fully connected layer with. To fine-tune a pretrained BERT model for a new text classification task with limited custom data, the model should be initialized with all pretrained weights and only the last classification layer should be replaced.

Modeling

Question

A company is building an application that can predict spam email messages based on email text. The company can generate a few thousand human-labeled datasets that contain a list of email messages and a label of "spam" or "not spam" for each email message. A machine learning (ML) specialist wants to use transfer learning with a Bidirectional Encoder Representations from Transformers (BERT) model that is trained on English Wikipedia text data. What should the ML specialist do to initialize the model to fine-tune the model with the custom data?

Options

  • AInitialize the model with pretrained weights in all layers except the last fully connected layer.
  • BInitialize the model with pretrained weights in all layers. Stack a classifier on top of the first output
  • CInitialize the model with random weights in all layers. Replace the last fully connected layer with a
  • DInitialize the model with pretrained weights in all layers. Replace the last fully connected layer with

How the community answered

(35 responses)
  • A
    6% (2)
  • B
    3% (1)
  • C
    3% (1)
  • D
    89% (31)

Why each option

To fine-tune a pretrained BERT model for a new text classification task with limited custom data, the model should be initialized with all pretrained weights and only the last classification layer should be replaced.

AInitialize the model with pretrained weights in all layers except the last fully connected layer.

Initializing random weights in any layer other than the task-specific head would discard the valuable knowledge gained during pretraining, making fine-tuning less efficient.

BInitialize the model with pretrained weights in all layers. Stack a classifier on top of the first output

While a classifier is stacked, the crucial part for a new task is that the original classification head for the pretraining task must be replaced or adapted for the new downstream task's output classes.

CInitialize the model with random weights in all layers. Replace the last fully connected layer with a

Initializing all layers with random weights completely negates the benefit of transfer learning, making the model learn from scratch, which is not ideal when using a pretrained model with limited custom data.

DInitialize the model with pretrained weights in all layers. Replace the last fully connected layer withCorrect

For transfer learning with a pretrained BERT model, initializing the model with all pretrained weights from the original task (e.g., English Wikipedia) is crucial to leverage the rich linguistic representations already learned. The last fully connected layer, which is specific to the original pretraining task's output, should then be replaced with a new layer appropriate for the specific spam/not spam classification task.

Concept tested: Transfer learning with BERT for text classification

Source: https://huggingface.co/docs/transformers/training#fine-tuning-a-pretrained-model

Topics

#Transfer Learning#BERT#Fine-tuning#Text Classification

Community Discussion

No community discussion yet for this question.

Sign up to join the discussion
Full MLS-C01 Practice