Your NLU Model

This assignment will guide you in building a simple yet powerful model for natural language understanding (NLU) tasks, such as identifying a user's intent and extracting key information (slots) from their input. The incomplete model is found in model.py in your repository.

Step 1: Understand the Problem

• What you are building: A model that uses BERT, a pre-trained language model, to classify user intents (e.g., "find_recipe") and identify specific slots (e.g., "cuisine: Japanese").

• Key Components:

  • Intent Classification: Determines the overall goal of the user.

  • Slot Filling: Extracts detailed information from the input text.

Step 2: Prepare for Development

1. Decide Input and Output:

   • The model takes tokenized input (a sequence of numbers representing words) and a corresponding attention mask (to focus on meaningful tokens).

   • It outputs:

     • A logit vector for intents (one number per intent).

     • A logit matrix for slots (one number per token for each slot type).

2. Determine Number of Classes:

   • Count the number of unique intents in your dataset.

   • Count the number of unique slot labels (e.g., B-cuisine, I-cuisine).

Step 3: Define the Model Architecture

1. Use a Pre-trained BERT Model:

   • BERT is pre-trained to understand language; you'll use its outputs to classify intents and slots.

   • Load the pre-trained BertModel from the transformers library.

   • Hint for later: One can find information on a pre-trained model by looking under model.config.____

2. Add a Linear Layer for Intent Classification:

   1. Intent Classifier:

      • What it does: Classifies the entire sentence's intent.

      • Input size: The size of the pooled output from BERT. (Hint: This would be the hidden_size of the last BERT layer)

      • Output size: The number of intents in your dataset.

   2. BERT provides a special [CLS] token embedding for the entire input text.

   3. Use this embedding to classify intents by passing it through a linear layer that maps to the number of intent classes.

3. Add a Linear Layer for Slot Classification:

   1. Slot Classifier:

      • What it does: Predicts a slot label for each token in the input.

      • Input size: The size of the last hidden state embeddings from BERT.

      • Output size: The number of slot labels in your dataset.

   2. BERT generates embeddings for all tokens in the input sequence.

   3. Use these token embeddings to classify slots by passing them through another linear layer.

Step 4: Implement the Forward Pass

1. Tokenize Input:

   • Input text is converted into token IDs and an attention mask (already done in preprocessing).

2. Pass Input Through BERT:

   • Use the BERT model to generate two outputs:

     • last_hidden_state: A sequence of embeddings for all tokens.

     • pooler_output: A single embedding summarizing the entire input.

3. Generate Predictions:

   • Feed the pooler_output to the intent classifier.

   • Feed the last_hidden_state to the slot classifier.

4. Return Both Outputs:

   • Return intent logits (one per intent class) and slot logits (one per slot label for each token).