Content Comparison

Getting Started

...

go to main.py

Image Removed

Ontology

What is an Ontology?

For general information please check out: Preliminaries and Quiz Materials.

For this project:

1. Intents represent the user's high-level actions or goals.

   • Example: When a user says, "Can you recommend a recipe?", the intent could be requestRecommendation.

2. Slots define specific pieces of information extracted from the user’s input.

   • Example: In the query "Add garlic and chicken thighs to the recipe filter," garlic and chicken thighs are slot values of the ingredient slot type.

The ontology file is where all the possible intents and slots for the system are defined.

Steps to Analyze the Ontology

1. Open the Ontology File

   • Locate the ontology file (ontology.json) in your project. This file contains two key sections:

     • Intents: A list of all possible intents your system can predict.

     • Slots: A dictionary where the keys represent slot types (e.g., ingredient) and the values are lists of possible slot values (e.g., garlic, chicken thighs).

2. Review the Intents

   • Look at the intents section in the file. Each intent represents a unique user goal or action.

   • Reflect on the variety of intents. For example:

     • What do intents like greeting or farewell imply about the system's capabilities?

     • How does the system distinguish between recipeRequest and requestRecommendation?

3. Explore the Slots

   • Examine the slots section. This is a dictionary of slot types and their potential values.

   • Key questions to consider:

     • How many slot types are defined? Examples might include ingredient, cuisine, or recipe.

     • Are there any patterns in the slot values?

     • How do these slots connect to our MARBEL agent potentially?

4. Think About Model Outputs

   • Your model will predict one intent per input (intent classification) and assign a slot label to each token in the input (slot filling).

   • Understanding the ontology helps you map these predictions to actionable output

Fitting Encoders

What Are Encoders?

Encoders translate text-based labels (like intents and slot types) into numerical values that the model can work with. This process standardizes the inputs and outputs, ensuring consistency across training, evaluation, and inference.

1. Intent Label Encoder:

   • Maps each intent from the ontology (e.g., recipeRequest, greeting) to a unique number.

   • Used for intent classification.

2. Slot Label Encoder:

   • Converts slot types and their corresponding BIO-format tags (B-slot, I-slot, O) into numbers.

   • Used for slot filling at the token level.

Steps in the Encoding Process

Take a close look at the fit_encoders function in dataset.py. It performs the following steps:

...

Load the Ontology:

• The function reads the ontology file to extract the list of intents and slot types:

...

Fit the Intent Label Encoder:

• The intent encoder assigns a unique numerical label to each intent in the ontology:

  intent_label_encoder.fit(intents)

• Key Insight: This step ensures that intent classification produces outputs in a consistent format.

Generate BIO Tags for Slots:

...

Slot tags are converted into BIO format:

• B-{slot}: Beginning of a slot entity.

• I-{slot}: Inside a slot entity.

• O: Outside of any slot entity.

All slot tags are compiled into a single list:

...

Image Added

Natural Language Understanding (NLU) is a core component of conversational AI systems, enabling machines to interpret and act on user input in natural language. This Intent and Slot Classifier project is designed to help students understand the pipeline involved in building an NLU model that performs intent classification and slot filling. These tasks allow AI models to classify a user's goal (intent) and extract key information (slots) from their input.

For instance, given the input:
"Find me a Japanese recipe for lunch,"
the model would classify the intent as "find_recipe" and extract slots like "cuisine: Japanese" and "mealType: lunch".

This project uses BERT, a state-of-the-art transformer-based language model, to perform these tasks effectively. The system provides end-to-end functionality for training, evaluating, and running inference on the classifier.

...

Purpose of the Project

The primary goals of this project are:

1. To demonstrate the application of transformers for intent classification and slot filling.

2. To provide students with hands-on experience in creating and working with NLU pipelines.

3. To showcase best practices for preparing datasets, building models, and evaluating their performance.

To complete these goals you must go through all subpages in this section starting with Ontology.

...

Repository Overview

The repository is structured as follows:
utils/
├── checkpoints/          # Directory for saving trained model checkpoints
├── data/                 

...

These tags are then fitted to the slot encoder:

...

Why BIO Format?: This labeling scheme helps identify the boundaries of multi-token slot entities.

• Think about why this could be important in our context and what slots could specifically benefit.

Dataset

Preprocessing Steps

Raw Data Before Preprocessing

The dataset begins as a collection of raw examples in JSON format (see train.json) where each entry includes:

• A unique ID.

• The text of the user's input.

• The intent of the input.

• A dictionary of slots, mapping slot types to their values.

Example Raw Data:

{
    "id": "st041",
    "text": "I’d like a meal that’s fast to prepare.",
    "intent": "addFilter",
    "slots": {
        "shortTimeKeyWord": "fast"
    }
}

Steps in Preprocessing

1. Tokenization:

   • The text is broken into smaller units (tokens) using the BERT tokenizer:

     Code Block
     tokens = tokenizer.tokenize("I’d like a meal that’s fast to prepare.")

   • Output:

     Code Block
     ['i', '’', 'd', 'like', 'a', 'meal', 'that', '’', 's', 'fast', 'to', 'prepare', '.']

2. BIO Slot Label Encoding:

   • Slot values are matched with their positions in the tokenized text.

   • BIO-format labels are generated:

     • B-{slot}: Beginning of the slot value.

     • I-{slot}: Inside the slot value.

     • O: Outside any slot.

   Example:

   • Slot Definition: "shortTimeKeyWord": "fast"

   • Output BIO Tags:

     Code Block
     ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-shortTimeKeyWord', 'O', 'O', 'O']

3. Intent Encoding:

   • The intent is mapped to a numerical label using the intent encoder:

     Code Block
     intent_label = intent_label_encoder.transform(["addFilter"])[0]

   • Output:

     Code Block
     0 # (Example numerical label for "addFilter")

4. Padding and Truncation:

   • Sequences are padded or truncated to a fixed length (max_length):

     • Token IDs are padded with zeros.

     • BIO tags are padded with O.

   • Example:

     • Original Tokens: ['i', '’', 'd', 'like', 'a', 'meal', 'that', '’', 's', 'fast', 'to', 'prepare', '.']

     • Padded Tokens: ['i', '’', 'd', 'like', 'a', 'meal', 'that', '’', 's', 'fast', 'to', 'prepare', '.', '[PAD]']

     • BIO Tags are similarly padded to match the length.

Processed Data After Preprocessing

After preprocessing, each example is transformed into a structured format, including:

1. input_ids: Tokenized input converted to numerical IDs.

2. attention_mask: Mask indicating which tokens are real (1) and which are padding (0).

3. intent_label: Encoded intent label.

4. slot_labels: Encoded BIO-format slot labels.

Example Processed Data:

{
    'input_ids': torch.tensor([101, 1045, 1521, 1005, 1040, 2066, 1037, 7953, 2008, 1521, 1055, 3435, 2000, 6624, 1012, 0]),
    'attention_mask': torch.tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0]),
    'intent_label': torch.tensor(0),  # Encoded "addFilter"
    'slot_labels': torch.tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0])  # BIO tags for "fast"
}

Summary of Preprocessing

1. Before Processing:

   • Raw data includes plain text, an intent label, and slots with values.

   • Example: "text": "I’d like a meal that’s fast to prepare."

2. After Processing:

   • Tokenized, encoded, padded, and structured data ready for model input.

   • Example includes token IDs, attention mask, intent label, and BIO slot labels.

This transformation ensures the data is in the correct format for the BERTNLUModel and PyTorch pipeline.

Distribution

Objective

Analyzing the distribution of intents and slots in your dataset helps you understand its structure, identify potential issues, and ensure the model learns effectively across all labels. In this section, you’ll compute and interpret the frequency of intents and slots for both training and testing datasets.

How to Analyze Distribution

1. Write a Distribution Analysis Function

   • See main.py for the incomplete function.

   • This function should calculate how often each intent and slot appears in the dataset.

   • Think about what fields in the dataset you’ll need:

     • Intent: Directly accessible as example['intent'].

     • Slots: Comes from example['slots'] but might need to be flattened into a single list.

   • Use a counting method to track the frequency of intents and slots.

   Tips:

   • Use tools like collections.Counter for efficient counting.

   • Ensure your function handles edge cases, such as examples without any slots.

2. Run the Function on Training and Testing Data

   • Call your distribution function for both datasets.

   • Print the results to inspect the frequency of each intent and slot.

   Tips:

   • Compare the distributions of training and test datasets.

   • Look for imbalances or unexpected gaps. For example:

     • Are certain intents or slots underrepresented or missing?

     • Does the test set mirror the training set?

3. Interpret the Results

   • Once you have the distributions, analyze them to answer key questions:

     • Which intents or slots are the most frequent? The least?

     • Are there any imbalances that might cause the model to focus too heavily on common labels?

     • Are rare intents or slots important for the system’s performance?

   • Reflect on how these observations might affect training.

   Tips:

   • If rare intents or slots are crucial, consider strategies like data augmentation or using weighted loss functions during training.

   • If the test distribution doesn’t match the training distribution, think about how this might affect evaluation.

Hints for Implementation

• Think about how to efficiently loop through the dataset:

  • For intents, you can directly extract them from the dataset examples.

  • For slots, remember to handle nested structures since slots are stored as dictionaries.

• Use your existing knowledge of Python tools to count and organize results:

  • A Counter object can help you group and tally items.

• Make sure to test your function on small, simple datasets before running it on the full dataset to ensure correctness.

Reflection Questions

After analyzing the distributions, think about:

1. Dataset Balance:

   • Are the distributions skewed? How might this affect the model?

   • Are all intents and slots well-represented, or are some missing?

2. Test Dataset Representativeness:

   • Does the test set reflect the training set’s distribution?

   • If not, how might this impact evaluation?

By following these steps and reflecting on the results, you’ll gain a deeper understanding of your dataset and its potential challenges. This analysis is crucial for making informed decisions during model training and evaluation.

Train

Evaluate

...

# Directory containing data files (ontology, train/test datasets)
│   ├── ontology.json     # Ontology file containing intents, slots, and synonyms
│   ├── train.json        # Training dataset
│   ├── test.json         # Test dataset
│   └── synonyms.json     # Synonyms for slot normalization
├── data_processing.py    # Utilities for additional data preprocessing (if needed)
├── dataset.py            # Dataset preparation and preprocessing module
├── evaluation.py         # Model evaluation and metrics generation
├── run_train_test.py     # Main script to run training, evaluation, and inference
├── model.py              # Defines the BERT-based model architecture
├── predict.py            # Inference module for predicting intents and slots
├── requirements.txt      # Python dependencies for the project
├── train.py              # Training module for the intent-slot classifier
└── utils.py              # Helper functions for argument parsing, slot normalization, and synonym resolution

...

Explanation of Key Modules

run_train_test.py

This is the central script for orchestrating the intent and slot classifier independently. It integrates data preparation, training, evaluation, and inference, all controlled via command-line arguments.

dataset.py

This module handles data loading, preprocessing, and BIO tagging. It uses the NLURecipeDataset class for creating PyTorch-compatible datasets.

model.py

Defines the BERTNLUModel, a BERT-based architecture with separate heads for intent classification and slot filling.

train.py

Implements the training routine, including optimization and backpropagation.

evaluation.py

Provides functionality to evaluate the model’s performance, generating accuracy scores and classification reports.

predict.py

Handles inference by predicting intents and slot tags for new inputs. It includes functions for extracting and normalizing slots using the ontology.

utils.py

Contains helper functions for:

• Parsing command-line arguments.

• Loading and normalizing synonyms.

• Extracting and formatting slot values from predictions.

...

Arguments

The utils.py module defines the command-line arguments. Here’s a summary:

...

Use

1. Viewing Dataset Distribution

To analyze the distribution of intents and slots in the dataset:

python run_train_test.py --show_dist 

2. Training the Model

To train the model using the training dataset:

python run_train_test.py --train_model

3. Evaluating the Model

To evaluate a pre-trained model on the test dataset:

python run_train_test.py --evaluate 

4. Running Inference

To predict the intent and slots for a given input text:

python run_train_test.py --inference_text "Find me a Japanese recipe for lunch."

5. Run with ASR

When complete with this section read https://socialrobotics.atlassian.net/wiki/spaces/PCA2/pages/2709488567/Run+your+Conversational+Agent#Run-your-Intent-and-Slot-Classifier-with-WHISPER to connect your intent and slot classifier with WHISPER