Skip to end of metadata
Go to start of metadata

You are viewing an old version of this content. View the current version.

Compare with Current View Version History

« Previous Version 5 Next »

Getting Started

  • go to main.py

giphy (3).webp

Ontology

What is an Ontology?

For general information please check out: Preliminaries and Quiz Materials.

For this project:

  1. Intents represent the user's high-level actions or goals.

    • Example: When a user says, "Can you recommend a recipe?", the intent could be requestRecommendation.

  2. Slots define specific pieces of information extracted from the user’s input.

    • Example: In the query "Add garlic and chicken thighs to the recipe filter," garlic and chicken thighs are slot values of the ingredient slot type.

The ontology file is where all the possible intents and slots for the system are defined.

Steps to Analyze the Ontology

  1. Open the Ontology File

    • Locate the ontology file (ontology.json) in your project. This file contains two key sections:

      • Intents: A list of all possible intents your system can predict.

      • Slots: A dictionary where the keys represent slot types (e.g., ingredient) and the values are lists of possible slot values (e.g., garlic, chicken thighs).

  2. Review the Intents

    • Look at the intents section in the file. Each intent represents a unique user goal or action.

    • Reflect on the variety of intents. For example:

      • What do intents like greeting or farewell imply about the system's capabilities?

      • How does the system distinguish between recipeRequest and requestRecommendation?

  3. Explore the Slots

    • Examine the slots section. This is a dictionary of slot types and their potential values.

    • Key questions to consider:

      • How many slot types are defined? Examples might include ingredient, cuisine, or recipe.

      • Are there any patterns in the slot values?

      • How do these slots connect to our MARBEL agent potentially?

  4. Think About Model Outputs

    • Your model will predict one intent per input (intent classification) and assign a slot label to each token in the input (slot filling).

    • Understanding the ontology helps you map these predictions to actionable output

Stuff to Think About

  1. Study the Ontology File

    • Open ontology.json and carefully review the intents and slots.

    • Make notes on any patterns, ambiguities, or gaps you observe.

  2. Answer the Following Questions

    • What are the most common intents in the file? Are there any that seem rarely used or overly specific?

    • What slot types and values do you think will be the most challenging for the model to predict? Why?

    • How does the structure of the ontology affect how you might design a dataset or interpret the model’s outputs?

Fitting Encoders

What Are Encoders?

Encoders translate text-based labels (like intents and slot types) into numerical values that the model can work with. This process standardizes the inputs and outputs, ensuring consistency across training, evaluation, and inference.

  1. Intent Label Encoder:

    • Maps each intent from the ontology (e.g., recipeRequest, greeting) to a unique number.

    • Used for intent classification.

  2. Slot Label Encoder:

    • Converts slot types and their corresponding BIO-format tags (B-slot, I-slot, O) into numbers.

    • Used for slot filling at the token level.

Steps in the Encoding Process

Take a close look at the fit_encoders function in dataset.py. It performs the following steps:

  1. Load the Ontology:

    • The function reads the ontology file to extract the list of intents and slot types:

  2. Fit the Intent Label Encoder:

    • The intent encoder assigns a unique numerical label to each intent in the ontology:

      intent_label_encoder.fit(intents)

    • Key Insight: This step ensures that intent classification produces outputs in a consistent format.

  3. Generate BIO Tags for Slots:

    • Slot tags are converted into BIO format:

      • B-{slot}: Beginning of a slot entity.

      • I-{slot}: Inside a slot entity.

      • O: Outside of any slot entity.

    • All slot tags are compiled into a single list:

      all_slot_tags = ['O'] + [f'B-{slot}' for slot in slots.keys()]
                   + [f'I-{slot}' for slot in slots.keys()]

    • These tags are then fitted to the slot encoder:

    • Why BIO Format?: This labeling scheme helps identify the boundaries of multi-token slot entities.

      • Think about why this could be important in our context and what slots could specifically benefit.

Dataset

Preprocessing Steps

Raw Data Before Preprocessing

The dataset begins as a collection of raw examples in JSON format (see train.json) where each entry includes:

  • A unique ID.

  • The text of the user's input.

  • The intent of the input.

  • A dictionary of slots, mapping slot types to their values.

Example Raw Data:

{
    "id": "st041",
    "text": "I’d like a meal that’s fast to prepare.",
    "intent": "addFilter",
    "slots": {
        "shortTimeKeyWord": "fast"
    }
}

Steps in Preprocessing

  1. Tokenization:

    • The text is broken into smaller units (tokens) using the BERT tokenizer:

      tokens = tokenizer.tokenize("I’d like a meal that’s fast to prepare.")

    • Output:

      ['i', '’', 'd', 'like', 'a', 'meal', 'that', '’', 's', 'fast', 'to', 'prepare', '.']

  2. BIO Slot Label Encoding:

    • Slot values are matched with their positions in the tokenized text.

    • BIO-format labels are generated:

      • B-{slot}: Beginning of the slot value.

      • I-{slot}: Inside the slot value.

      • O: Outside any slot.

    Example:

    • Slot Definition: "shortTimeKeyWord": "fast"

    • Output BIO Tags:

      ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-shortTimeKeyWord', 'O', 'O', 'O']

  3. Intent Encoding:

    • The intent is mapped to a numerical label using the intent encoder:

      intent_label = intent_label_encoder.transform(["addFilter"])[0]

    • Output:

      0  # (Example numerical label for "addFilter")

  4. Padding and Truncation:

    • Sequences are padded or truncated to a fixed length (max_length):

      • Token IDs are padded with zeros.

      • BIO tags are padded with O.

    • Example:

      • Original Tokens: ['i', '’', 'd', 'like', 'a', 'meal', 'that', '’', 's', 'fast', 'to', 'prepare', '.']

      • Padded Tokens: ['i', '’', 'd', 'like', 'a', 'meal', 'that', '’', 's', 'fast', 'to', 'prepare', '.', '[PAD]']

      • BIO Tags are similarly padded to match the length.

Processed Data After Preprocessing

After preprocessing, each example is transformed into a structured format, including:

  1. input_ids: Tokenized input converted to numerical IDs.

  2. attention_mask: Mask indicating which tokens are real (1) and which are padding (0).

  3. intent_label: Encoded intent label.

  4. slot_labels: Encoded BIO-format slot labels.

Example Processed Data:

{
    'input_ids': torch.tensor([101, 1045, 1521, 1005, 1040, 2066, 1037, 7953, 2008, 1521, 1055, 3435, 2000, 6624, 1012, 0]),
    'attention_mask': torch.tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0]),
    'intent_label': torch.tensor(0),  # Encoded "addFilter"
    'slot_labels': torch.tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0])  # BIO tags for "fast"
}

Summary of Preprocessing

  1. Before Processing:

    • Raw data includes plain text, an intent label, and slots with values.

    • Example: "text": "I’d like a meal that’s fast to prepare."

  2. After Processing:

    • Tokenized, encoded, padded, and structured data ready for model input.

    • Example includes token IDs, attention mask, intent label, and BIO slot labels.

This transformation ensures the data is in the correct format for the BERTNLUModel and PyTorch pipeline.

Distribution

Objective

Analyzing the distribution of intents and slots in your dataset helps you understand its structure, identify potential issues, and ensure the model learns effectively across all labels. In this section, you’ll compute and interpret the frequency of intents and slots for both training and testing datasets.

Implementation needed for this section!

How to Analyze Distribution

  1. Write a Distribution Analysis Function

    • See main.py for the incomplete function.

    • This function should calculate how often each intent and slot appears in the dataset.

    • Think about what fields in the dataset you’ll need:

      • Intent: Directly accessible as example['intent'].

      • Slots: Comes from example['slots'] but might need to be flattened into a single list.

    • Use a counting method to track the frequency of intents and slots.

    Tips:

    • Use tools like collections.Counter for efficient counting.

    • Ensure your function handles edge cases, such as examples without any slots.

  2. Run the Function on Training and Testing Data

    • Call your distribution function for both datasets.

    • Print the results to inspect the frequency of each intent and slot.

    Tips:

    • Compare the distributions of training and test datasets.

    • Look for imbalances or unexpected gaps. For example:

      • Are certain intents or slots underrepresented or missing?

      • Does the test set mirror the training set?

  3. Interpret the Results

    • Once you have the distributions, analyze them to answer key questions:

      • Which intents or slots are the most frequent? The least?

      • Are there any imbalances that might cause the model to focus too heavily on common labels?

      • Are rare intents or slots important for the system’s performance?

    • Reflect on how these observations might affect training.

    Tips:

    • If rare intents or slots are crucial, consider strategies like data augmentation or using weighted loss functions during training.

    • If the test distribution doesn’t match the training distribution, think about how this might affect evaluation.

Hints for Implementation

  • Think about how to efficiently loop through the dataset:

    • For intents, you can directly extract them from the dataset examples.

    • For slots, remember to handle nested structures since slots are stored as dictionaries.

  • Use your existing knowledge of Python tools to count and organize results:

    • A Counter object can help you group and tally items.

  • Make sure to test your function on small, simple datasets before running it on the full dataset to ensure correctness.

Reflection Questions

After analyzing the distributions, think about:

  1. Dataset Balance:

    • Are the distributions skewed? How might this affect the model?

    • Are all intents and slots well-represented, or are some missing?

  2. Test Dataset Representativeness:

    • Does the test set reflect the training set’s distribution?

    • If not, how might this impact evaluation?

By following these steps and reflecting on the results, you’ll gain a deeper understanding of your dataset and its potential challenges. This analysis is crucial for making informed decisions during model training and evaluation.

Train

Evaluate

Inference

  • No labels