Introduction to the NLU Pipeline Project

Welcome to the Natural Language Understanding (NLU) Pipeline Project! This comprehensive system is designed to enable interaction with a conversational agent, leveraging cutting-edge technologies for dialogue management, natural language understanding, and recipe recommendations. Whether you are exploring machine learning, dialogue systems, or speech processing, this project offers a hands-on opportunity to build and integrate a sophisticated conversational AI pipeline.

Pipeline Overview

Core Components

1. Dialogflow (Initial Step):

   • Purpose: Provides a pre-built NLU solution to kickstart integration with the MARBEL agent.

   • Use Case: Allows the MARBEL agent to handle user interactions before the intent and slot classifier and WHISPER integration are complete.

   • Why Dialogflow?: By using Dialogflow, you can test and update the MARBEL agent’s dialogue management without waiting for the intent-slot classifier and ASR/TTS modules.

2. Intent and Slot Classifier (Custom NLU):

   • Purpose: Replaces Dialogflow to provide a self-built, fine-tuned classifier for interpreting user input.

   • Technology: BERT-based transformer model tailored for intent classification and slot filling.

   • Example:

     • Input: "Find me a Japanese recipe for lunch."

     • Output: Intent: addFilter, Slots: {cuisine: Japanese, mealType: lunch}

3. MARBEL Agent:

   • Purpose: Manages dialogue flow and user interaction through structured conversational patterns.

   • Capabilities:

     • Handles recipe selection, filtering, and confirmation tasks.

     • Manages unexpected user inputs and integrates with the recipe database.

4. Prolog Recipe Database:

   • Purpose: Provides structured storage and retrieval for recipes.

   • Integration: Works with the MARBEL agent to recommend recipes based on user preferences.

5. WHISPER (ASR and TTS):

   • Purpose: Adds speech capabilities to the conversational agent.

   • Features:

     • Converts user speech to text (ASR).

     • Translates agent responses into speech (TTS).

Why Start with Dialogflow?

To ensure the MARBEL agent can be developed and tested in parallel with the NLU and ASR/TTS systems, this project starts with Dialogflow as the initial NLU backend. Dialogflow allows you to:

• Define intents and entities easily using a GUI.

• Create training phrases for various intents and test responses immediately.

• Focus on refining the MARBEL agent’s dialogue patterns without waiting for the custom NLU pipeline.

Transitioning from Dialogflow

Once the intent and slot classifier is operational, you will replace Dialogflow with the self-built classifier, gaining full control over NLU customization and performance optimization.

Project Goals

The project is designed to:

• Provide a seamless transition from a pre-built to a custom NLU system.

• Demonstrate the application of transformers for intent classification and slot filling.

• Offer hands-on experience in building and integrating NLU, dialogue management, and speech systems.

Pipeline Components in Detail

1. Dialogflow:

   • Used for initial intent and slot classification.

   • Allows rapid prototyping and testing of MARBEL agent functionalities.

2. Intent and Slot Classifier:

   • Replaces Dialogflow once operational.

   • Provides a robust, customizable, transformer-based solution.

3. MARBEL Agent:

   • Designed for structured, multi-turn dialogue interactions.

   • Handles recipe selection, filtering, and confirmation tasks.

4. Prolog Recipe Database:

   • Offers efficient recipe retrieval based on user preferences.

5. WHISPER:

   • Adds ASR and TTS functionalities for speech-based interaction.

Development Workflow

The development workflow is divided into three structured phases to guide students through building and extending the NLU pipeline:

• Phase 1: Inclusion Filtering
  Students begin by implementing inclusion filtering capabilities, such as "Find me a recipe with salt from Italy." This involves defining intents and slots, training the intent and slot classifier, and integrating it with the MARBEL agent to enable Prolog-based recipe retrieval.

• Phase 2: Exclusion Filtering
  The next step is extending the system to handle exclusion filters, like "I want a recipe without salt." Students update datasets, retrain the model to recognize exclusion intents and slots, and adapt MARBEL patterns to manage combined filtering scenarios.

• Phase 3: Custom Extensions
  Students then innovate by adding their own extensions, such as multi-turn dialogue, advanced filtering for dietary restrictions, or enhanced context handling. This phase emphasizes creativity and the practical application of NLU principles to real-world problems.

This phased workflow ensures an incremental learning process, fostering a comprehensive understanding of NLU systems and their integration into conversational agents.

How you separate work is up to your group.