This page provides an overview of the first 10 capabilities that you will need to develop for your conversational agent. These capabilities will result in an agent that will be able to recommend recipes to a user from a database with almost 900 recipes.

...

Each section below has the same structure: a short description, a task overview, and a link to more detailed instructions. The tasks for each capability come in three, differently color-coded parts:

When you have implemented the first 10 capabilities of your conversational agent, it is up to you to add one or more additional capabilities (see Agent Capability 11+: Extend Your Agent below) that you decide your conversational agent should have. We provide some general suggestions to help you get started to think about some useful capabilities that you can add.

Starting your NLU Pipeline

Summary Description

The first step is to get your conversational agent up and running. There is one ingredient that is still missing, but required before you can run the code provided to you. You need to create your own Dialogflow agent that you will use throughout the project and connect it to the MARBEL agent that manages the dialog (or conversation) between a user and your agent. When you have created this agent, you will be able to run your agent. Next, you will also be asked to put something in the agenda of the MARBEL agent to get things going. And, last but not least, you need to add code for creating an initial start screen (a simple webpage). When you have done this too, you can run your conversational agent and should see the start webpage that you created. Your agent won’t do any talking yet, though. It is just a first step.

Implementation Tasks Overview

Instructions

Building Your Intent and Slot Classifier

•  TODO

...

Introduction to the NLU Pipeline Project

Welcome to the Natural Language Understanding (NLU) Pipeline Project! This comprehensive system is designed to enable interaction with a conversational agent, leveraging cutting-edge technologies for dialogue management, natural language understanding, and recipe recommendations. Whether you are exploring machine learning, dialogue systems, or speech processing, this project offers a hands-on opportunity to build and integrate a sophisticated conversational AI pipeline.

...

Pipeline Overview

Core Components

1. Dialogflow (Initial Step):

   • Purpose: Provides a pre-built NLU solution to kickstart integration with the MARBEL agent.

   • Use Case: Allows the MARBEL agent to handle user interactions before the intent and slot classifier and WHISPER integration are complete.

   • Why Dialogflow?: By using Dialogflow, you can test and update the MARBEL agent’s dialogue management without waiting for the intent-slot classifier and ASR/TTS modules.

2. Intent and Slot Classifier (Custom NLU):

   • Purpose: Replaces Dialogflow to provide a self-built, fine-tuned classifier for interpreting user input.

   • Technology: BERT-based transformer model tailored for intent classification and slot filling.

   • Example:

     • Input: "Find me a Japanese recipe for lunch."

     • Output: Intent: addFilter, Slots: {cuisine: Japanese, mealType: lunch}

3. MARBEL Agent:

   • Purpose: Manages dialogue flow and user interaction through structured conversational patterns.

   • Capabilities:

     • Handles recipe selection, filtering, and confirmation tasks.

     • Manages unexpected user inputs and integrates with the recipe database.

4. Prolog Recipe Database:

   • Purpose: Provides structured storage and retrieval for recipes.

   • Integration: Works with the MARBEL agent to recommend recipes based on user preferences.

5. WHISPER (ASR and TTS):

   • Purpose: Adds speech capabilities to the conversational agent.

   • Features:

     • Converts user speech to text (ASR).

     • Translates agent responses into speech (TTS).

...

Why Start with Dialogflow?

To ensure the MARBEL agent can be developed and tested in parallel with the NLU and ASR/TTS systems, this project starts with Dialogflow as the initial NLU backend. Dialogflow allows you to:

• Define intents and entities easily using a GUI.

• Create training phrases for various intents and test responses immediately.

• Focus on refining the MARBEL agent’s dialogue patterns without waiting for the custom NLU pipeline.

Transitioning from Dialogflow

Once the intent and slot classifier is operational, you will replace Dialogflow with the self-built classifier, gaining full control over NLU customization and performance optimization.

...

Project Goals

The project is designed to:

• Provide a seamless transition from a pre-built to a custom NLU system.

• Demonstrate the application of transformers for intent classification and slot filling.

• Offer hands-on experience in building and integrating NLU, dialogue management, and speech systems.

...

Pipeline Components in Detail

1. Dialogflow:

   • Used for initial intent and slot classification.

   • Allows rapid prototyping and testing of MARBEL agent functionalities.

2. Intent and Slot Classifier:

   • Replaces Dialogflow once operational.

   • Provides a robust, customizable, transformer-based solution.

3. MARBEL Agent:

   • Designed for structured, multi-turn dialogue interactions.

   • Handles recipe selection, filtering, and confirmation tasks.

4. Prolog Recipe Database:

   • Offers efficient recipe retrieval based on user preferences.

5. WHISPER:

   • Adds ASR and TTS functionalities for speech-based interaction.

...

Development Workflow

The development workflow is divided into three structured phases to guide students through building and extending the NLU pipeline:

• Phase 1: Inclusion Filtering
  Students begin by implementing inclusion filtering capabilities, such as "Find me a recipe with salt from Italy." This involves defining intents and slots, training the intent and slot classifier, and integrating it with the MARBEL agent to enable Prolog-based recipe retrieval.

• Phase 2: Exclusion Filtering
  The next step is extending the system to handle exclusion filters, like "I want a recipe without salt." Students update datasets, retrain the model to recognize exclusion intents and slots, and adapt MARBEL patterns to manage combined filtering scenarios.

• Phase 3: Custom Extensions
  Students then innovate by adding their own extensions, such as multi-turn dialogue, advanced filtering for dietary restrictions, or enhanced context handling. This phase emphasizes creativity and the practical application of NLU principles to real-world problems.

This phased workflow ensures an incremental learning process, fostering a comprehensive understanding of NLU systems and their integration into conversational agents.

How you separate work is up to your group.