Dialogflow → Intent and Slot Classifier with WHISPER

While your intent and slot classifier is in development, you can use Dialogflow, a powerful Google Cloud tool, to handle Automatic Speech Recognition (ASR) and Intent and Slot Classification for your conversational agent. Dialogflow allows you to optimize your agent's understanding of user input by creating intents with training phrases and entities that define the vocabulary and natural language understanding capabilities needed for recipe recommendations.

In this project, before your intent and slot classifier is trained and tested, Dialogflow serves as the core Natural Language Understanding (NLU) component, transcribing user utterances into text, classifying them into intents, and extracting relevant information (entities) for recipe recommendations.

The Dialogflow agent acts as an NLU agent, which is connected to a MARBEL agent. The MARBEL agent receives transcripts, intents, and entities from Dialogflow and uses this information to manage the dialogue with the user. It structures interactions through conversational patterns, serving as the dialogue manager. To help you get started, we’ve provided a basic MARBEL agent, but your first task is to create your team’s Dialogflow agent. Only one team member needs to create the agent and share it with the rest of the team, as your team requires just one Dialogflow agent to collaborate effectively.

Once your custom intent and slot classifier is fully developed, it will replace Dialogflow and work in conjunction with WHISPER, an advanced ASR system, to manage the entire NLU pipeline. WHISPER will convert user speech into text, and your classifier will process the text to identify user intents and extract entities. This replacement will give you complete control over the NLU system, allowing for tailored adjustments and deeper integration into your agent's architecture. Unlike Dialogflow, your classifier can be customized specifically for recipe recommendations, enabling improved accuracy and responsiveness to user inputs.

With the replacement in place, your custom system will continue to interface with the MARBEL agent, providing it with the same intent and entity data, but with enhanced flexibility and scalability. This shift eliminates dependency on external tools like Dialogflow, ensuring better alignment with your project’s needs and granting your team full ownership of the conversational pipeline. By transitioning to your classifier and WHISPER, you’ll achieve a more robust, efficient, and adaptive solution for managing user interactions.

Create a Dialogflow agent

• Please follow the instructions here: https://cloud.google.com/dialogflow/es/docs/quick/build-agent#create-an-agent.

• Share the Dialogflow agent with your other team members (assign them the Developer role): https://cloud.google.com/dialogflow/es/docs/access-control.

• Use the zip file found in the repository to make an agent follow the steps below to make a Dialogflow agent:

...

Connect it to the MARBEL agent

• Create and download your Dialogflow agent’s JSON key file by following the instructions here: https://cloud.google.com/iam/docs/creating-managing-service-account-keys#get-key. Note that you will have to create a service account for the project (button at the top under the blue bar). When you have done this, you can click on the email for the service account. Note that the JSON file is automatically downloaded when it is created.

• One of you should add the JSON key file to your team’s Git repository in the agent folder of that repository and everyone should pull to update their local versions.

• Retrieve the Project ID of your Dialogflow agent: you can find this by clicking on the settings (cogwheel) icon next to your Dialogflow agent’s name here: https://dialogflow.cloud.google.com/.

• Open the .mas2g file in Eclipse. Then (1) add the name of the JSON key file as the value of the flowkey environment initialization parameter to the mas file, and (2) the Project ID of your Dialogflow agent as the value of the flowagent parameter.

Run the code!

You now should be able to run the code. Try it by following the instructions here: [TBU]Run your Conversational Agent. What do you see? Why are you seeing this? Inspect the html.pl file to understand what has happened.

A Pattern to Get Started

The dialog manager agent uses an agenda to structure the conversation with a user. An agenda consists of one or more conversational patterns; conversational patterns are templates for small sub-dialogs or mini-dialogs that can be viewed as building blocks to create a larger dialog or conversation. By inspecting this agenda, the agent can figure out what it should do and in what order things should be handled in the conversation. The agenda is managed by a MARBEL agent, which implements the dialog manager. There already is some code for initializing the agenda in the dialog_init.mod2g module. Open this file and navigate to the code line where the agenda is inserted into the agent’s database (state). As you can see, the agenda in the agent we provided to you as a starting point is still empty (it is the empty list []). We will use some of the patterns that we will create in the project to define the agent’s agenda by adding them to the list. To get started, we need to add something already to the agenda. We can use the start pattern for this which has already been defined in the patterns.pl file (check it out). This pattern just waits for the user to click a button on the screen. Below we will create such a screen where a user can click a start button to initiate the interaction. For now, let’s just add the start pattern to the agent’s agenda:

Visuals

The conversational agent that we will create not only talks but also shows things using dynamic web pages that are displayed on a screen. These dynamic web pages will provide the user with some additional visual support to be able to keep track of some of the things that happened in the conversation. We want the agent, for example, to display information about where we are at in the conversation by, for example, displaying subtitles that refer to different parts (conversational patterns) of the conversation. We will also want the agent to display the preferences for or constraints on recipes that have already been added by the user and show how many recipes still fit those preferences. This will help the user understand what the agent is doing and help them remember which preferences they already indicated to the agent.

The instructions that we will give you will guide you on how to implement quite a basic visual design. Thereafter it is up to you to make things look better and make additional design choices of your own. Creating the basic visual design should not require too much effort and should leave you with plenty of opportunity and time to extend and experiment with your own visual design choices!

...

The code for generating HTML is provided in the Prolog html.pl file that is part of your MARBEL agent project. Open this file to inspect it. When you scroll through the file, you can see that the file is organized into four parts:

1. Page layouts: Here you should put all the code for the various webpage designs that are introduced with the capabilities you need to implement. You should add code for the start screen right at the top of this part of the file. The place to put this code is already indicated by a code comment for your convenience.

2. Code for generating HTML elements: This part provides Prolog definitions for generating elements of an HTML page. One of the first rules in this section defines the button(Content, Html) predicate. The clause defining this predicate first collects some template code for creating a simple button element in HTML (the template sits right above the defining clause in the file) and then applies this template using the Content parameter to generate a button that says whatever you put in Content. We will use this predicate to add a button on our start page below! As you can see in the file, there are predicates for adding several other types of content to the HTML page, including, for example, lists of items, and images. To better understand the definitions in the html.pl file, and for more background on how to create HTML code using Bootstrap read the[TBU]Visual Support Guide.

3. Code for generating the content for the body of an HTML webpage: This part defines a predicate html(Content, AddHeader, AddFooter, Html) for generating the main HTML code and when indicated also a header (when AddHeader is instantiated with true) and/or footer (when AddFooter is instantiated with true). The part also introduces various other useful predicates for facilitating the interaction with a user, e.g., for enabling audio interaction on a webpage and displaying a microphone icon.

4. A helper predicate: A dynamic predicate pageUpToDate(Name) that will be used by the agent to keep track of whether a page that is displayed is up to date or not (and needs updating in that case).

Start Page

So let's get started! The first page that we will implement together is a start page. We will provide a detailed step-by-step guide for you on how to create it. Pages are defined by Prolog rules. The Prolog rule that we will code here for creating a start page will also serve as a kind of template example of how you can create the other pages using Prolog, HTML, and Bootstrap. The basic idea is to define a predicate page(PageName, Text, Html) which generates HTML code (a complete webpage) that is returned in the variable Html for a page named PageName; the idea behind the Text parameter is to use it for adding specific text to a page.

Each of the pages that we will create is connected to a specific conversational pattern. Most of the patterns for these pages will still need to be implemented as part of later capabilities too. But the initial pattern linked to the start page that we will add here already exists and is the start pattern that we already added to the agenda above. As a convention, we will always use the name of a conversational pattern as a name for a page that we create too. There is some logic behind this choice that we will explain below. For the start page, we therefore need to specify a rule that defines the page(start, _, Html) that will generate HTML code for a page called start.

The main requirement for the start page is that it has a button the user can click when they are ready to start a conversation. The basic idea thus is to display an HTML page at the start before a conversation has started that allows a user to start the conversation with a button click. The start page can also be used to provide some information about the agent before a user starts talking to it. The basic page layout that we will create consists of four parts: a title, some introductory text, an instruction text on when to press the start button, and, last but not least, the button itself that says “Start”.

The Prolog rule that we will create for the page(start, _, Html) has a basic template structure that we can reuse for most other pages too. It always 1: starts with checking that the active conversational pattern (at ‘top level’, but disregard that for now) has the same name (pattern ID) as the page name by using the currentTopLevel(PatternID) predicate; in our case, here we replace PatternID with start. Only if that is the case, the rule will succeed, and code for a complete HTML page will be generated. After performing this check, 2: Prolog code for constructing HTML code for the page follows. We typically organize pages in rows (often using matching Bootstrap row elements) each of which matches with one of the three parts of our page layout (title, text, button). This way of organizing our HTML pages provides a generic kind of template approach to structure any page that can be easily reused. Then 3: we use a built-in predicate atomic_list_concat(+List, -Atom) for putting the different parts (rows) together. And, finally, 4: we use a predefined predicate html(Body, false, false, Html) for generating a Bootstrap-compliant HTML page without a header and without a footer. The overall structure of our start page code then will look like this:

 %%% Page layout for start page (before conversation has started).
 page(start, _, Html) :-
    % 1: Condition for when to show this page.
		currentTopLevel(start),
	% 2: Constructing the HTLM page.
	% First row: a warning inside a Jumbotron element.
		CODE WE STILL NEED TO FIGURE OUT, SEE BELOW.
	% Second row: introductory text inside a Jumbtron element.
		CODE WE STILL NEED TO FIGURE OUT, SEE BELOW.
	% Third row: instruction and Start button inside an alert.
		CODE WE STILL NEED TO FIGURE OUT, SEE BELOW.
	% 3: Putting everything together.
		atomic_list_concat([FirstRow, SecondRow, ThirdRow], MainElementContent),
	% 4: Create the HTML page.
		html(MainElementContent, false, false, Html).

Note that in the above code template at line 15 three variables FirstRow, SecondRow, and ThirdRow have been used to refer to the first, second, and third row of our HTML code. The atomic_list_concat/2 predicate is used to concatenate these variables (as elements of a list) into the overall MainElementContent HTML code for our page. The predicate concatenates atoms which means that the terms bound to these variables should be atoms too. The next thing we need to do is to complete the code template by writing code that generates the HTML code for each of the three rows and substituting those code snippets, respectively, at lines 7, 9, and 11.

...

div('<h1 class="text-center">Please read this first:</h1>', 'jumbotron', '', FirstRow)

Note that a string between single quotes also is an atom in SWI Prolog (if you want to know more see here). For those of you who know basic HTML the string between the quotes should look familiar: the HTML code creates a large heading text using the tags <h1> and </h1>. For those of you for whom HTML is new, you can find out more here.

Second row. For the second row, we will add a few lines of text to a paragraph and embed that again in a https://www.w3schools.com/bootstrap4/bootstrap_jumbotron.asp. We will first concatenate the lines as atomic strings (between single quotes) using the atomic_list_concat/2 predicate, we will then create a paragraph element using this content by using the predefined applyTemplate(Template, Content, Html) predicate and a simple HTML code template for a paragraph element with centered text, and, finally, add that into a Bootstrap Jumbotron element. The three-part code snippet that we are looking for is this:

atomic_list_concat([
    'You are about to interact with our agent <b>YourAgentName</b>.<br>',
	'It can help you find a recipe to your taste.'
], ParagraphContent),
applyTemplate('<p><center>~a</center></p>', ParagraphContent, ParagraphElement),
div(ParagraphElement, 'jumbotron', '', SecondRow)

We have just put some text there for you to get things moving but, of course, feel free to change it in any way you want. The main point that we want to illustrate here is that it is convenient to use the atomic_list_concat/2 predicate to concatenate a list of string atoms (lines) into a single Prolog atom. It helps you keep track and provides a nice overview of the lines of text you want to put on a page. It also avoids potential syntax issues, as using whitespaces and newlines in Prolog code may often cause problems. Check out the utils.pl file for the definition of the applyTemplate/3 predicate and the html.pl file for the definition of the div/4 predicate. The applyTemplate/3 predicate adds the few lines of text in our example code into an HTML paragraph element using the <p> and </p> tags. The <center> tag is used to center the text inside the paragraph. In our code example above, the applyTemplate/3 substitutes the (term bound to the) variable ParagraphContent for the ~a placeholder that is part of the template string. Thereafter the paragraph element is embedded in a Bootstrap alert element using the div/4 predicate. You can find out more on the Bootstrap template for alerts here. Finally, the result is unified with our SecondRow variable.

...

button('<b>Start</b>', 'btn btn-lg btn-info', 'start', Button)

We want to use a (somewhat smaller than before) heading text for the instruction and show that before our button. We use the atomic_list_concat/2 predicate for concatenating these two elements:

atomic_list_concat(['<h3>Done reading? Then please press start to begin</h3>', Button], 
  ThirdRowContent)

And, finally, we embed this content inside an alert element, with some added classes to style the alert, using the div/4 predicate:

div(ThirdRowContent, 'alert alert-info text-center', '', ThirdRow)

We now can fill in the gaps in the Prolog rule for defining our first page using the three different code snippets for the three different parts (rows) we want to display on our start page. It remains for you to put this all together and add it to the html.pl file in the right place. Just to be sure, when you copy-paste the rows into the overall page template code, make sure to also add the commas that are still missing at the end of each of the three code snippets.

You now have created your first page that your conversational agent can display. The page does not look particularly great yet and there is still much that can be improved. How exactly you and your team organize or create your HTML pages is up to you, as long as you fulfill the minimal conditions that the page must meet. Hence, make sure to revisit this page and revise it again later during the project!

Run your agent again!

Run your agent again, see here for the instructions for [TBU]Run your Conversational Agent.

You should now see the start page that we created! Don’t yet press the button…

Switch to the Debug perspective in Eclipse, select the MARBEL Debugging Engine in the Debug area, and press pause. Inspect the agent’s state and check out in particular the session/1 predicate.

Now start the agent again by pressing resume. And then press the start button on the page. Go back again to the Debug perspective and pause the agent again. Now again inspect the agent’s state and in particular the session/1 predicate. Double-click on the session/1 fact if you can't see it completely.

What has happened?

Info

All done?

1. Getting Started with your MARBEL Agent

Summary Description

The first step is to get your conversational agent up and running. There is one ingredient that is still missing, but required before you can run the code provided to you. You need to create your own Dialogflow agent that you will use throughout the project and connect it to the MARBEL agent that manages the dialog (or conversation) between a user and your agent. When you have created this agent, you will be able to run your agent. Next, you will also be asked to put something in the agenda of the MARBEL agent to get things going. And, last but not least, you need to add code for creating an initial start screen (a simple webpage). When you have done this too, you can run your conversational agent and should see the start webpage that you created. Your agent won’t do any talking yet, though. It is just a first step.

Implementation Tasks Overview

1. Set Up Dialogflow Agent

   • Create a Dialogflow agent with default settings (English as the language).

   • Share it with team members by assigning the Developer role.

   • Import the provided zip file for basic setup.

2. Integrate Dialogflow with MARBEL

   • Download the JSON key file for Dialogflow and add it to the agent folder in your repository.

3. Initialize MARBEL Agenda

   • Replace the empty agenda in the dialog_init.mod2g file with the start pattern (agenda([start])).

4. See the Start Page

5. Run and Debug the Agent

   • Test the integration by running the MARBEL agent and verifying state changes in Eclipse’s Debug perspective.

Instructions

Getting Started with your MARBEL Agent

2. Greet, and Self-Identify

Summary Description

Natural conversations are typically opened. Our agent should not just straight away embark on the task of recommending a recipe. It should first secure the attention of its user. Opening a conversation is a basic conversational competence. A common way of opening a conversation is by greeting each other. You should provide your agent with this basic conversational competence too. It should, moreover, be able to introduce itself to its user as in the following conversational pattern. A welcoming webpage should also be created.

Implementation Tasks Overview

Tasks Focused on Dialogue Patterns

1. Implement Basic Greeting Pattern

   • Define the c10 pattern in patterns.pl for the greeting exchange: [agent, greeting] and [user, greeting].

   • Add the condition agentName('') to ensure it is used only when the agent has no name.

2. Add Self-Identification Pattern

   • Extend c10 in patterns.pl to include [agent, selfIdentification].

   • Add not(agentName('')) to ensure it is used only when the agent has a name.

3. Specify Agent Responses

   • Add phrases for greeting and selfIdentification intents in responses.pl.

   • Dynamically retrieve the agent’s name using agentName(Name) to generate self-identification responses.

4. Add Patterns to the Agenda

   • Update dialog_init.mod2g to include c10 in the agenda after start.

...

Tasks Focused on Visual Support

1. Create a Welcoming Page.

...

Tasks Focused on Debugging and Testing

1. Test Greeting Patterns

   • Test the basic greeting pattern by running the agent and initiating interaction from the Start page.

   • Set the agent’s name in dialog_init.mod2g and verify self-identification functionality.

2. Test User Response Variations

   • Provide non-greeting inputs to test the agent’s response and inspect session updates in Debug mode.

Instructions

Greet, and Self-Identify

3. Request a Random Recipe Recommendation

Summary Description

The second capability will enable the agent to recommend a recipe when a user does not care (to refine) what kind of recipe they are looking for (see the example conversational pattern below). A new pattern will be introduced to this purpose for selecting a recipe, and a new intent should be added to your Dialogflow agent to enable understanding user requests for recommending “just some recipe”. Quite some work will need to be done for introducing the first code needed for retrieving recipes from the recipe database. The first versions of two new pages will also need to be created. One page for making clear to the user that they should inform the agent about features of recipes that help refine the selection of recipes and clarify what the user is looking for. Another page for asking a user to (dis)confirm that the recipe that the agent has selected and recommends to the user is what the user is looking for.

Implementation Tasks Overview

Tasks Focused on Dialogue Patterns

1. Implement Recipe Selection Pattern

   • Define the a50recipeSelect pattern in patterns.pl with these moves:

     • [agent, specifyGoal]: Agent asks the user what recipe they want.

     • [user, requestRecommendation]: User requests a random recipe.

     • [agent, recommend]: Agent suggests a recipe.

   • Add [agent, insert(a50recipeConfirm)] to direct the agent to the recipe confirmation pattern.

2. Implement Recipe Confirmation Pattern

   • Define the a50recipeConfirm pattern in patterns.pl to allow users to confirm or disconfirm a recipe.

3. Specify Agent Responses

   • Add responses in responses.pl:

     • text(specifyGoal, "..."): Agent asks about recipe preferences.

     • text(recommend, "..."): Agent suggests a random recipe.

...

Tasks Focused on Recipe Filtering

1. Add Recipes to the Database

   • Each team member adds one favorite recipe to recipe_database.pl, ensuring unique recipe IDs and consistent ingredient names.

2. Implement Recipe Selection Logic

   • Add rules in recipe_selection.pl:

     • currentRecipe(RecipeID): Retrieves the selected recipe from memory.

     • recipeIDs(RecipeIDs): Collects all recipe IDs from the database.

     • recipesFiltered(RecipeIDs): Filters recipes based on user preferences or constraints.

3. Update Conversational Memory

   • Add a rule in dialog_update.mod2g to update the memory with the randomly selected recipe when a user requests a recommendation.

...

Tasks Focused on Visual Support

1. Design Recipe Overview 1 Page

2. Design Recipe Confirmation Page

...

Tasks Focused on Debugging and Testing

1. Test Recipe Recommendation

   • Add a50recipeSelect to the agenda in dialog_init.mod2g and verify the agent's ability to:

     • Ask for recipe preferences.

     • Respond with a random recipe recommendation.

2. Test Recipe Confirmation

   • Ensure the agent correctly displays the recommended recipe and allows user confirmation.

Instructions

Request a Recommendation

4. Select Recipes by Name

Summary Description

Instead of leaving it up to the agent to suggest a recipe and choose one out of all remaining recipes, a user also can mention a specific recipe themselves and ask the agent to present that. This means that the Dialogflow agent should have knowledge of all of the recipes that are in the database too. We will make this available to that agent by adding a recipe entity (type). That will also enable the agent to recognize these recipes in user expressions.

Implementation Tasks Overview

Tasks Focused on Dialogue Patterns

1. Add a New Recipe Request Pattern

   • Define a variant of the a50recipeSelect pattern in patterns.pl with these moves:

     • [agent, specifyGoal]: Agent asks what recipe the user wants to cook.

     • [user, recipeRequest]: User requests a specific recipe by name.

     • [agent, recipeChoiceReceipt]: Agent acknowledges the selected recipe.

     • [agent, insert(a50recipeConfirm)]: Move to the confirmation phase.

2. Specify the Agent’s Recipe Acknowledgment Response

   • In responses.pl, define a rule for text(recipeChoiceReceipt, Txt) to construct the response dynamically.

   • Retrieve the recipe name from memory using currentRecipe/1 and construct the acknowledgment (e.g., "Artichoke and pine nut pasta is a great choice!") using string_concat/3.

...

Tasks Focused on Visual Support

1. Update Visuals for Recipe Request

...

Tasks Focused on Debugging and Testing

1. Test Recipe Request Handling

   • Add the new pattern to the agent’s agenda in dialog_init.mod2g after the c10 pattern.

   • Test the interaction flow:

     • Agent asks for a recipe.

     • User requests a specific recipe.

     • Agent acknowledges the recipe choice and moves to the confirmation phase.

2. Verify Intent Recognition

   • Check the Dialogflow Training Tool to confirm the recipeRequest intent is recognized.

   • Verify the SIC server logs show the correct intent and transcript.

3. Handle ASR Failures

   • Use the chatbox feature to avoid ASR failures and ensure the agent can handle specific recipe requests reliably.

Instructions

Select Recipes by Name

5. Filter Recipes by Inclusion

Summary Description

Implementation Tasks Overview

Tasks Focused on Filtering Logic

Refine the Recipe Filtering Logic

• Expand the recipesFiltered/3 predicate to handle lists of filters.

• Implement recursive filtering by applying each filter to the recipe list sequentially:

  • Apply the first filter and pass the remaining recipes to the next recursive call.

  • Continue until all filters have been applied.

Define Ingredient-Based Filtering

• Add rules for the hasIngredient/2 predicate:

  • Check if a recipe contains a specific ingredient (e.g., "ginger").

  • Check if a recipe uses an ingredient type (e.g., "meat") by associating it with specific examples (e.g., "chicken").

  • Use the typeIngredient/2 predicate for hierarchical ingredient filtering.

Add Filtering by Cuisine and Meal Type

• Extend the applyFilter/4 predicate to include filtering by:

  • Cuisine: Ensure recipes match a specified cuisine type (e.g., "Italian").

  • Meal Type: Filter recipes based on meal categories (e.g., "dessert").

Support Dietary Restrictions

• Add logic for filtering recipes by dietary restrictions:

  • Implement the diet/2 predicate to check if a recipe satisfies restrictions like vegetarian, vegan, gluten-free, or spicy.

  • Use the ingredientsMeetDiet/2 helper predicate to validate dietary restrictions for all ingredients in a recipe.

...

Tasks Focused on Dialogue Patterns

Enable Recipe Feature Requests

• Add a new a21featureRequest pattern for user requests to refine filters dynamically.

• Design two variants of the pattern:

  1. Recipe Selection Context: The user requests a feature during the recipe selection phase (a50recipeSelect).

  2. Recipe Confirmation Context: The user requests a feature while reviewing a specific recipe (a50recipeConfirm).

Handle Conflicting Feature Requests

• Use the removeConflicts(Params) action to handle conflicts when a new feature request overrides an existing one.

• Ensure the agent acknowledges conflicts and updates filters accordingly.

Agent Responses

• Implement agent intents and responses:

  • ackFilter: Acknowledge successful application of filters.

  • noRecipesLeft: Inform the user when no recipes match their filters.

  • featureRemovalRequest: Prompt the user to remove conflicting filters.

...

Tasks Focused on Visual Support

Extend Recipe Overview Pages

• Recipe Overview 1 Page:

  • Show user-requested filters when more than 15 recipes remain.

  • Display only a summary of filters to avoid overwhelming the user.

• Recipe Overview 2 Page:

  • Display recipe titles and images when the filtered list has 15 or fewer recipes.

  • Use visually appealing cards to present recipe details.

Extend the Recipe Confirmation Page

• Add detailed recipe information, including:

  • Cooking instructions.

  • Ingredient list with quantities.

  • Estimated preparation/cooking time.

  • Number of servings.

• Optionally, include a summary of user-applied filters for context.

...

Tasks Focused on Debugging and Testing

Test Filtering Functionality

• Test the new filtering logic with various scenarios:

  • Combine filters for ingredients, cuisine, and dietary restrictions.

  • Verify that recipes are correctly filtered and conflicts are resolved.

Validate Feature Requests

• Confirm that the agent correctly handles feature requests in both a50recipeSelect and a50recipeConfirm contexts.

• Check the agent’s responses and ensure they align with the updated logic.

Ensure Robust Filtering

• Test edge cases:

  • Apply filters that leave no recipes (e.g., contradictory filters).

  • Apply multiple filters that narrow down to fewer than 16 recipes.

...

Instructions

Filtering by Inclusion

6. Connecting Your Entire Pipeline

Connect MARBEL with Intent/Slot Classifier and WHISPER remove Dialogflow.

Instructions

Connecting Your Entire Pipeline

7. Unexpected Intents

Summary Description

A conversational agent should be able to handle dialog moves of users that do not necessarily fit into the currently active pattern. Such moves are “unexpected” in the sense that they do not fit into a conversational pattern, but should be expected as users will almost always slightly deviate from a rigidly framed pattern. Typical examples are expressions of appreciation and requests for information about how the agent can assist a user (a capability check). A second type of unexpected move is not due to user behavior but due to failures of speech recognition. Dialogflow will match with a so-called default fallback intent in cases where it is unable to recognize what a user says and cannot classify it as one of the (other) intents of the agent. There is another case that we will look into too where what a user says does not seem to fit into the conversational context. Again, we can capture the “unexpected” in patterns, which will enable the conversational agent to handle them:

Implementation Tasks Overview

...

Tasks Focused on Repair Patterns

1. Responding to Fallbacks

• Scenario: When the agent cannot recognize a user’s intent (i.e., Dialogflow matches the fallback intent).

  • Example:

    • User: "Have you read The Hobbit?"

    • Agent: "What do you mean?"

• Steps:

  1. Add a b12 pattern in patterns.pl to handle fallback intents.

  2. Define a paraphrase request (paraphraseRequest) as the agent’s response.

  3. Add text responses in responses.pl for the paraphraseRequest intent.

...

2. Responding to Out-of-Context Intents

• Scenario: When the agent recognizes a user’s intent but finds it irrelevant to the current conversational context.

  • Example:

    • Agent: "What recipe would you like to cook?"

    • User: "Hey there."

    • Agent: "I am not sure what that means in this context."

• Steps:

  1. Add a b13 pattern in patterns.pl:

     • First move: Any user intent.

     • Second move: Agent’s contextMismatch(Intent) response, using the intent as a parameter.

  2. Regulate this pattern in the updateSession.mod2g file as the last fallback option for recognized intents.

  3. Add responses for contextMismatch(Intent) in responses.pl:

     • Use text/3 to customize responses based on the conversational context.

     • Include an acknowledgment of the mismatch and provide guidance on expected user behavior.

...

Tasks Focused on Enhancing User Interaction

1. Handling User Appreciation

• Scenario: When the user expresses gratitude.

  • Example:

    • User: "Thanks."

    • Agent: "You're welcome."

• Steps:

  1. Add a b42 appreciation pattern in patterns.pl.

  2. Define appreciation (user intent) and appreciationReceipt (agent response) intent labels.

  3. Add responses for appreciationReceipt in responses.pl.

...

2. Handling Capability Checks

• Scenario: When the user asks what the agent can do.

  • Example:

    • User: "What can you do?"

    • Agent: "I can help you find recipes, filter them by preferences, and guide you through the cooking process."

• Steps:

  1. Add a c30 capability check pattern in patterns.pl:

     • Intent labels: checkCapability (user intent) and describeCapability (agent response).

  2. Define responses for describeCapability in responses.pl:

     • Provide a concise, context-aware summary of the agent’s capabilities.

     • Use text/3 to tailor responses based on the ongoing conversational context.

...

Tasks Focused on Visual Support

Update Visuals

• Enhance the visuals to support the repair patterns:

  • Fallbacks: Display a friendly prompt asking the user to rephrase their input.

  • Out-of-Context Intents: Provide subtle visual feedback indicating the agent’s confusion and suggest appropriate actions.

  • Capability Checks: Add a visually appealing list of capabilities or icons to accompany the agent’s verbal response.

...

Tasks Focused on Debugging and Testing

Test Repair Patterns

1. Fallbacks:

   • Test the agent’s ability to respond with a paraphrase request when encountering unrecognized intents.

   • Example: Say random phrases like “the sky is blue” and verify the agent responds appropriately.

2. Out-of-Context Intents:

   • Test responses to user intents that do not align with the current context.

   • Example: Interrupt the recipe selection process with a greeting and ensure the agent provides context-aware feedback.

3. Capability Checks:

   • Test responses to general and context-specific capability inquiries.

   • Example: Ask "What can you do?" at different stages of the conversation.

...

Instructions

Handle Unexpected Intents

8. Confirm the Chosen Recipe

Summary Description

When users have finished their search for a nice recipe to cook, their choice should be displayed (which we already implemented for Filtering by Inclusion ) and they should be asked to check the details to confirm the recipe is indeed what they would like to cook. Of course, we need to take at least two scenarios into account: (1) a user confirms they are happy with the recipe, or (2) they indicate it is after all not quite what they were looking for. If all is fine and well, the agent should close the conversation by saying farewell. Otherwise, the conversation should move back to the recipe selection stage. The following conversational pattern needs to be implemented:

Implementation Tasks Overview

Tasks Focused on Confirming Recipe Choice

1. Implementing Recipe Confirmation Patterns

• Scenario: After showing a recipe, the agent asks the user to confirm their choice.

  • Example:

    • Agent: "Can you confirm that Chicken Alfredo is the recipe you would like to cook?"

    • User: "Yes." (Alternatively: "No.")

    • Agent: "Great!" (Alternatively: "That is unfortunate. Let's pick another recipe.")

• Steps:

  1. Add three variants of the a50recipeConfirm pattern in patterns.pl:

     • User Confirms: User says "yes" or expresses appreciation. The conversation moves to the farewell phase.

     • User Disconfirms: User says "no." The agent returns to the recipe selection phase (a50recipeSelect).

     • Fallback Handling: Handle unrecognized or ambiguous responses gracefully (optional).

  2. Handle appreciation as a confirmation:

     • In the a50recipeConfirm pattern, treat appreciation as equivalent to confirmation when it occurs in this context.

  3. Add logic for what happens after confirmation:

     • User Confirms: Transition to the c43 farewell pattern.

     • User Disconfirms: Transition back to the a50recipeSelect pattern to allow the user to choose another recipe.

...

2. Agent Responses for Recipe Confirmation

• Add responses for the recipeCheck intent in responses.pl:

  • For Confirmation:

    • Example: "Great! I’m happy you like this recipe."

  • For Disconfirmation:

    • Example: "That’s unfortunate. Let’s pick another recipe."

• Dynamically insert the recipe name into the confirmation question:

  • Example: "Can you confirm that Chicken Alfredo is the recipe you would like to cook?"

...

Tasks Focused on Saying Farewell

1. Implementing Farewell Patterns

• Scenario: The agent bids farewell to conclude the conversation.

  • Example:

    • Agent: "Goodbye!"

    • User: "Bye."

    • Agent: "Feel free to start again if you’d like to find more recipes."

• Steps:

  1. Add a c43 pattern in patterns.pl to model the farewell sequence:

     • Agent Goodbye: The agent says goodbye.

     • User Goodbye: The user responds with a farewell.

  2. Add a final topic check:

     • Before ending the conversation, the agent asks if the user wants to find another recipe.

     • Example:

       • Agent: "Is there anything else you’d like to do? Would you like to find another recipe?"

       • If the user confirms: Add [agent, restart] to the pattern to restart the conversation.

...

2. Agent Responses for Saying Farewell

• Add responses for the farewell intent in responses.pl:

  • Examples:

    • "Goodbye! Have a great day."

    • "Thanks for chatting! See you next time."

• Add responses for the topic check if the user wants to restart:

  • Example: "Okay, let’s start fresh! What recipe are you looking for?"

...

Tasks Focused on Visual Support

1. Create a Farewell Page

• Purpose: Match the top-level intent c43 to provide a closing webpage that thanks the user and invites them to return.

• Content:

  • A warm farewell message:

    • Example: "Thank you for using Recipe Assistant! We hope you found something delicious."

  • An encouragement to return:

    • Example: "Feel free to start again to explore more recipes."

• Design:

  • Simple, clean layout with visually appealing graphics (e.g., a smiling chef or a plate of food).

  • Prominently display a "Start Over" button (if applicable).

...

Tasks Focused on Debugging and Testing

1. Test Recipe Confirmation Flow

• Add a50recipeConfirm to patterns.pl but do not add it to the agent’s agenda:

  • This pattern is inserted dynamically after a recipe is selected, so no need to include it in the agenda manually.

• Verify:

  • The agent transitions to the confirmation phase after a recipe is selected.

  • The agent handles "yes," "no," and appreciation responses correctly.

  • The conversation flows back to recipe selection after disconfirmation.

2. Test Farewell Flow

• Add c43 to the agent’s agenda in dialog_init.mod2g.

• Verify:

  • The agent says goodbye and asks if the user wants to restart.

  • If the user confirms restarting, the conversation resets to the initial agenda.

  • If the user declines, the conversation ends gracefully.

...

Instructions

Confirmation and Closing

9. Removing Filters and Showing Recipes on Demand

Summary Description

The last capability on our list, before we ask you to extend the capabilities of the conversational agent using your own insights and ideas, is about allowing users to remove requests they made before and to allow them to indicate that they are done providing their preferences. You should implement the following patterns:

Implementation Tasks Overview

...

Tasks Focused on Allowing Users to Finalize Recipe Constraints

1. User Stops Adding Preferences

• Scenario: The user indicates they do not want to add more constraints.

  • Example:

    • Agent: "Can you elaborate on what you're aiming for in your recipe?"

    • User: "I don't want to add anything else."

    • Agent:

      • If ≤100 recipes remain: "OK. Here is a list of recipes that you can choose from."

      • If >100 recipes remain: "Sorry, there are still too many recipes left to show them all. Please add more preferences."

• Steps:

  1. Implement the a21noMoreFilters Pattern:

     • Two Variants:

       • pictureGranted: Triggered when ≤100 recipes remain.

       • pictureNotGranted: Triggered when >100 recipes remain.

  2. Agent Memory Update:

     • If the agent grants the user’s request, add [agent, update(['show'='true'])] to the pattern to update the agent’s memory.

     • Modify the condition for showing the second recipe overview page (html.pl) to check for the memory key-value pair 'show', 'true'.

...

2. Agent Responses for Finalizing Preferences

• Add responses for the pictureGranted and pictureNotGranted intents in responses.pl:

  • pictureGranted:

    • Example: "Here is a list of recipes that you can choose from."

  • pictureNotGranted:

    • Example: "Sorry, there are still too many recipes left to show them all. Please add more preferences."

• Dynamically calculate the number of remaining recipes using the recipesFiltered/1 predicate to determine which response to use.

...

Tasks Focused on Removing Filters

1. Completing Filter Removal Responses

• Scenario: The user requests to remove filters, and the agent provides feedback based on the number of remaining recipes.

  • Four Cases:

    1. Large Number of Recipes (>800):

       • Example: "What kind of recipe would you like?"

    2. Moderate Number of Recipes (16–800):

       • Example: "What other preference would you like to add?"

       • Condition: Memory key-value pair 'show', 'true' must not be set.

    3. No Recipes Left (0):

       • Example: "There are no recipes, please remove more requirements."

    4. Small Number of Recipes or Show Command (<16 or show=true):

       • Example: "Here are some recipes that fit your requirements."

• Steps:

  1. Add rules for the featureInquiry intent in responses.pl:

     • Use recipesFiltered/1 to calculate the number of remaining recipes.

     • Use length/2 to count the recipes and apply appropriate conditions.

  2. Handle the show=true memory key condition for small or moderate recipe sets.

2. Debugging the a21removeKeyFromMemory Patterns

• Check the existing patterns in patterns.pl for removing filters (a21removeKeyFromMemory).

• Verify that all filter removal scenarios trigger the correct featureInquiry response.

...

Tasks Focused on Visual Support

Update Visuals for Recipe Overviews

• Recipe Overview 2 to Recipe Overview 1:

  • When users remove filters or finalize preferences, ensure a seamless transition between pages:

    • Example: From showing recipe titles and images (Overview 2) to just a list of filters (Overview 1).

  • Use dynamic visual cues to inform users of what has changed (e.g., "Filter removed!").

Dynamic Recipe Display

• Modify the second recipe overview page to account for the memory key-value pair 'show', 'true':

  • Show recipes even if there are more than 15 left, provided the user requested to stop adding filters.

...

Tasks Focused on Debugging and Testing

1. Test Finalizing Preferences

• Verify the agent correctly handles cases where the user stops adding filters:

  • Test with ≤100 and >100 recipes left to ensure the pictureGranted and pictureNotGranted responses trigger appropriately.

  • Confirm the memory update (show=true) triggers the correct recipe overview page.

2. Test Filter Removal

• Test removing filters at different recipe counts:

  • Large Numbers: Prompt the user to add more specific preferences.

  • Moderate Numbers: Suggest adding more preferences unless show=true.

  • No Recipes: Ask the user to remove filters.

  • Small Numbers or Show Command: Display the remaining recipes.

3. Test Visual Transitions

• Ensure smooth transitions between Recipe Overview 1 and Recipe Overview 2:

  • Test with dynamic changes in recipe counts and user actions (e.g., removing filters or finalizing preferences).

...

Instructions

Remove a Filter and See Recipes on Demand