Dialogflow → Intent and Slot Classifier with WHISPER

While your intent and slot classifier is in development, you can use Dialogflow, a powerful Google Cloud tool, to handle Automatic Speech Recognition (ASR) and Intent and Slot Classification for your conversational agent. Dialogflow allows you to optimize your agent's understanding of user input by creating intents with training phrases and entities that define the vocabulary and natural language understanding capabilities needed for recipe recommendations.

In this project, before your intent and slot classifier is trained and tested, Dialogflow serves as the core Natural Language Understanding (NLU) component, transcribing user utterances into text, classifying them into intents, and extracting relevant information (entities) for recipe recommendations.

The Dialogflow agent acts as an NLU agent, which is connected to a MARBEL agent. The MARBEL agent receives transcripts, intents, and entities from Dialogflow and uses this information to manage the dialogue with the user. It structures interactions through conversational patterns, serving as the dialogue manager. To help you get started, we’ve provided a basic MARBEL agent, but your first task is to create your team’s Dialogflow agent. Only one team member needs to create the agent and share it with the rest of the team, as your team requires just one Dialogflow agent to collaborate effectively.

Once your custom intent and slot classifier is fully developed, it will replace Dialogflow and work in conjunction with WHISPER, an advanced ASR system, to manage the entire NLU pipeline. WHISPER will convert user speech into text, and your classifier will process the text to identify user intents and extract entities. This replacement will give you complete control over the NLU system, allowing for tailored adjustments and deeper integration into your agent's architecture. Unlike Dialogflow, your classifier can be customized specifically for recipe recommendations, enabling improved accuracy and responsiveness to user inputs.

With the replacement in place, your custom system will continue to interface with the MARBEL agent, providing it with the same intent and entity data, but with enhanced flexibility and scalability. This shift eliminates dependency on external tools like Dialogflow, ensuring better alignment with your project’s needs and granting your team full ownership of the conversational pipeline. By transitioning to your classifier and WHISPER, you’ll achieve a more robust, efficient, and adaptive solution for managing user interactions.

Starting with Your MARBEL Agent

If your project installation is complete, you now should be able to run the code. Try it by following the instructions here: Run your Conversational Agent. You should see a start page.

A Pattern to Get Started

The dialog manager agent uses an agenda to structure the conversation with a user. An agenda consists of one or more conversational patterns; conversational patterns are templates for small sub-dialogs or mini-dialogs that can be viewed as building blocks to create a larger dialog or conversation. By inspecting this agenda, the agent can figure out what it should do and in what order things should be handled in the conversation. The agenda is managed by a MARBEL agent, which implements the dialog manager. There already is some code for initializing the agenda in the dialog_init.mod2g module. Open this file and navigate to the code line where the agenda is inserted into the agent’s database (state). As you can see, the agenda in the agent we provided to you as a starting point is still empty (it is the empty list []). We will use some of the patterns that we will create in the project to define the agent’s agenda by adding them to the list. To get started, we need to add something already to the agenda. We can use the start pattern for this which has already been defined in the patterns.pl file (check it out). This pattern just waits for the user to click a button on the screen. Below we will create such a screen where a user can click a start button to initiate the interaction. For now, let’s just add the start pattern to the agent’s agenda:

Visuals

The conversational agent that we will create not only talks but also shows things using dynamic web pages that are displayed on a screen. These dynamic web pages will provide the user with some additional visual support to be able to keep track of some of the things that happened in the conversation. We want the agent, for example, to display information about where we are at in the conversation by, for example, displaying subtitles that refer to different parts (conversational patterns) of the conversation. We will also want the agent to display the preferences for or constraints on recipes that have already been added by the user and show how many recipes still fit those preferences. This will help the user understand what the agent is doing and help them remember which preferences they already indicated to the agent.

We provide a basic visual design. Thereafter it is up to you to make things look better and make additional design choices of your own.

All HTML templates for pages can be found in social-interaction-cloud/sic_framework/services/webserver/templates . Feel free to create them anyways you would like. They do have to include some information. Please see Visuals and Visual Requirements for specifics.

Start Page

To finish this page your start.html page must be present and complete.

Run your agent again!

Run your agent again, see here for the instructions for Run your Conversational Agent.

You should now see the start page that we created! Don’t yet press the button…

Switch to the Debug perspective in Eclipse, select the MARBEL Debugging Engine in the Debug area, and press pause. Inspect the agent’s state and check out in particular the session/1 predicate.

Now start the agent again by pressing resume. And then press the start button on the page. Go back again to the Debug perspective and pause the agent again. Now again inspect the agent’s state and in particular the session/1 predicate. Double-click on the session/1 fact if you can't see it completely.

What has happened?