Project: Introduction and Overview

Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) have matured to a level where it is possible to translate a user's utterances into text (ASR) and to classify text into intents (NLU) to make sense of what a user says. Also, Text-To-Speech (TTS) can be used for speech synthesis to produce well-pronounced spoken utterances from written text. Yet, conversational agents have not become mainstream, and whoever has used a home assistant (Google Home or Apple Siri) has experienced being misunderstood. These assistants are typically able to perform well on basic Question-Answering (QA) interactions, which most of the time consist of just two conversational turns: a question and an answer. However, conducting longer conversations tends to be more challenging. This is because a longer conversation can take (too) many directions, and the chance that a user says something unexpected significantly increases. We will investigate this challenge in this project.

In this project, you will be developing a conversational recipe recommendation agent that uses speech to interact and is able to conduct a conversation for selecting a recipe to cook. Your agent should be able to assist a user in selecting a recipe using a variety of filters. The agent does not need to be able to assist a user with the instruction steps of the recipe itself, which is out of scope of the Project MAS course. We chose to focus on the recipe selection activity since it already poses several challenges for building an effective and robust conversational agent. First, there are many different ways in which this conversation may be conducted, and many different ways in which a user can phrase what it wants from the agent. A user can specify different aspects of a recipe that the recipe it will finally select should satisfy (e.g., type of ingredients, cooking duration, type of course, etc.). Second, the recipe recommendation domain already is a broad knowledge space that the agent needs to be able to handle to understand what the user is looking for. The agent will have to reason over its database of recipes to filter for recipes that fit the user’s preferences.

You are provided with a Prolog knowledge base of close to a 1,000 recipes and their components, that still requires an effort of your group to make it usable by the recipe recommendation agent. In the instructions, we walk you through the procedure by pinpointing aspects of the agent that need to be altered or filled in. If you fill in these blanks, the agent should work, but it will still be pretty basic…

Therefore, the project does not stop there, and we challenge you to extend the basic bot with your own flare and ideas, to earn a higher grade. You can read more about this in theExtensions section, which also has some suggestions for extending your agent.

Already during development and definitely after finishing your agent, it is essential to do some testing! You and your group members can test your agent. But generally speaking, developers are not the best testers of their own software. Therefore, it will also be important to involve other users outside your team (e.g., your friends, family, another team doing this project) and ask them to interact with your agent (and, of course, you should return the favor and check out another team’s agent too!). You can read more about this in the Agent Testing and User Study section.

Before you get started, make sure to check out the main Project Deliverables. Apart from a working agent (a MARBEL and Dialogflow agent) that you evaluated, your project work should also result in a Final Report in which you describe its main features and its performance based on the testing you did, amongst other things. You should also maintain a log book of the main project activities that you worked on, and make the data available that you used to evaluate your agent. You can also already look ahead at how your work will be evaluated by checking out the Assessment Rubric.

From us you shall receive in a GitHub repository the following:

A folder of Entities titled StudentVersionEntities: contains Dialogflow entities for you to upload when creating your agent

A folder of Intents that has the addFilter Intent .json file
The agent code directory

Okay, now you should be ready to Get Started.