Background Knowledge Preparation

Before beginning your project we would like you all to familiarize yourself with some background knowledge about Git and Dialogflow, which are needed to successfully complete your project.

Everything we think you need to know can be found here: 2023: Getting Started: What to Know About Git and DialogFlow

To test your knowledge of these subjects there are two Canvas quizzes on the subjects that must be completed.

Additionally, on the page 2023: Getting Started: The Tools Used we explain and help you set up the infrastructure needed to complete this project.

Please complete the instructions on both of these pages before beginning your project.

The Project

In this project, you will be developing a conversational cooking assistant that uses speech to interact and is able to conduct a conversation for selecting a recipe to cook. Automatic Speech Recognition (ASR) has matured to a level where it is possible to categorize a user's utterances adequately to make sense out of them. Also, Speech Synthesis (TTS) can be used to produce well-pronounced spoken utterances from written text. Yet, conversational agents have not become mainstream, and whoever has used a home assistant (Google Home or Apple Siri) has experienced being misunderstood. These assistants are typically able to perform well on basic Question-Answering (QA) interactions, which most of the time consist of just two conversational turns: a question and an answer. However, conducting longer conversations tends to be more challenging. This is because a longer conversation can take (too) many directions and the chance that a user says something unexpected significantly increases. We will investigate this challenge in this project.

In this project, you will develop a cooking assistant that conversationally assists a user to select a recipe based on a variety of filters, whereas the instruction of the recipe itself is not in the scope of the Project MAS course. We chose to focus on the recipe selection activity of cooking support, since several challenging elements of building a conversational agent come together in this segment. First, there are many different ways in which this conversation may conducted, with different aspects of recipes that may be specified by the user to come to a selection (type of ingredients, cooking duration, course, etc.), and the option to mention multiple of those elements in one utterance. Second, the agent has to cover a broad knowledge space to understand what the user is looking for. Third, the agent has to reason over its database of recipes to filter for recipes that fit the user’s preferences.

You are provided with a prolog knowledge base of close to 1,000 recipes and their components, that still requires an effort of your group to make it usable by the cooking assistant. In the instructions we walk you through the procedure by pinpointing aspects of the agent that need to be altered or filled in. If you fill in these blanks the agent should work, but it will be pretty basic…

The project does not stop there. We challenge you to extend this basic bot with your own flare and ideas, so as to earn a higher grade. Read a bit more about this in our Extensions section at the end of your project.

During development and after finishing the agent, it is essential to do some testing! This will also involve having another team conversing with your agent, and checking out another team’s agent yourself. You can read about this in our System testing and User study section.

Apart from a working and tested agent, your project work will result in an end report in which you describe its different features and its performance according to the testing, among others. More information on that can be found in the 2023: End Report Section.

For now, let’s get started by dividing work among teammates.

Team Forming and Initial Set Up

At the start of the course, you are expected to have formed teams of 6, which is divided into three roles: Dialogflow and Filters (2 team members); Visual Support (2 team members); and Patterns and Responses (2 team members). The two team members with a similar role will work together, but the three teams of two need to communicate well to make sure those three aspects of the agent are integrated.

Instructions for the tools and setup can be found here:

The Roles: D&F, VS, P&R

At the end of this project, you and your team should have a fully functioning conversational agent!

The agent is composed of modules in Eclipse and Dialogflow which you need to help finish. The goal of this project is to use Dialogflow, MARBEL, and Prolog to develop a conversational agent with the knowledge to assist us in a given task. In this case, we have a large recipe database (recipe_database.pl) that the agent draws upon to help the user choose what to eat. The user should be able to tell the agent certain criteria and the agent should present a fitting recipe (if available in the database).

To help you decide which team member fits best to which role, the three roles are described briefly below. Make sure you also read the System testing page as you should be continuously testing your bot once it is minimally functioning.

Dialogflow and Filters

Do you want to get to know a very cool Google Cloud tool, and like to optimize the agent’s understanding of what the user says? In the introductory part of the course, you should have learned a bit about Dialogflow. If that sounded super cool to you then this could be the section for you! You will create Intents, Entities, and Prolog Filter Rules in order to provide the agent with the vocabulary comprehension and filtering abilities it needs to converse about recipes.

For more information go to the Dialogflow and Filters Section:

Visual Support

Do you want to get creative? Your agent will not only have a conversational component but a visual one. The program uses dynamic webpages to provide the user with visual support in their conversation. This not only includes subtitles to the conversation, but also information in support of what is talked about at any moment - such as the recipes that fit the preferences voiced by the user. If you are ready to break out a bit of HTML and Prolog, you can create cool pages through rules. The Visual Support team will incorporate visuals to enrich the conversation.

For more information go to the Visual Support Section:

Patterns and Responses

Do you like to think in more detail about the anatomy of a conversation? Then this role could be of interest to you. The patterns and responses section focuses on the conversation part of the conversational agent. Your bot will not be the most robust it can only handle some specific tasks. Thus, we create patterns of what we think conversations in the agent’s intended domain will look like so it knows what to expect and how to respond. You will encode these patterns in Prolog and add responses for every situation.

For more information go to the Pattern and Responses Section: