Background Knowledge Preparation

Before beginning your project we would like you all to familiarize yourself with some background knowledge needed to successfully complete your project. These tools are Git and Dialogflow.

Everything we think you need to know can be found here: 2023: Getting Started: What to Know About Git and DialogFlow

To test your knowledge of these subjects there are two Canvas quizzes on the subjects that must be completed.

Additionally on the page 2023: Getting Started: The Tools Used we explain and help you set up the tools needed to complete this project.

The Project

In this project, you will be developing a conversational cooking assistant that uses speech to interact and is able to conduct a conversation for selecting a recipe and for extended recipe instruction. Automatic Speech Recognition (ASR) has matured to a level where it is possible to categorize a user's utterances adequately to make sense out of them. Also, Speech Synthesis (TTS) can be used to produce well-pronounced spoken utterances from written text. Yet, conversational agents have not become mainstream, and whoever has used a home assistant (Google Home or Apple Siri) has experienced being misunderstood. These assistants are typically able to perform well on basic Question-Answering (QA) interactions, which most of the time consist of just two conversational turns: a question and an answer. However, conducting longer conversations tends to be more challenging. This is because a longer conversation can take (too) many directions and the chance that a user says something unexpected significantly increases. We will investigate this challenge in this project.

Our conversational agent is a cooking assistant that should be able to support a user in the food domain. There are tons of recipes on the internet and sometimes finding something to choose from can prove to be difficult, especially when you want something specific. Sometimes at night, I think to myself “I really want a Mexican recipe, that has less than 10 steps and no bell peppers in it”, but how could I even go about finding my perfect recipe? Maybe with a super huge database of recipes and a bot! In this project, you will be exploring a solution for this problem and develop a cooking assistant that conversationally assists a user in selecting a recipe based on a variety of filters.

We shall provide a bundle of files that still need a bit of work done to complete them. In the instructions we give you we walk you through how to fill in the blanks. If you fill in the blanks the agent should work… but it will be pretty basic…

...

But what would be really cool (and earn you more than a passing grade) is to extend this bot with your own flare and ideas. Read a bit more about this in our Extensions section at the end of your project.

After you finish and extend your bot you get to do some testing! I mean if you bother to code it you should probably make sure it works… You will also have the privilege of checking out another team’s bot. But that's a later problem(after your bot is done circa Week 3) that you can read about in ourTesting and Evaluation section.

After all of that, we have blessed you with the privilege of writing a report about the experience. If you paid attention while working through the instructions and in class, it should really be nice and easy. More information on that can be found in the 2023: End Report Section.

For now, let’s get started with the fun stuff: dividing up work among teammates.

Team Forming and Initial Set Up

By the end of the week, you will need to form teams of 6 and begin to prepare for your project. Roles are divided in this project into three categories: Dialogflow and Filters(2); Visual Support(2); and Patterns and Responses(2). The number in parenthesis is the number of people we recommend you assign to each category. You will still need to work together in many regards, so make sure there is still communication between categories. Together you will set up your project and familiarize yourself with the project tools.

Instructions for the tools and setup can be found here:

...

The Roles: D&F, VS, P&R

Assigning Roles Within Your Team

At the start of the course, you are expected to have formed a team of 6.One of the first things to think about as a team is how to divide the three main tasks or roles amongst your team members:

1. Dialogflow and Filters (2 team members);

2. Visual Support (2 team members); and

3. Patterns and Responses (2 team members).

The two team members with a similar role should work together in pairs. You can successfully do pair programming in person or remotely. As you will all need to work on one and the same agent, however, it will also be essential for the three pair programming subteams to communicate well with each other to make sure the three programming tasks are adequately integrated and work well together. Another task that you will have as a team that will be essential to develop an effective and robust agent is to continuously evaluate and test your bot once it is up and running (even if it is still only minimally functioning!). For this, make sure you also read the Agent testing page.

At the end of this project, you and your team should have a fully functioning conversational agent that is able to assist users with selecting a recipe they would like to cook!

...

The bot is made up of parts This agent will be composed of MARBEL modules and Prolog files which you can develop and edit in Eclipse and Dialogflow which which will interact with a Dialogflow agent that you need to help finish. The goal of this project is to use Dialogflow, MARBEL, and Prolog to provide a conversational agent with the knowledge to assist us in a certain task. In this case, we have a crazy large recipe database(recipe_database.pl) and we really need some help choosing what to eat. We create and further develop. As a start, we will use a large recipe database that the agent can use to suggest recipes to users. The user should be able to tell the agent certain criteria and it should provide us with a fitting recipe. As mentioned your team will be split into pairs. The sections of work are described a briefly below so you can try and figure out who wants to do what.

...

what they would like, and the agent should try to find a matching recipe (of course, assuming that one is available in the database).

To help you decide which team member fits best to which role, the three roles are described briefly below.

Role 1: Dialogflow and Filters (D&F)

Do you want to get to know a very cool Google Cloud tool, and like to optimize the agent’s understanding of what the user says? In the introductory part of the course, you should have learned a bit about Dialogflow. If that sounded super cool to you, then this could be the section for you! You will create Intents, Entities, and Prolog Filter Rules in order to provide the agent with the vocabulary comprehension and filtering abilities it needs to converse about recipes.

For more information, go to the Dialogflow and Filters Section:

...

Role 2: Visual Support (VS)

Do you want to get creative? Your agent will not only have a conversational component, but a visual one. The program uses dynamic webpages to provide the user with visual support in their conversation. This includes not only includes subtitles to the conversation but also so much more, but also information in support of what is talked about at any moment - such as the recipes that fit the preferences voiced by the user. If you are ready to break out a bit of HTML and Prolog, you can create cool pages through rules. The Visual Support team will incorporate visuals to enrich the conversation.

For more information, go to the Visual Support Section:

...

Role 3: Patterns and Responses (P&R)

Do you like to have think in more detail about the anatomy of a conversation? Well either way Then this section role could be of interest to you. The patterns and responses section focuses on the conversation part of the conversational agent. Your bot will not be the most robust it can only handle some specific tasks. Thus, we create patterns of what we think conversations in the agent’s intended domain will look like so it knows what to expect and how to respond. You will encode these patterns in Prolog and add responses for every situation.

For more information go to the Pattern and Responses Section:

...

Setting up the Tools You Need

Before you actually install the tools that you need for this project, first have a brief look at this page to get a quick overview of what you will use for developing your agent:

...

Now you have some idea of what software tools you will be using, get started by following the instructions on this page for setting up the tools you will need:

...

Getting up to Speed: the Background Knowledge You Need

Before beginning your project, we would like you all to familiarize yourself with some important background knowledge about Git and Dialogflow. You will find you will need this to successfully navigate this project, and it will help you to better understand what some of the tools and one of the key components of the agent are all about. The basic stuff that we think you really need to know can be found here:

...