Page Comparison

Project MAS Rubric

Agent functionality

Criteria

Poor

Average

Good

Ratings

Max Pts

Opening and recipe selection

The agent can only open with a greeting and does not assist the user in

selecting a recipe. (0)

The agent can do a basic filtering of recipes based on features indicated by the user. (5)

The agent can effectively assist the user in selecting a recipe in a conversational (rather than website-like) way,

coming to a subset of recipes for the user to choose from. (9)

9

Recipe instruction and closing

The agent can only instruct the recipe step-by-step as directed by the user, and properly end the recipe instruction and conversation. (0)

The agent can cope with some deviations from the recipe instruction, such as clarification, appraisal, switching recipes, or a capability check. (5)

The agent can cope with many deviations from the recipe instruction, and apply slot-filling where needed. (9)

9

Navigation

The agent can hardly accommodate navigation other than proceeding in the happy flow. (0)

The agent can do some backward navigation (such as undoing a chosen feature during recipe selection, or going back one recipe step) and/or go to a specific step in the recipe, but not at all relevant points in the conversation. (4)

The agent can accommodate navigation at all relevant points in the conversation. (7)

7

Repair

The agent can only perform the basic fallback repair when a user intent is not understood. (0)

The agent can successfully initiate fallback repair and out-of-context repair, and respond properly to user-initiated repair at different positions in the conversation. (4)

In addition to initiating and responding to repair, where possible the agent can assist the user with information (in speech and visual presentation) on what part of its utterance needs to be addressed in order to repair. The agent will not end up in an endless repair cycle. (7)

7

Extensions

No extensions to the original assignment have been implemented. (0)

Some functionality has been added to the agent’s capacities, albeit not very challenging or creative. (4)

The agent has been extended with (a) creative and challenging component(s) that makes a considerable difference to the utility of the cooking assistant. (8)

8

Total points

40

Visual support

Criteria

Poor

Average

Good

Ratings

Max Pts

Orientation

The visuals merely display what the agent says at any point. (0)

The visuals indicate the status of the conversation (e.g.: filtering step, chosen recipe, current step) at some points, as well as the current conversation options of the user, but not at all points in the conversation or not insightful enough. (3)

The visuals give an insightful account of the conversation status and directions to take at

any point in the conversation. (5)

5

Overview

It is difficult to interpret the displayed information at many points in the conversation, or too little information is presented. (0)

The information presented on the display is too messy or unintuitive at points in the conversation. (3)

Overall the visuals strike the right balance between the amount of information presented and overview of the information. (5)

5

Appearance

No effort is spent to display information in an attractive way to the user. (0)

The appearance is somewhat appealing, with images to support certain information, and a design structure that fits the information given at different points. (3)

The display is highly appealing, with images to support information where relevant, and a design structure that fits the information given at any point. (5)

5

Total points

15

Agent quality and design

Criteria

Poor

Average

Good

Ratings

Max Pts

Robustness

The assistant easily breaks down during a conversation. (0)

The assistant can cope with some deviations

by the user from the happy conversation flow. (3)

The assistant is rather robust and can cope with several deviations

by the user from the happy conversation flow. (5)

5

Conversation design

Agent utterances are not very enticing, and conversation patterns do not cater for much flexibility in the conversation. (0)

The user is to some extent considered in the way that agent utterances and conversation patterns were implemented. (3)

The group has written agent utterances in a style that engages the user, and has implemented many conversation patterns to cope with the different conversation paths that are expected to be conducted in the agent-user interactions. (5)

5

Quality of coding and documentation

Many parts of the code

are not reusable in multiple places, and comments are hardly added. (0)

The coding is well-documented, but not all comments make clear the idea behind a code snippet. Some parts of the code are needlessly hard-coded. (2)

The coding is of high quality and well-documented. (5)

5

Total points

15

Written report

Criteria

Poor

Average

Good

Ratings

Max Pts

Introduction and Dialog

engine

The project is poorly introduced, and the groups show little understanding of the dialog manager architecture and functionality. (0)

The group properly introduces the project and shows some understanding of the dialog manager architecture and functionality. (2)

The group properly introduces the project and shows in-depth understanding of the dialog manager architecture and functionality. (4)

4

Design rationale

The

design choices are difficult to understand from how they are presented. (0)

The

design choices are written down in a mostly understandable way, while part of the choices is not well-motivated. (5)

The

design choices are written down in an understandable and convincing way. (9)

9

Testing

It is unclear from the testing

section how the assistant is performing. (0)

The testing

section gives a reasonable account of how the assistant was tested and what were the main findings. (3)

The capacities and points of improvement of the assistant are evaluated in a structured way and clearly presented in the testing

section.

(5)

5

Improvements

The identified

points of improvement are unclear or not addressable in any way. (0)

The identified

points of improvement are clear, but

the way in which they may be implemented is not properly described, unrealistic or unfeasible. (2)

The identified

points of improvement are clear and

the way in which they may be implemented is convincingly presented. (4)

4

Clarity and presentation

The writing style, structure and lay-out are messy and unclear. (0)

The writing style and presentation are up to standards, but some parts are unclear. (4)

Very clear report in terms of writing style, structure and lay-out. (8)

8

Total points

30