Page Comparison

Project Conversational Agents - Rubric

Basic agent

Criteria

Poor

Average

Good

Ratings

Max Pts

Dialogflow intents and entities

Poor or lacking implementation of intents and entities, preventing the agent from functioning. (0)

Not all intents and entities that were instructed were properly implemented, disabling certain functionalities. (3)

All instructed intents and entities were properly implemented. (5)

5

Robustness of intent recognition

Intents were trained with a minimum of training phrases, enabling only understanding of very specific input. (0)

Only part of the intents were implemented with a wide coverage of possible user

expressions, or the implementation of the intents was not made robust enough against confusing one intent with another. (4)

Intents were trained with a wide coverage of possible user

expressions and entities (if applicable), while the chance for confusion between intents was reduced to a minimum. (7)

7

Recipe filtering

Poor or lacking implementation of recipe filtering, preventing the agent from functioning. (0)

Not all instructed filtering functions were properly implemented, disabling certain functionalities. (4)

All instructed recipe filtering functions were properly implemented. (8)

8

Conversation patterns and agent responses

Poor or lacking implementation of conversational patterns and agent responses, preventing the agent from functioning. (0)

Not all conversation patterns and agent responses that were instructed were properly implemented, disabling certain functionalities. (4)

All instructed conversational patterns and agent responses were properly implemented. (8)

8

Visuals

Poor or lacking implementation of visuals, preventing the agent from functioning. (0)

Not all instructed visuals were properly implemented, making some of the pages unclear. (4)

All instructed pages were properly implemented, and their information is clear to the user. (8)

8

Total points

36

Agent extensions

Criteria

Poor

Average

Good

Ratings

Max Pts

Task fulfillment

Little to no extensions to improve on the functionalities of the agent have been made. Extensions made hardly improve how users can find a recipe from the database. (0)

One or more extensions to improve on the functionalities of the agent have been made. They improve to some extent how users can find a recipe from the database, but were not very difficult to implement, are not all functioning well and/or are not very original. For example, extensions mostly copy a feature from the basic agent and replace one or two elements with something new. (4)

One or more extensions were made to the NLU, the filtering capacities of the agent and the way in which filtering outcomes are communicated visually and orally, that were not easy to implement and improve the way in which users can find a recipe from the database. (8)

8

Conversational competence

Little to no conversational patterns and agent responses were added. The added patterns cover rare or non-relevant interaction scenario’s and do not improve on conversational repair quality, nor add to more conversational freedom in the scenario of recipe selection. (0)

The added conversational patterns cover some additional directions that the conversations may take. Little is done to make conversational repair more effective. (3)

The added conversational patterns cover a proper variety of directions that the conversations may take, and the patterns and agent responses make for effective conversational repair strategies in case of misunderstanding (see the Extensions section for some pointers). (7)

7

Navigation and flexibillity

Little to no navigational features have been added to the agent. The features that were added are not working properly. (0)

Only some navigational features to improve on the flexibility of the agent are added (restarting, stopping or removing filters), or the added features do not all function / are unclear to the user. (3)

The agent enables the user to restart, stop and remove filters conversationally, and makes insightful to the user what options it has at any point in the conversation, by means of visuals and agent utterances. (7)

7

Agent design

Little to no extensions to improve on the design of the agent have been made to the visual support section or the conversation patterns and agent responses section. (0)

The agent has been given an own look and feel to some extent, but this is not very extensive and/or original, or limited to only the visuals or utterances of the agent. (4)

The group went all out to add an engaging and consistent look and feel to the agent visuals, utterances and conversational style. The chosen design is instrumental to recipe selection with a specific target audience in mind. (8)

8

Total points

30

Written report

Criteria

Poor

Average

Good

Ratings

Max Pts

Agent overview

The description of how the agent interacts is unclear and does not support a test of the different agent features. (0)

Some, but not all features of the agent are clear from the section on how the agent interacts, which makes it difficult to test some of its features. (2)

There is a complete and clear description of how the agent interacts, which supports

proper

testing of the different agent features. (4)

4

Design rationale

The design choices are difficult to understand from how they are presented. (0)

The design choices are written down in a mostly understandable way, while

some of the choices

are not well-motivated. (4)

The design choices are

clearly described and motivated and written down in an understandable and convincing way. (7)

7

Agent Testing

It is unclear from the

testing section

what data was collected during agent testing and what were the main insights. (0)

The

testing section gives a reasonable account of how the

agent was tested during development and what were the main findings. (4)

There is a clear presentation of

what results were obtained about the agent’s performance by testing throughout the project, with a clear set

of findings and

insights. (8)

8

User Study

It is unclear what results were obtained from performing the pilot user study

and how the

agent was performing. (0)

The

pilot user study results and analysis are reasonably well presented and it is clear what the main findings are. (4)

Results from the pilot user study about how the agent and its extensions performed are

thoroughly analyzed and

clearly presented in a structured way

. (8)

8

Clarity and presentation

The writing style, structure and lay-out are messy and unclear. (0)

The writing style and presentation are up to standards, but parts are unclear. (4)

Very clear report in terms of writing style, structure and lay-out. (7)

7

Total points

34