Assessment Rubric

Project MAS Rubric

Basic agent

Criteria

Poor

Average

Good

Ratings

Max Pts

Dialogflow intents and entities

Poor or lacking implementation of intents and entities, preventing the agent from functioning

Not all intents and entities that were instructed were properly implemented, disabling certain functionalities

All instructed intents and entities were properly implemented

Recipe filtering

Poor or lacking implementation of recipe filtering, preventing the agent from functioning

All instructed recipe filtering functions were properly implemented

Conversation patterns and agent responses

Poor or lacking implementation of conversational patterns and agent responses, preventing the agent from functioning

All instructed conversational patterns and agent responses were properly implemented

Visuals

Poor or lacking implementation of visuals, preventing the agent from functioning

All instructed pages were properly implemented

Total points

Agent robustness

Criteria

Poor

Average

Good

Ratings

Max Pts

Robustness of intent recognition

The visuals merely display what the agent says at any point. (0)

Robustness of conversation

Navigation and repair

Total points

Agent extensions

Criteria

Poor

Average

Good

Ratings

Max Pts

Level of difficulty

No extensions have been implemented, or the extensions mostly comprise duplicating components of the basic agent.

Task fulfillment quality

Extensions were made to the filtering capacities of the agent and the way in which filtering outcomes are communicated visually and orally, that considerably improve the way in which users can find a recipe from the database

Agent design quality

Little to no extensions to improve on the design of the agent have been made to the visual support section or the conversation patterns and agent responses section. Extensions made hardly improve the agent design quality.

not orginal, or limited to visuals / conversaion patterns and agent responses

original

Total points

Written report

Criteria

Poor

Average

Good

Ratings

Max Pts

Introduction and Dialog engine

Testing

It is unclear from the testing section how the assistant is performing. (0)

The testing section gives a reasonable account of how the assistant was tested and what were the main findings. (3)

The capacities and points of improvement of the assistant are evaluated in a structured way and clearly presented in the testing section.

(5)

Clarity and presentation

The writing style, structure and lay-out are messy and unclear. (0)

The writing style and presentation are up to standards, but some parts are unclear. (4)

Very clear report in terms of writing style, structure and lay-out. (8)

Total points