Assessment Rubric

Project MAS Rubric

Basic agent

Criteria

Poor

Average

Good

Ratings

Max Pts

Dialogflow intents and entities

Poor or lacking implementation of intents and entities, preventing the agent from functioning

Not all intents and entities that were instructed were properly implemented, disabling certain functionalities

All instructed intents and entities were properly implemented

Recipe filtering

Poor or lacking implementation of recipe filtering, preventing the agent from functioning

Not all instructed filtering functions were properly implemented, disabling certain functionalities

All instructed recipe filtering functions were properly implemented

Conversation patterns and agent responses

Poor or lacking implementation of conversational patterns and agent responses, preventing the agent from functioning

Not all conversation patterns and agent responses that were instructed were properly implemented, disabling certain functionalities

All instructed conversational patterns and agent responses were properly implemented

Visuals

Poor or lacking implementation of visuals, preventing the agent from functioning

Not all instructed visuals were properly implemented, preventing certain pages to properly render

All instructed pages were properly implemented

Total points

Agent robustness and usability

Criteria

Poor

Average

Good

Ratings

Max Pts

Robustness of intent recognition

Intents were trained with a wide coverage of possible user utterances and entities (if applicable), while the chance for confusion between intents was reduced to a minimum.

Robustness of conversation

The conversational patterns cover a proper variety of directions that the conversations may take, and the patterns and agent responses make for effective repair strategies (both user initiated and agent initiated) in case of misunderstanding.

Navigation and usability

The agent enables the user to restart, stop and remove filters conversation, and makes insightful to the user what options it has at any point in the conversation.

Total points

Agent extensions

Criteria

Poor

Average

Good

Ratings

Max Pts

Level of difficulty

No extensions have been implemented, or the extensions mostly comprise duplicating components of the basic agent.

Task fulfillment quality

Extensions were made to the filtering capacities of the agent and the way in which filtering outcomes are communicated visually and orally, that considerably improve the way in which users can find a recipe from the database

Agent design quality

Little to no extensions to improve on the design of the agent have been made to the visual support section or the conversation patterns and agent responses section. Extensions made hardly improve the agent design quality.

not orginal, or limited to visuals / conversaion patterns and agent responses

original

Total points

Written report

Criteria

Poor

Average

Good

Ratings

Max Pts

Agent architecture

Testing

It is unclear from the testing section how the assistant is performing. (0)

The testing section gives a reasonable account of how the assistant was tested and what were the main findings. (3)

The capacities and points of improvement of the assistant are evaluated in a structured way and clearly presented in the testing section.

(5)

Clarity and presentation

The writing style, structure and lay-out are messy and unclear. (0)

The writing style and presentation are up to standards, but some parts are unclear. (4)

Very clear report in terms of writing style, structure and lay-out. (8)

Total points