Assessment Rubric

Project MAS Rubric
Basic agent
Criteria	Poor	Average	Good	Ratings	Max Pts
Dialogflow intents and entities	Poor or lacking implementation of intents and entities, preventing the agent from functioning	Not all intents and entities that were instructed were properly implemented, disabling certain functionalities	All instructed intents and entities were properly implemented
Recipe filtering	Poor or lacking implementation of recipe filtering, preventing the agent from functioning		All instructed recipe filtering functions were properly implemented
Conversation patterns and agent responses	Poor or lacking implementation of conversational patterns and agent responses, preventing the agent from functioning		All instructed conversational patterns and agent responses were properly implemented
Visuals	Poor or lacking implementation of visuals, preventing the agent from functioning		All instructed pages were properly implemented
Total points
Agent robustness
Criteria	Poor	Average	Good	Ratings	Max Pts
Robustness of intent recognition	The visuals merely display what the agent says at any point. (0)
Robustness of conversation
Navigation and repair
Total points
Agent extensions
Criteria	Poor	Average	Good	Ratings	Max Pts
Level of difficulty	No extensions have been implemented, or the extensions mostly comprise duplicating components of the basic agent.
Task fulfillment quality			Extensions were made to the filtering capacities of the agent and the way in which filtering outcomes are communicated visually and orally, that considerably improve the way in which users can find a recipe from the database
Agent design quality	Little to no extensions to improve on the design of the agent have been made to the visual support section or the conversation patterns and agent responses section. Extensions made hardly improve the agent design quality.	not orginal, or limited to visuals / conversaion patterns and agent responses	original
Total points
Written report
Criteria	Poor	Average	Good	Ratings	Max Pts
Introduction and Dialog engine

Testing	It is unclear from the testing section how the assistant is performing. (0)	The testing section gives a reasonable account of how the assistant was tested and what were the main findings. (3)	The capacities and points of improvement of the assistant are evaluated in a structured way and clearly presented in the testing section. (5)

Clarity and presentation	The writing style, structure and lay-out are messy and unclear. (0)	The writing style and presentation are up to standards, but some parts are unclear. (4)	Very clear report in terms of writing style, structure and lay-out. (8)
Total points