Assessment Rubric

Project MAS Rubric
Basic agent
Criteria	Poor	Average	Good	Ratings	Max Pts
Dialogflow intents and entities	Poor or lacking implementation of intents and entities, preventing the agent from functioning	Not all intents and entities that were instructed were properly implemented, disabling certain functionalities	All instructed intents and entities were properly implemented
Recipe filtering	Poor or lacking implementation of recipe filtering, preventing the agent from functioning	Not all instructed filtering functions were properly implemented, disabling certain functionalities	All instructed recipe filtering functions were properly implemented
Conversation patterns and agent responses	Poor or lacking implementation of conversational patterns and agent responses, preventing the agent from functioning	Not all conversation patterns and agent responses that were instructed were properly implemented, disabling certain functionalities	All instructed conversational patterns and agent responses were properly implemented
Visuals	Poor or lacking implementation of visuals, preventing the agent from functioning	Not all instructed visuals were properly implemented, preventing certain pages to properly render	All instructed pages were properly implemented
Total points
Agent robustness and usability
Criteria	Poor	Average	Good	Ratings	Max Pts
Robustness of intent recognition			Intents were trained with a wide coverage of possible user utterances and entities (if applicable), while the chance for confusion between intents was reduced to a minimum.
Robustness of conversation			The conversational patterns cover a proper variety of directions that the conversations may take, and the patterns and agent responses make for effective repair strategies (both user initiated and agent initiated) in case of misunderstanding.
Navigation and usability			The agent enables the user to restart, stop and remove filters conversation, and makes insightful to the user what options it has at any point in the conversation.
Total points
Agent extensions
Criteria	Poor	Average	Good	Ratings	Max Pts
Level of difficulty	No extensions have been implemented, or the extensions mostly comprise duplicating components of the basic agent.
Task fulfillment quality			Extensions were made to the filtering capacities of the agent and the way in which filtering outcomes are communicated visually and orally, that considerably improve the way in which users can find a recipe from the database
Agent design quality	Little to no extensions to improve on the design of the agent have been made to the visual support section or the conversation patterns and agent responses section. Extensions made hardly improve the agent design quality.	not orginal, or limited to visuals / conversaion patterns and agent responses	original
Total points
Written report
Criteria	Poor	Average	Good	Ratings	Max Pts
Agent architecture

Testing	It is unclear from the testing section how the assistant is performing. (0)	The testing section gives a reasonable account of how the assistant was tested and what were the main findings. (3)	The capacities and points of improvement of the assistant are evaluated in a structured way and clearly presented in the testing section. (5)

Clarity and presentation	The writing style, structure and lay-out are messy and unclear. (0)	The writing style and presentation are up to standards, but some parts are unclear. (4)	Very clear report in terms of writing style, structure and lay-out. (8)
Total points