...
Project MAS Rubric | |||||
Basic agent | |||||
Criteria | Poor | Average | Good | Ratings | Max Pts |
Dialogflow intents and entities | Poor or lacking implementation of intents and entities, preventing the agent from functioning | Not all intents and entities that were instructed were properly implemented, disabling certain functionalities | All instructed intents and entities were properly implemented |
| |
Recipe filtering | Poor or lacking implementation of recipe filtering, preventing the agent from functioning | Not all instructed filtering functions were properly implemented, disabling certain functionalities | All instructed recipe filtering functions were properly implemented | ||
Conversation patterns and agent responses | Poor or lacking implementation of conversational patterns and agent responses, preventing the agent from functioning | Not all conversation patterns and agent responses that were instructed were properly implemented, disabling certain functionalities | All instructed conversational patterns and agent responses were properly implemented | ||
Visuals | Poor or lacking implementation of visuals, preventing the agent from functioning | Not all instructed visuals were properly implemented, preventing certain pages to properly render | All instructed pages were properly implemented |
| |
Total points |
| ||||
Agent robustness and usability | |||||
Criteria | Poor | Average | Good | Ratings | Max Pts |
Robustness of intent recognition | The visuals merely display what the agent says at any point. (0)Intents were trained with a wide coverage of possible user utterances and entities (if applicable), while the chance for confusion between intents was reduced to a minimum. |
| |||
Robustness of conversation | The conversational patterns cover a proper variety of directions that the conversations may take, and the patterns and agent responses make for effective repair strategies (both user initiated and agent initiated) in case of misunderstanding. |
| |||
Navigation and repair usability | The agent enables the user to restart, stop and remove filters conversation, and makes insightful to the user what options it has at any point in the conversation. | ||||
Total points |
| ||||
Agent extensions | |||||
Criteria | Poor | Average | Good | Ratings | Max Pts |
Level of difficulty | No extensions have been implemented, or the extensions mostly comprise duplicating components of the basic agent. |
| |||
Task fulfillment quality | Extensions were made to the filtering capacities of the agent and the way in which filtering outcomes are communicated visually and orally, that considerably improve the way in which users can find a recipe from the database |
| |||
Agent design quality | Little to no extensions to improve on the design of the agent have been made to the visual support section or the conversation patterns and agent responses section. Extensions made hardly improve the agent design quality. | not orginal, or limited to visuals / conversaion patterns and agent responses | original |
| |
Total points |
| ||||
Written report | |||||
Criteria | Poor | Average | Good | Ratings | Max Pts |
Introduction and Dialog engineAgent architecture |
| ||||
| |||||
Testing | It is unclear from the testing section how the assistant is performing. (0) | The testing section gives a reasonable account of how the assistant was tested and what were the main findings. (3) | The capacities and points of improvement of the assistant are evaluated in a structured way and clearly presented in the testing section. (5) |
| |
| |||||
Clarity and presentation | The writing style, structure and lay-out are messy and unclear. (0) | The writing style and presentation are up to standards, but some parts are unclear. (4) | Very clear report in terms of writing style, structure and lay-out. (8)
|
| |
Total points |
| ||||
...