...
Project MAS Rubric | |||||||||||||||
Basic agent | |||||||||||||||
Criteria | Poor | Average | Good | Ratings | Max Pts | ||||||||||
Dialogflow intents and entities | Poor or lacking implementation of intents and entities, preventing the agent from functioning. (0) | Not all intents and entities that were instructed were properly implemented, disabling certain functionalities. (3) | All instructed intents and entities were properly implemented. (5) |
| 5 | ||||||||||
Robustness of intent recognition | Intents were trained with a minimum of training phrases, enabling understanding very specific. (0) | Only part of the intents were implemented with a wide coverage of possible user utterances, or the implementation of the intents was not made robust enough against confusing one intent with another. (3) | Intents were trained with a wide coverage of possible user utterances and entities (if applicable), while the chance for confusion between intents was reduced to a minimum. (7) |
| 7 | ||||||||||
Recipe filtering | Poor or lacking implementation of recipe filtering, preventing the agent from functioning. (0) | Not all instructed filtering functions were properly implemented, disabling certain functionalities. (4) | All instructed recipe filtering functions were properly implemented. (8) | 8 | |||||||||||
Conversation patterns and agent responses | Poor or lacking implementation of conversational patterns and agent responses, preventing the agent from functioning. (0) | Not all conversation patterns and agent responses that were instructed were properly implemented, disabling certain functionalities. (3) | All instructed conversational patterns and agent responses were properly implemented. (7) | 7 | |||||||||||
Visuals | Poor or lacking implementation of visuals, preventing the agent from functioning. (0) | Not all instructed visuals were properly implemented, preventing certain pages to properly render. (4) | All instructed pages were properly implemented |
| Robustness of intent recognition | Intents were trained with a wide coverage of possible user utterances and entities (if applicable), while the chance for confusion between intents was reduced to a minimum. . (8) |
| 8 | |||||||
Total points |
| 4035 | |||||||||||||
Agent robustnessextensions | |||||||||||||||
Criteria | Poor | Average | Good | Ratings | Max Pts | Richness of conversation | The conversational patterns cover a proper variety of directions that the conversations may take, and the patterns and agent responses make for effective repair strategies (both user initiated and agent initiated) in case of misunderstanding. |
| Navigation and flexibillity (can you undo stuff) | The agent enables the user to restart, stop and remove filters conversation, and makes insightful to the user what options it has at any point in the conversation. | Total points |
| |||
Agent extensions | |||||||||||||||
Criteria | Poor | Average | Good | Ratings | Max Pts | ||||||||||
Task fulfillment quality | Extensions were made to the Task fulfillment | Little to no extensions to improve on the functionalities of the agent have been made. Extensions made hardly improve how users can find a recipe from the database. (0) | One or more extensions to improve on the functionalities of the agent have been made. They improve to some extent how users can find a recipe from the database, but were not very difficult to implement, are not all functioning well and/or are not very original. For example, extensions mostly copy a feature from the basic agent and replace one or two elements with something new. (4) | One or more extensions were made to the NLU, the filtering capacities of the agent and the way in which filtering outcomes are communicated visually and orally, that were not easy to implement and considerably improve the way in which users can find a recipe from the database. (8) |
| 8 | |||||||||
Richness of conversation | Little to no conversational patterns and agent responses were added. The added patterns cover rare or non-relevant interaction scenario’s and do not improve on conversational repair quality, nor add to more conversational freedom in the scenario of recipe selection. (0) | The added conversational patterns cover some additional directions that the conversations may take. Little is done to make conversational repair more effective. (3) | The added conversational patterns cover a proper variety of directions that the conversations may take, and the patterns and agent responses make for effective repair strategies (both user-initiated and agent-initiated) in case of misunderstanding. (7) |
| 7 | ||||||||||
Navigation and flexibillity (can you undo stuff | Little to no navigational features have been added to the agent. The features that were added are not working properly. (0) | Only some navigational features to improve on the flexibility of the agent are added (restarting, stopping or removing filters), or the added features do not all function / are unclear to the user. (3) | The agent enables the user to restart, stop and remove filters conversationconversationally, and makes insightful to the user what options it has at any point in the conversation, by means of visuals and agent utterances. (7) | 7 | |||||||||||
Agent design quality | Little to no extensions to improve on the design of the agent have been made to the visual support section or the conversation patterns and agent responses section. Extensions made hardly improve the agent design quality. not orginal (0) | The agent has been given an own look and feel to some extent, but this is not very extensive and/or original, or limited to visuals / conversaion patterns and agent responses original only the visuals or utterances of the agent. (4) | The group went all out to add an engaging and consistent look and feel to the agent visuals, utterances and conversational style. The chosen design is instrumental to recipe selection with a specific target audience in mind. (8) |
| 8 | ||||||||||
Total points |
| 30 | |||||||||||||
Written report | |||||||||||||||
Criteria | Poor | Average | Good | Ratings | Max Pts | ||||||||||
Agent architecture |
| ||||||||||||||
| |||||||||||||||
Testing | It is unclear from the testing section how the assistant is performing. (0) | The testing section gives a reasonable account of how the assistant was tested and what were the main findings. (3) | The capacities and points of improvement of the assistant are evaluated in a structured way and clearly presented in the testing section. (5) |
| |||||||||||
| |||||||||||||||
Clarity and presentation | The writing style, structure and lay-out are messy and unclear. (0) | The writing style and presentation are up to standards, but some parts are unclear. (4) | Very clear report in terms of writing style, structure and lay-out. (8)
|
| |||||||||||
Total points |
| 35 | |||||||||||||
...