Agent Testing and User Study

Open

Evaluation is an essential step to evolve your recipe recommendation agent from an agent that can conduct a basic conversation that offers minimal functionality to a user to an agent that can provide a great user experience for all users. Effectiveness, efficiency, robustness, and user satisfaction are important metrics here. An agent is effective if it can guarantee task completion. Such an agent adequately assists a user with completing the task at hand. It is efficient if the number of turns needed to achieve the goal in a conversation is optimized. It is robust if it can handle all phrases that users might reasonably be expected to say and can handle the different directions that a conversation may take. Finally, a conversational agent is satisfying if a user experiences the conversation as satisfying.

By testing, you can find out the deficiencies of the agent that need to be addressed so that it will not break down easily. This is typically done in an agile manner, starting with a simple agent that functions minimally, where rounds of tests and improvements follow one another. You will follow two stages in the current project.

First, you will focus on agent testing:

Second, at the end of the project, you will do a user study:

You will need to document agent testing and the user study that your team performed in your final report.