In the small pilot user study at the end of the project, in the last week, you will test an agent from another team and another team will test yours! Check out the Course Schedule for when this pilot study is planned.
Preparation: By this time, you should have an agent that is ready for final evaluation. For your agent to be ready for the pilot user study evaluation, you need to make sure that:
...
Setting up your user study: Your team will run the pilot user study for your own agent and collect what you think is relevant data to analyze the performance of your agent. Consider some of the metrics that were already explained on the Agent Testing and Pilot User Study https://socialrobotics.atlassian.net/wiki/pages/createpage.action?spaceKey=PM2&title=2025%20Agent%20Testing%20and%20Pilot%20User%20Study page: effectiveness, efficiency, robustness, and user satisfaction. So think about what you want to evaluate and how to do that. Also, consider that the greatest advantage of having another team test your agent is that they likely will interact differently and may have different conversations with your agent than you have seen before in your own agent testing (that’s why it makes sense to already involve people from outside your team during agent testing).
Reporting: In your Final Report https://socialrobotics.atlassian.net/wiki/pages/createpage.action?spaceKey=PM2&title=2025%20Final%20Report you should focus on the analysis of the data that you collected. You should have collected data for close to 20 conversational interactions with your agent. For these conversations, you can report some of the more interesting and basic https://en.wikipedia.org/wiki/Descriptive_statistics. Descriptive statistics provide simple summaries about the sample (your participants). You do not need to provide figures or tables (you won’t have much space for that in your report; If you want to provide details you can add these in an appendix to your report) but should focus on the more interesting findings. Most importantly, briefly discuss and interpret the data that you collected to explain what the data can tell us about the performance of your agent.
...