...
Written Report | ||||||
|---|---|---|---|---|---|---|
Criteria | Poor | Average | Good | Excellent | Max Points | |
Title and Formatting | Title and formatting are unclear or incomplete, missing required elements (0). | Title and formatting are present but lack clarity or professionalism (0.5). | Clear title and formatting; minor improvements needed (1). | Concise and professional title, all required elements are clear and complete (1.5). | 1.5 | |
Introduction | Introduction lacks clarity and fails to define the project, goals, or key terms (0). | Basic introduction provided but lacks depth or proper framing of goals and context (1). | Clear and informative introduction with a good explanation of goals and context (2). | Strong, engaging introduction with a clear definition of project goals, context, and significance (3). | 3 | |
Pipeline Explanation | Pipeline How does your agent work? | Pipeline description is vague and lacks detail; user interaction is unclear (0). | Basic pipeline description provided but missing depth or key details (1.5). | Well-detailed pipeline with clear functionality and conversational flow explained (3). | Comprehensive, clear, and visually aided pipeline description that is easy to understand (4). | 4 |
Intent and Slot Classifier | Explanation of intent and slot classifier is missing or vague; no performance analysis (0). | Basic explanation of intent and slot classifier with minimal performance data (1.5). | Detailed explanation with good performance analysis and discussion of challenges (3). | Comprehensive explanation, with strong performance analysis, detailed metrics, and innovative extensions (4). | 4 | |
Exclusion Mechanism | Exclusion mechanism is unclear, with no testing or pros and cons analysis (0). | Basic explanation provided, but lacks clarity in implementation and testing (1.5). | Clear explanation with testing and pros/cons analysis, but room for improvement (3). | Thorough explanation of implementation, testing, pros/cons, and strong performance data (4). | 4 | |
Extensions to the Bot | Extensions are unclear or not described; impact is not evident (0). | Extensions described but lack depth or clear motivation (1). | Well-documented extensions with clear motivation and impact analysis (1.5). | Comprehensive description of innovative extensions with clear benefits and motivations (2).2 | 4 | |
Pilot User Study | User study setup and results are missing or unclear (0). | Basic user study presented with limited results and insights (2). | Well-structured user study with good results and analysis of findings (3). | Detailed, well-analyzed user study with strong quantitative and qualitative insights (4). | 4 | |
Conclusion | Conclusion is missing or vague, with no reflection or future suggestions (0). | Basic summary provided but lacks depth or critical reflection (1). | Clear conclusion with reflection and practical improvement suggestions (1.5). | Strong, insightful conclusion with critical reflection and actionable improvement ideas (2).2 | 4 | |
Clarity and Presentation | Writing is unclear and poorly structured; formatting is messy (0). | Writing is somewhat clear but lacks polish and structure (1). | Clear and well-structured writing with minor presentation issues (2). | Very clear, professional writing with excellent structure and layout (3).3 | 6 | |
Total Points | 30 |
Extension and Exclusions | |||||
|---|---|---|---|---|---|
Criteria | Poor | Average | Good | Excellent | Max Points |
Exclusion Implementation | Exclusion functionality is missing, incomplete, or unable to effectively be used at all. Little collaboration or effort to integrate exclusion approaches is evident. (0) | Exclusion functionality includes basic capabilities such as excluding a single type of slot but lacks refinement. Limited approach or minimal testing and analysis. (4) | Exclusion functionality works well for at least 2 slots. Multiple changes in the pipeline are made and combined effectively. Testing and trade-offs are adequately considered. (7) | Exclusion is comprehensive and well-integrated, handling multiple slots. Can be combined effectively with inclusion. Approaches are thoughtfully combined and tested rigorously, and limitations are clearly minimized. Collaboration across team members and sections is evident. (10) | 10 |
Extensions to Agent Functionality (Recipe Filtering) | Little to no extensions to improve agent functionality were implemented. Extensions do not significantly enhance user experience or recipe filtering. (0) | Extensions improve functionality but are minimal, lack originality, or are not fully operational. They may replicate basic agent features with minor variations. (3) | Extensions improve key aspects such as NLU, filtering, and visual/aural communication. They are moderately complex and enhance user interaction meaningfully. (5) | Extensions are innovative, complex, and significantly enhance agent functionality, filtering, and user interaction. They demonstrate originality and are well-implemented. (8) | 8 |
Conversational Competence and Navigation (Extension to Dialogue Patterns) | Few or no conversational patterns, repair strategies, or navigational features are added. Added features (e.g., restarting, stopping filters) are non-functional or unclear. (0) | Basic conversational patterns and navigational features (e.g., restarting or stopping filters) are added, but they lack depth, are inconsistently functional, or fail to address misunderstandings effectively. (3) | Conversational patterns cover relevant scenarios, and navigational features (e.g., restarting, stopping, removing filters) work reliably. Basic repair strategies and improved conversational flow are present. (5) | Conversational patterns and navigational features are well-integrated and intuitive. Patterns handle misunderstandings effectively, and navigation options (e.g., restarting, stopping, removing filters) are user-friendly, enhancing flexibility and user experience. (8) | 8 |
Design and User Engagement (Extension to Visual Support) | Little or no effort to improve agent design in terms of visuals, utterances, or conversational style. (0) | Basic design improvements are present but lack originality, cohesion, or a clear target audience focus. Limited to either visuals or utterances. (2) | Agent design is engaging and consistent, with clear visual and conversational improvements tailored to a general audience. (3) | Design is highly engaging, cohesive, and tailored to a specific target audience. Visuals, utterances, and conversational style enhance usability and align with recipe selection goals. (4) | 4 |
Total Points | 30 |
...