...
Instructions for Your Report Submission
Your report should adhere to the following guidelines to ensure clarity and consistency:
Length: The report must not exceed 10 pages.
...
While additional materials can be included in appendices
...
, the
...
main
...
report
...
should be self-
...
contained and provide all necessary information for evaluation without requiring reference to the appendices. Markdown does not have pages necessarily. As a rule of thumb, one could say a page is about 500 words.
Submission: The final report must be uploaded to the main branch of your
...
Your report needs to have the following structure, and should comply with the maximum page limit indications for each section:
Title
Intro
How does your conversational agent work?
Intent and Slot Classifier - performance, extensions to model.
Exclusion - how, testing, comparison with and without exclusion models accuracy
Extensions to bot (not including hpo and model updates)
Pilot User Study
...
GitHub repository. Name the file Report.md.
Markdown Format: Use Markdown (
.mdformat) for your report. Markdown is a simple text formatting language widely used for documentation and README files. It allows for clean, structured text that can be easily viewed and edited.Updating Your Report on GitHub:
Navigate to the
Report.mdfile in your repository.Click the pencil icon to edit the file.
Make your changes, add a descriptive commit message, and click Commit changes to save.
Report Structure
Your report must follow the structure outlined below, with clear adherence to the page limits for each section.
1. Title (0.
...
25 page)
Content:
...
Add the report title.
...
Write a concise and descriptive title for your report.
Below the title, list your:
Group number
Student names
Emails
Student numbers
Notes:
Make the title eye-catching and informative to immediately communicate the essence of your project
...
2. Introduction (
...
0.75 page)
Content:
...
Introduction to the
...
Project:
...
Define a conversational recipe recommendation agent
...
and its purpose.
Introduce Task-Oriented Spoken Dialogue Systems
...
Summarize Preliminaries: Describe what definitions and knowledge you utilized to complete this project.
...
State the goals your team aimed to achieve.
Tips:
Ensure clarity and conciseness. This section sets the stage for the reader.
Provide some background on the significance of conversational agents in recipe recommendations.
...
(TOSDS), emphasizing their role in handling structured tasks like recipe recommendations.
Goals:
State your team’s objectives clearly. Examples:
Build an agent capable of personalized recipe recommendations.
These should be concrete and specific goals to your version of the agent.
Etc.
...
3. How Does Your Conversational Agent Work? (2 pages)
Content:
...
Describe your pipeline, and give an overview.
...
Explain the overall functionality of your agent.
What problems does it solve?
What can users achieve by interacting with the agent?
...
Functionality:
Highlight the agent’s primary use cases.
Conversational Flow:
Walk through a typical user interaction.
Explain how user queries are processed through intent recognition, slot filling, and database queries.
Examples: Give us typical interactions that would work with your agent. Show us what your agent can do by giving examples that illustrate your agent's capabilities. Note: these examples should be useful for testing (we may try them out).
...
4. Intent and Slot Classifier (1-2 pages)
Content:
Intent and Slot Classifier:
Explain the
...
Summarize training and testing
Performance analysis:
...
importance of these components in identifying user intentions and extracting relevant information (e.g., cuisine type, dietary restrictions).
Performance Analysis:
Present metrics:
Accuracy, precision, recall, F1 score
...
.
Use tables or confusion matrices to compare results across iterations.
Discuss challenges faced (e.g.
...
, ambiguous intents, overlapping slots).
Discuss the comparison of your results and the Intent and Slot Classifier Evaluation Thresholds
Model Improvements:
Mention improvements like:
Use of
...
pre-trained models
...
Tips:
Include visual aids such as tables or charts to present performance data effectively.
Highlight innovative solutions your team implemented.
(e.g., BERT or GPT-based embeddings).
Hyperparameter tuning, training methodology augmentation, and architectural modifications.
...
5. Exclusion (2 pages)
Content:
Implementation:
How exclusion works (e.g., filtering excluding ingredients, cuisines, and tagsmealTypes).
Approaches used: intent-based, slot-based, rule-based, classifier-basedDescribe the approach your team used to implement Exclusion into your model.
Tools/technologies: Integration of MARBEL, Prolog, Python, and ontology updates.
TestingPros and Cons:
Comparison of inclusion-only vs. exclusion models.
Examples of exclusion in action (e.g., excluding dairy, gluten, or specific cuisines).Discuss the strengths and limitations of the exclusion mechanism you choose. What can your exclusion do, what can it not do?
Performance Analysis:
Accuracy with and without exclusion.
Trade-offs and impact on user satisfaction.
Pros and Cons:
Discuss strengths and limitations of the exclusion mechanismComparison of inclusion-only vs. exclusion models.
Tips:
Use examples and data to illustrate the effectiveness of the exclusion approach.
Be critical and discuss what could be improved.
...
6. Extensions to the
...
Pipeline (1 page)
Content:
Summarize the work done related to Extensions.
Describe additional enhancements:
New functionalities, filters, capabilities added.
Improvements in user interaction and experience (e.g., better response generation, conversational adaptability).
Explain the motivation and impact of these extensions.
...
Highlight how these extensions make the bot agent stand out beyond baseline requirements.
...
7. Pilot User Study (1
...
page)
Content:
Setup:
Who were the users?
What tasks were they asked to perform?
Methodology for data collection (e.g., surveys, observation, interaction logs).
Results from the Study:
Simple descriptive statistics gathered in the Pilot User Study
Quantitative data: Success rates, error rates, average response time, etc.
Qualitative data: User feedback, observations of user behavior.
Analysis:
Key findings: What worked well? What needs improvement?
Lessons learned and implications for future work.
Tips:
Use You do not need to use charts or graphs to present quantitative findings.
Include excerpts from user feedback to illustrate qualitative insights.
...
8. Conclusion (1 page)
Content:
Summarize the main outcomes of the project.
Reflect on the process:
What went well (e.g., teamwork, innovative solutions)?
What could be improved (e.g., time management, data quality)?
Suggest future improvements or extensions.
...
Keep it concise but reflective.
Focus on high-level takeaways and actionable insights for future work.
Title
Add a title. Add your group number, student names, emails, and student numbers right below the title.
Introduction (max 1 page)
Briefly introduce your conversational recipe recommendation agent project, and describe the main goals you as a team have set yourselves to achieve.
How does your conversational agent work? (max 3 pages)
...
work
...
Design choices and rationale (max 3 pages)
In this section, you should describe which design choices you have made while implementing your agent. Design choices should include the more important choices that you made for developing your Dialogflow agent (NLU), for your MARBEL Dialog manager (patterns, responses, recipe database logic), and for what your agent displays while talking (visuals, webpages). You should also describe which extensions you choose to implement. Make sure to not only describe these design choices, but also motivate why you made these choices.
Test results and discussion (max 4 pages)
In this section, you should talk about the findings of testing your own agent. What did you find? There are two sets of data that you should discuss: 1. The data your team collected while developing your conversational agent during [TBU]Pipeline Testing; 2. The data your team collected in the [TBU]Pilot User Study (where fellow students tested your agent).
You should briefly present and analyze the results from both datasets. What were the main findings? What does the data show about how well your agent performed? What issues did they uncover? What does the data tell about your Dialogflow agent, about your Dialog manager, and about the pages you developed? What kind of results did you obtain from your pilot user study? Present both quantitative (numbers, figures, surveys) and qualitative (observations, feedback) data.
You should next try to explain, interpret and discuss the data. What insights and lessons learned can you draw from the data you collected yourselves and from the pilot user study? What does the data say about your design choices and the extensions that you implemented? Can you say anything about how these impacted the performance of your agent? What does the data tell about the effectiveness, efficiency, robustness of your agent, and what users (from your pilot user study) think about your agent (satisfaction).
Conclusion (max 1 page)
...
.