As indicated on the Project Background Knowledge page, the main purpose of providing visual support is twofold: to provide (1) support to a user to reduce their cognitive load (the amount of information working memory needs to process at any given time) and (2) an indication of the progress on the task that has been made thus far. Visual support will be provided by means of a webpage. This page aims at explaining how you can develop code for your MARBEL agent to create such a webpage.

Using Prolog to build HTML elements in our agent

It is important to be able to synchronize the conversational interaction and the visuals displayed on the webpage shown while talking. The MARBEL agent manages the dialog, and for that reason also is the logical place to locate the control over the visual elements. This agent uses a renderPage(Html) action in the dialog_generation.mod2g module to render a webpage; when performing this action, the Prolog variable Html therefore should be instantiated with HTML webpage code. All we need to know before we can use this action, is how to create this HTML page in Prolog.

The basic idea is that we will create essentially a single string that consists of the HTML webpage code. We will utilize a combination of Prolog, HTML and Bootstrap to generate the code for dynamic webpages. Prolog rules are used to generate our HTML code. In other words, the HTML code represented as a Prolog atom, essentially a string, is manipulated with Prolog. So what does that look like? We will first take a look at how to construct some very basic HTML code, then explain how we can add more complicated HTML code that is also using Bootstrap components, and, finally, how we can piece together these components into a complete webpage.

Generating basic HTML elements with short tags

An HTML page consists of HTML elements. HTML elements consists of HTML tags that are used to organize content on that page. Tags often come in pairs of a start and end tag. Basic HTML tags, for example, are the tags <p>YOUR PARAGRAPH TEXT HERE</p> that defines an HTML paragraph, the <h1>...</h1> defining a large heading, <b>...</b> defining bold text, etc. There are many of these basic tags which, moreover, are also short which facilitates their frequent and easy use. The idea to generate basic HTML code for these elements in Prolog is very simple: Simply use single quotes '...' to generate such an element. The table next lists a few examples:

There are many simple, short tags where it is easiest to simply put them between single quotes to create a Prolog atom for generating the corresponding HTML code. Another class of short tags are the single tags such as '<br>' for breaking a line and '<hr>' for adding a horizontal line. These are also most easily generated by simply quoting them.

Generating more complex HTML elements with attributes

Most HTML tag pairs can be simply generated by using a start and end tag and by adding some text to create the corresponding HTML. For example, <button>Start</Button> can be used to generate a simple button saying Start. However, to unleash the full power of the HTML language, it is often useful to modify the type or style of an HTML element by adding additional attributes in the form of name="value" to the start tag.

Making a button

For example, the following HTML code creates a large, colored button using the class attribute and several button class values made available by Bootstrap:

<button class="btn btn-secondary btn-lg">Start</button>

Here, Start between the start and end tag is the content of the HTML element, button is used for the start and end tags to create a button element, and the class attribute uses the values btn-secondary to style the button and btn-lg to create a larger button.

Of course, we can still create this HTML code in Prolog by simply quoting it. And for a simple button this may still most often be quite OK. This is because the content we typically want to fit inside of a button element is often just some simple text. But it will be useful to allow for a more generic approach for more advanced HTML elements in which we want to embed other more complex HTML code as content. To illustrate our approach to generating HTML elements with attributes in Prolog we therefore begin with introducing a predicate button/3 for generating HTML code for a button.

In the button example above, we illustrated the overall approach and structure we will use for generating HTML code for elements with attributes but did not include the attribute part yet. We will now also add that to complete the button example. For the button example we will only include one attribute class. We thus end up with a button/3 predicate with three arguments button(Content, Class, Html). Predicates for other HTML tags, such as the div tag, for example, where we would also want to add a style attribute, would have 4 arguments. To conclude our button example, to generate the <button class="btn btn-secondary btn-lg">Start</button> code, we can use the Prolog code

button('Start', 'btn btn-secondary btn-lg', Html)

When evaluating this clause, we would end up with Html having the button HTML code as its value.

Changing and receiving button events in a MARBEL agent

If you would inspect the html.pl file and search for the button predicate, you will find a button/2 predicate too. You will also see that both the button/2 and the button/3 predicates actually are defined in terms of a button/4 predicate button(Content, Class, DataValue, Html). This predicate allows to set the value send as a percept to the agent when the button is clicked. The button/3 definition by default sets this value equal to the Content value displayed on the button. That, however, is not that useful if the content itself consists of HTML tags again. For example, button('<b>Start</b>', 'btn btn-secondary btn-lg', Html) would display a bold text Start but would also send the tags as a percept of the form answer('<b>Start</b>') when the button is clicked. To avoid that, we can use button('<b>Start</b>', 'btn btn-secondary btn-lg', 'start', Html) which would send the more plain and easy to process answer('start') to the MARBEL agent.

Other tag examples: ul, img, div, span

We introduce a few other tags and discuss how HTML code for these tags can be generated using some of the predefined predicates in the html.pl file. These examples should provide you with a general understanding of how you can generate HTML code for a variety of HTML elements.

ul tag

Displaying lists is a practical feature when you want to show a recipe and its associated ingredient list. The most basic way to do this is to use the unordered list ul and list item li tags. Here’s a simple example of what the HTML code would look like:

<ul class="list-group">
  <li class="list-group-item">1st item</li>
  <li class="list-group-item">2nd item</li>
  <li class="list-group-item">3rd item</li>
</ul>

We have kept the class attribute simple but Bootstrap list groups provides several other values for the class attribute that can be used to change the basic behavior of the ul tag. For example, use list-group-flush for removing the borders and list-group-horizontal for organizing the list items horizontally instead of vertically.

Even though ul is a short tag we still have introduced a corresponding ul/3 predicate ul(Content, Class, Html) because of the embedded content of list items that needs to be supplied and fit in between the start and end tags in this case. Using this predicate we can simply use ul(Content, 'list-group', Html) for generating the unordered list example above. Clearly, a simple quotation approach will not work here as the list of items may easily get very long and unyieldy and upfront we may not even know how many items we need to add to the list (how many ingredients, for example, does a recipe have?). To facilitate the creation of the HTML list item elements, we have provided the predicate itemList(List, Html) that generates the list item elements when provided with a list of the content for each item. For our example, itemList(['1st item', '2nd item', '3rd item'], Html) will generate what we need as content for the ul element.

The final step that remains is to piece the two things together to generate the complete HTML code for our example. For generating HTML code, we will use the fact that Prolog evaluates subqueries in a query in the order they appear. As a general rule of thumb, in this content, we should make sure to generate the HTML elements we need later first. That is, because we need the list item elements as content to complete the unordered list code, we need to create this code first. In other words, we will use the output argument of itemList/2 as the input argument for the content of ul/3. By piecing the separate queries we defined for the list itself and for the list items together and renaming the appropriate argument variables, we get what we need:

itemList(['1st item', '2nd item', '3rd item'], ListItems), ul(ListItems, 'list-group', Html)

This combined query will generate our example HTML code and return it as value in the Html variable.

img tag

Although this project is not about webpage development and is definitely not about promoting fancy interaction (remember to prioritize conversational interaction throughout the project!), by simply adding some images you can bring a webpage to life, making it more appealing to look at while also adding useful functionality. To start with the latter, users would like to see what the result would look like for the recipe they finally choose. To this end, links to the pictures of all of the recipes included in the recipe database have been included for you to make good use of (check out the recipe_database.pl file). But there are many other opportunities, of course, to add pictures if you like to your webpage design. To facilitate this, we have defined the img/4 predicate img(Source, Class, Style, Html) which allows you to create an img HTML element that looks like:

<img class="CLASS" style="STYLE" src="SOURCE" alt="Can not show this image">

Check out some of the class options here. The style attribute can be used, for example, to display an image inline (e.g., as part of a sentence in a paragraph) with style="display: inline;".

The Source argument for the predicate is a required input argument. You can use links to pictures that are online available here. For your convenience, for all of the recipes included in the database, the picture/2 predicate picture(RecipeNr, UrlLink) provides images for each recipe. As an example, to illustrate the use of the both picture/2 and img/4 predicates in a combined query, picture('1', Source), img(Source, 'img-thumbnail', 'height: 3rem', Html) would generate the following HTML element:

<img class="img-thumbnail" style="height: 3rem" src="https://mobkitchen-objects.imgix.net/recipes/849A3215-2.jpg?auto=format&crop=focalpoint&domain=mobkitchen-objects.imgix.net&fit=crop&fp-x=0.5&fp-y=0.5&h=827&ixlib=php-3.3.1&q=82&w=1300&s=e97db51272669560eff9df6badbe8bc7" alt="Can not show this image">

which would show a small thumbnail version of the first recipe picture

for sweet and spicy gochujang fried chicken.

Alternatively, instead of providing an url to an online image as the source input argument, you can also add image Base64 codes to the Prolog html.pl file and use those. Examples for variants of a microphone icon have already been included in this file. E.g., the mic_image_closed_src(Source) query will instantiate the Source variable with the Base64 code for the closed microphone icon

You can add your own images to the html.pl file by encoding them to Base64 format too. You can do so by following these steps:

1. Go to the https://www.base64encode.org/ site (don’t change any options on that page)

2. Go to the Encode files to Base64 format section

3. Upload an image (note that file size cannot exceed 192MB!)

4. Press the encode button, and

5. Download the code as a text file.

6. Copy the text in the file.

7. Add a fact with a predicate name for your image of your choosing (to illustrate we used myImageName) by adding the following template to the html.pl file: myImageName('data:image/png;base64,...')

8. Paste and replace the ... in the Prolog fact with the text you copied from the file (a very long encoded string representing your picture)

9. Now you can use myImageName(Source) instead of the picture(RecipeNr, Source) query to include your picture on your webpage

div tag

The div tag is a versatile block element that can be used to organize and style HTML content in various ways. In the html.pl file provided to you we have defined a div/4 predicate div(Content, Class, Style, Html) that facilitates the generation of div elements with class and style attributes that you can set. The code that will be generated looks simply like a basic div element without any content yet:

<div class="CLASS" style="STYLE">CONTENT</div>

In this code template, CLASS refers to the value of the Class variable, STYLE to the value of the Style variable, and CONTENT to the value of the Content variable in the div(Content, Class, Style, Html) query. All of these variables need to be fully instantiated to generate HTML code that is returned in the output argument Html.

Many Bootstrap components are classes that can be used by means of the div tag. These include, for example, the container, jumbotron, alert, progress bar, spinner, card, flexbox, media, and grid system row and column classes. Check out all of these components in the W3 School tutorial or Bootstrap’s own overview. We provide one example to illustrate the use of the div predicate. The following code would generate a Jumbotron element:

div(Content, 'jumbotron jumbotron-fluid px-3', 'text-align:center', Html)

Of course, this only works if you add another query (or queries) to generate the Content for this Jumbotron. To add couple of lines of formatted text (a large title head and a paragraph), we could use the following code for creating a complete Jumbotron element with some text as a simple example:

atomic_list_concat([
  '<h3>Introduction</h3>',
  '<p>You are about to interact with our conversational agent <b>YourAgentName</b>.'],
Content),
div(Content, 'jumbotron jumbotron-fluid px-3', 'text-align:center', Html)

The first query, using the built-in predicate atomic_list_concat/2 generates HTML code that consists of the two HTML elements of a h3 heading following by a paragraph. These elements are just simply quoted each (remember to always use single quotes!) and put in that order in a Prolog list [...]. The atomic_list_concat/2 predicate then returns a new atom that is the concatentiation of these two elements in the output argument Content. This (quoted) HTML code then is passed on to the div/4 query in its first input argument (using the same variable name) to embed it in the Jumbotron element.

The resulting HTML code looks like:

<div class="jumbotron jumbotron-fluid px-3" style="text-align:center">
  <h3>Introduction</h3>
  <p>You are about to interact with our conversational agent <b>YourAgentName</b>.</p>
</div>

span tag

The span tag is an inline element that is commonly used to style HTML content. We have added a span/3 predicate span(Content, Style, Html) that just does that and nothing else. An example of a span tag that can be useful is to style the content and change the font family that is used within some HTML content:

<span style="font-family:Roboto;">YOUR HTML CONTENT HERE</span>

A list of system or web-safe fonts that you can choose from (supported on most devices) can be found here.

The span tag has other uses too. It can be used for adding Bootstrap badges and borders to elements, for example. We have also used it for setting the value of a button in the template we used for defining the button/4 predicate.

Generating an HTML webpage

Prolog rules are used to add a condition to a webpage (i.e. webpage X is shown when Clause Y is true).

You add a condition that checks for a particular pattern ID to a Prolog rule in the html.pl file. Other simple conditions that you can add involve counting the number of recipes that are still available while in the recipe selection phase, etc. The general form of the page lay-out rules would be something like:

myPage(Txt, Button, Html) :-
  <HERE YOUR CONDITION FOR SHOWING THIS PAGE>, 
  % e.g. while recipe is being selected (check for e.g. currently active top level pattern id), we'll display recipe features
  % below code that specifies the page layout for this context
  applyTemplate('<div class="card mx-auto" style="width:67vw">~a</div>, Text, Html),
  …
  .

There are still many options to vary. For example, you can choose the parameters for the myPage predicate that fit your approach best.

Example Prolog Rule for a Page

page(c10, _, Html) :-
	% Condition for when to show this page
	CONDITION X,
	% Constructing HTML page
	atomic_list_concat(['<div class="alert alert-light"><center></br><h1>Hello!</br></br>','I am ~a</h1></br></br></center></div>'], Template),
	% Get the bot's name if it has one; other call it 'your assistant'
	(agentName(Name) -> N = Name ; N = 'your recipe selection assistant'), applyTemplate(Template, N, Body),
	% Create the HTML page
	html(Body, Html).

Adding a chatbox to your page layout for testing purposes

The speech interface is fitting to the context of a user in a kitchen, but not always feasible for testing. It may take additional time for the system to process speech input, you may be in a noisy environment, and there may be speech recognition failures. Hereby a pointer to easily enable a textbox to be rendered for input.

In the html.pl file, where you format the pages to render during the conversation, for each of the pages that you may display at any point in the conversation add the following code:

A chatbox can be added for using text instead of speech for input by adding
<div class="text-center"><p class="chatbox mx-auto"></p></div>, for example,
to the footer.

Using and Understanding the code in html.pl

We illustrate how the predicates introduced above can be used to create more complex HTML code with a card deck example. We also briefly discuss how the Prolog clauses for these predicates have been defined.

A Quick guide to using Bootstrap components not defined in html.pl

We briefly discuss an example to illustrate how to use the code in html.pl yourself with Bootstrap components that you might want to use that have not yet been introduced. To that end, suppose that you would like to create a lay-out with three images side-by-side. The first step you could take is to check which components Bootstrap offers to create such a lay-out. It turns out that a card deck might just be what you were looking for. It is advertised as something you would use when you “Need a set of equal width and height cards that aren’t attached to one another? Use card decks.” Bootstrap also provides the HTML template code as an example of how to use it. This is what that looks like:

<div class="card-deck">
  <div class="card">
    <img class="card-img-top" src="...">
    <div class="card-body">
      <h5 class="card-title">Card title</h5>
      <p class="card-text">This is a card.</p>
    </div>
  </div>
  <div class="card">
    <img class="card-img-top" src="...">
    <div class="card-body">
      <h5 class="card-title">Card title</h5>
      <p class="card-text">This is a card.</p>
    </div>
  </div>
  <div class="card">
    <img class="card-img-top" src="...">
    <div class="card-body">
      <h5 class="card-title">Card title</h5>
      <p class="card-text">This is a card.</p>
    </div>
  </div>
</div>

How do we break this code down into pieces that we can (already) manage? We first scan the code for the HTML tags that are used in it. We find that the div tag is often used (with the classes "card-deck", "card", and "card-body"). We also find that the img tag is used as well as some variants of the <h5> and <p> tags with specific card-related classes added. We note that for each of these tags we already have the tools to generate them in Prolog. We can use the div/4 and img/4 predicates for the former tags and can use our strategy to quote basic HTML elements for the h5 and p elements (you can also introduce new predicates for these tags in combination with a class attribute if you like but we will keep things simple here).

Now we have established that all the basic ingredients are available, the next step in our recipe is to create the separate parts of the HTML code by - in a sense - looking from the inside out. We first will generate the elements that are most embedded and then use those to build the elements in which they are directly embedded. The most deeply embedded elements are the heading and paragraph elements. As suggested above, we simply quote these. In order to create a single atom we also concatenate these elements using the atomic_list_concat/2 predicate as follows:

atomic_list_concat([
  '<h5 class="card-title">Card title</h5>',
  '<p class="card-text">This is a card.</p>'], CardBodyContent)

We introduced the CardBodyContent variable here to indicate that the HTML code generated by the atomic_list_concat query will be used as content for the card-body class element.

For the card-body class we can use the div/4 predicate, where we use the CardBodyContent variable to replace the default input argument Content variable for this predicate and fill in the class and style arguments. We name the output argument CardBodyElement and then we get as our next piece of code:

div(CardBodyContent, 'card-body', '', CardBodyElement)

For creating the card, we also need an image (again we do not bother to fill in the ... in the example, which we leave up to you; check out the https://socialrobotics.atlassian.net/wiki/spaces/PM2/pages/2264236033/Visual+Support+Guide#img-tag section above on how to do that). We use the img(+Source, +Class, +Style, -Html) predefined predicate and use CardImage as output variable:

img('...', 'card-img-top', '', CardImage)

Now we have code to generate the two elements that we need to create a card class element, we can piece these two elements together using the div/4 predicate to generate code for that element. Phrases like piece these together by now should have alerted you to the fact that atomic_list_concat might be useful! We write the following Prolog code to generate the card element:

atomic_list_concat([CardImage, CardBodyElement], CardContent),
% We need three cards!
atomic_list_concat([CardContent, CardContent, CardContent], CardContent3Times),
div(CardContent3Times, 'card', '', CardDeckContent)

Finally, as a last step, we now can create our card-deck using div/4 again as follows:

div(CardDeckContent, 'card-deck', '', Html)

We now have all the different Prolog queries that we need, in the right order, to create one big query that will generate the complete HTML code for the card deck. We simply need to add commas in between these queries (used as conjuncts) to obtain our complete Prolog query:

% Prolog query that generates HTML code for a three-card card deck
% First, we generate the content for the card body
atomic_list_concat([
  '<h5 class="card-title">Card title</h5>',
  '<p class="card-text">This is a card.</p>'], CardBodyContent),
% then we generate the card body element
div(CardBodyContent, 'card-body', '', CardBodyElement),
% then we generate the card image element
img('...', 'card-img-top', '', CardImage),
% and then we generate the content for the card deck
atomic_list_concat([CardImage, CardBodyElement], CardContent),
% We need three cards!
atomic_list_concat([CardContent, CardContent, CardContent], CardContent3Times),
div(CardContent3Times, 'card', '', CardDeckContent),
% so we can finally create the card deck html element
div(CardDeckContent, 'card-deck', '', Html).

% The Html output variable will return the complete card deck HTML code we showed above.

The HTML code and query above, of course, will not work yet as we used three dots ... for the src attribute. You will still need to make sure that the right image URLs (or Base64 codes) are used to replace these dots and modify the accompanying text in the card bodies to match with these images.

Defining new predicates for generating HTML elements

When you inspect the code in the html.pl file, you will see that a returning pattern has been used to define Prolog predicates for corresponding HTML tags. The general pattern for a tag named tagname looks like this:

% tagname/1 is used to introduce an HTML template with ~a to indicate placeholders for a 
%   tag tagname; in this example, there are just two placeholders and we assume that the tag
%   has a start <tagname> and end tag </tagname>
tagname('<tagname... ~a ...  >~a</tagname>')

One of the placeholders always should represent the content of the tagname element (which typically will appear last in the HTML template code, right between the start and end tag). Other placeholders could be used for class or other attributes. The second clause that defines the tagname predicate that generates the HTML code for the element should have N+1 arguments where N is the number of placeholders in the template. To generate the HTML code, we use the predefined applyTemplate(+Template, +ArgumentList, -Html) query for instantiating the placeholders in the HTML template. The typical pattern looks like this:

% tagname/3 predicate for generating HTML code for the tag element;
% we suppose, as an example, that the other placeholder in the template is a class attribute
tagname(Content, Class, Html) :- 
    tagname(Template), applyTemplate(Template, [Class, Content], Html).

Above, we suggested that we could also introduce a predicate for a paragraph with a class attribute, e.g. for HTML code that looks like <p class="card-text">SOME TEXT CONTENT</p>. Perhaps this is not that often needed in practice, but for illustrative purposes we will now show as an example how such a predicate can be defined.

The first thing we always need to do is to add a template for the HTML element. We will simply use the p/1 predicate in line with our convention to name the predicate the same as the HTML tag, which is convenient We want to be able to fill in the class attribute and the paragraph content, so we insert two placeholders ~a (one for each of these):

% Paragraph template with class and content placeholders
p('<p class="~a">~a</p>').

To define a p/3 predicate p(Content, Class, Html) that allows for filling in a class attribute as well as content for the paragraph element, we use the template and the applyTemplate/3 predicate:

% Rule for paragraph predicate p/3 for a paragraph element with a class attribute
p(Content, Class, Html) :- p(Template), applyTemplate(Template, [Class, Content], Html).

Note that by convention we put the Content argument first. This is just a convention but by sticking with it we keep our code consistent and avoid confusion about what the first argument of our predicates should be instantiated with. The elements in the list [Class, Content] fed to the applyTemplate predicate, however, should match the order of the placeholders ~a in the paragraph template we introduced above to make sure that they are substituted in the right order into this template.

As an example, we can now use our p/3 predicate to replace our quoted paragraph element '<p class="card-text">This is the third card.</p>' in our card deck example above as follows:

p('This is a card.','card-text', ParagraphElement)

Note that we need to introduce an output argument variable name to retrieve the HTML code for the paragraph in.