Guess the Number with Pepper using tablet

This tutorial will show you how to interact with Pepper tablet to play a “guess the number” game.

The ideas are:

Make pepper display a random number between 1-10 on a webpage on its tablet with a start button.
Upon a user pressing the start button, make Pepper say: which number is displayed on my tablet?
User mentions number, audio picked up by Pepper's microphone array.
Pepper transcribes the audio and checks number with the one of the tablet, and responds either: Yes, that's correct, OR No, [too high/too low]. Try again.

Approach

Before creating the demo, it’s essential to understand that the design of the WebserverComponent goes hand in hand with the requirements of the demo, as the WebserverComponent is the component that handles any events from a web client. For this specific tutorial, the WebserverComponent should handle a button click event and send live transcripts produced by Dialogflow service to a connected client over a WebSocket connection.

This tutorial can be divided into two parts:

The design of HTML and WebserverComponent
1. How to create a button and catch a click event on it?
2. How to render Dialogflow intermediate transcripts dynamically in a text box?
The demo program connects all the components (NaoqiMicrophoneSensor(or local DesktopMicrophone), WebserverComponent, DialogflowService, etc)

The design of html and Webserver

1. How to create a button and catch a click event on it?

In the html, we need to add a button and make it listen to the click event. Once the button is clicked, the isClicked flag will be set to true, and it sends the flag to the Flask server via WebSocket.

// Flag to track if the game should start
var isClicked = false; 

// Get a reference to the startButton element
var startButton = document.getElementById('startButton');

// Event listener for the start button click
startButton.addEventListener('click', function() {
  isClicked = true; // Set the flag
  sendFlagToServer(isClicked);
});

// Function to send flag to Flask server
function sendFlagToServer(flag) {
  socket.emit('clicked_flag', flag);
}

On the Flask server, we need to register a handler to react to the "clicked_flag" event.

# register clicked_flag event handler
@self.socketio.on('clicked_flag')
def handle_flag(flag):
    if flag:
        self.output_message(ButtonClicked(button=flag))

If the flag is set to true, a SICMessage “ButtonClicked” will be sent to the SIC channel as output of this component. This output will be received on a callback in the demo side to determine if the button has been clicked or not.

2. How to render Dialogflow intermediate transcripts dynamically in a text box?

In the html, first add a textarea element inside the body section.

<div>
  <textarea id="textbox" rows="4" cols="50"></textarea>
</div>

In Javascript, we create a function to update the content of the textbox .

// Function to update the textbox
function updateTextbox(text) {
  document.getElementById('textbox').value = text;
}

When a web client receives an “update_textbox” event from the flask server, the event handler is triggered. Then it calls the updateTextbox(text) function passing the received text as an argument.

    // Event handler for receiving updated text
    socket.on('update_textbox', function(text) {
      updateTextbox(text);
    });

On the Flask server, we emit the “update_textbox” event along with the transcript transcribed from Dialogflow.

self.socketio.emit('update_textbox', self.transcript)

The demo program connects all the components

There are many SIC components we need to connect to in this demo.

Components running on a pepper:

NaoqiMicrophoneSensor
NaoqiTextToSpeechActuator
NaoqiTabletComponent

Components running on a local machine:

WebserverComponent
DialogflowService