Installation & Getting started

You can find the instructions on how to install the framework here: Getting started

To include a robot in the mix read Getting started with the Nao robot

API

Running a hello world on a NAOv6 robot is as simple as

from sic_framework.devices import Nao
from sic_framework.devices.common_naoqi.naoqi_text_to_speech import NaoqiTextToSpeechRequest

nao = Nao(ip='192.168.0.151')  # change into IP-address of your Nao robot.
nao.tts.request(NaoqiTextToSpeechRequest("Hello world!"))

Components

Quickly link per-trained models and cloud solutions together using components, such as:

OpenAI Whisper Speech to Text

OpenAI ChatGPT

There are plenty of demo’s on how to use various components in https://github.com/Social-AI-VU/sic_applications, so check them out!

Overview of framework structure

To give an example of how the framework is structured, here are three usecases and the code a student would have to write

A student wants to display face recognition on their laptop.
1 input, 1 output to student device
A student wants to send and image to face recognition and save the result
1 input, 1 output that must be tied to the input
A student wants to send robot audio to dialogflow and wave when it detects “Hello”
2 asynchronous inputs (audio, command), 2 asynchronous outputs (transcript, intent)

Display face recognition

Do face recognition on a nao’s camera stream

image = None
def set_image_variable(img):
  image = img

bbox = None
def set_bbox_variable(box):
  bbox = box
  
nao = Nao(ip="192.168.0.181")  # change into IP-address of your Nao robot.
face_recognition = FaceRecognition(ip="127.0.0.1")

nao.top_camera.register_callback(set_image_variable)

face_recognition.connect(nao.top_camera)
face_recognition.register_callback(set_bbox_variable)

while True:
  image.draw(bbox)
  display(image)

Do a single face recognition

Recognize the faces in a picture from the student’s laptop

face_recognition = FaceRecognition()

image = load_from_disk("picture.jpg")

image_request = FaceRecognitonRequest(image=image)
bboxes = face_recognition.request(image_request)

draw_bbox_on_image(image.image, bboxes.bboxes[0])

display(image.image)

Dialogflow hello detection

A demo that has a robot wave whenever someone says “hello” and prints the detected transcript on the laptop.

The Social Interaction Cloud (SIC)

The Social Interaction Cloud (SIC) V2