The Social Interaction Cloud (SIC) is a light-weight, easy-to-use framework for developing socially interactive systems. It has been developed with the aim to facilitate social interaction with physical devices. The framework can be also used to easily create data pipelines and enables scaling to support more advanced architectures using more advanced cloud-based computing, for example.

API

Running a hello world on a NAOv6 robot is as simple as

from sic_framework.devices import Nao
from sic_framework.devices.common_naoqi.naoqi_text_to_speech import NaoqiTextToSpeechRequest

nao = Nao(ip='192.168.0.151')
nao.tts.request(NaoqiTextToSpeechRequest("Hello world!"))

Installation & Getting started

You can find the instructions on how to install the framework here: Install

To get started read Getting started with a robot

Components

Quickly link per-trained models and cloud solutions together using components, such as:

Image Modifiedhttps://bitbucket.org/socialroboticshub/framework/src/master/sic_framework/services/openai_whisper_speech_to_text/

Image Modifiedhttps://bitbucket.org/socialroboticshub/framework/src/master/sic_framework/services/openai_gpt/

Image Modifiedhttps://bitbucket.org/socialroboticshub/framework/src/master/sic_framework/services/dialogflow/

Image Modifiedhttps://bitbucket.org/socialroboticshub/framework/src/master/sic_framework/services/face_detection_dnn/

Image Modifiedhttps://bitbucket.org/socialroboticshub/framework/src/master/sic_framework/services/text2speech/

Image Modified

There are plenty of demo’s on how to use various components in https://bitbucket.org/socialroboticshub/framework/src/master/sic_framework/tests/, so check them out!

Overview of framework structure

To give an example of how the framework is structured, here are three usecases and the code a student would have to write

1. A student wants to display face recognition on their laptop.
   1 input, 1 output to student device

2. A student wants to send and image to face recognition and save the result
   1 input, 1 output that must be tied to the input

3. A student wants to send robot audio to dialogflow and wave when it detects “Hello”
   2 asynchronous inputs (audio, command), 2 asynchronous outputs (transcript, intent)

Display face recognition

Do face recognition on a nao’s camera stream

Image Modified

image = None
def set_image_variable(img):
  image = img

bbox = None
def set_bbox_variable(box):
  bbox = box
  
nao = Nao(ip="192.168.0.181")
face_recognition = FaceRecognition(ip="127.0.0.1")

nao.top_camera.register_callback(set_image_variable)

face_recognition.connect(nao.top_camera)
face_recognition.register_callback(set_bbox_variable)

while True:
  image.draw(bbox)
  display(image)

Do a single face recognition

Recognize the faces in a picture from the student’s laptop

Image Modified

face_recognition = FaceRecognition()

image = load_from_disk("picture.jpg")

image_request = FaceRecognitonRequest(image=image)
bboxes = face_recognition.request(image_request)

draw_bbox_on_image(image.image, bboxes.bboxes[0])

display(image.image)

Dialogflow hello detection

A demo that has a robot wave whenever someone says “hello” and prints the detected transcript on the laptop.

Image Modified

pepper = Pepper(ip="192.168.0.123")


keyfile_json = json.load(open("sail-380610-0dea39e1a452.json"))
conf = DialogflowConf(keyfile_json=keyfile_json,
                      sample_rate_hertz=16000, )
dialogflow = Dialogflow()

dialogflow.connect(pepper.microphone)
dialogflow.register_callback(print_transcript)

while True:
  intent = dialogflow.request(ListenForIntentRequest())
  if intent.name == "hello":
    wave_req = PepperMotionRequest("wave")
    pepper.motion.request(wave_req)