How does SIC work?
SIC provides a framework to combine different robots, online services, and data processing tools together into more powerful applications, extending their individual capabilities. For example, with SIC, you can connect a robot to the OpenAI API, enabling you to have a conversation with it as if it were ChatGPT. It also provides an interface to manually control robots such as Nao and Pepper.
The Most Basic Understanding
A SICComponent is the core building block of the framework. A component can receive some form of input, perform processing, and possibly provide an output. You can think of components sort of like LEGO blocks, and a SIC application as putting those LEGO blocks together in a certain way. While in reality SIC is more complicated than just running different components—it requires other pieces to manage, connect, and send messages between them—as an introduction this understanding suffices. For a more advanced understanding, see ‘Dissecting Face Detection Example’.
Robot greeting example
To make a robot greet a person walking up to it, for example, we’ll need to combine a camera component, face detection component, and a text-to-speech component. We can show this in a more schematic approach. For example, here are the building blocks to get started in detecting a person walking up to the robot:
Notice that in this schematic “SIC program”, which is the program that you write, is shown as a component. This is technically not a component defined within the SIC framework (it is the program you create with SIC), but it can receive inputs and provide outputs, and can therefore also be seen as a sort of component.
Here is the code to first create the two components:
nao = Nao(ip="192.168.0.148")
face_rec = FaceDetection()Showing the robot’s camera stream
On their own, components don’t do very much. Lets make it interesting by connecting them! We can connect to the Nao robot’s camera like this:
The code to connect the CameraSensor component to our SIC program is the following:
imgs_buffer = queue.Queue()
def on_image(image_message: CompressedImageMessage):
imgs_buffer.put(image_message.image)
nao.register_callback(on_image)
while True:
img = imgs_buffer.get()
cv2.imshow("robot's view", img)
cv2.waitKey(1)Notice the line nao.register_callback(on_image). This call connects the stream of images from the robot’s camera to our SIC application. It makes it so every time the Nao’s camera publishes a new image, the on_image function will be called. The on_image function then puts the image into a queue/buffer within our application which we can then view.
If you’re thinking “this seems complicated” and wondering why we need to register callback functions, see Redis for more info.
To view the image, we call the .get() method on the queue, and use the python cv2 (computer vision) module to display it. With this code, we can already view Nao’s camera stream from our computer!
Detecting a face in an image
Alright, so we have a camera stream now. How do we detect faces in it? Lets take a small step back, and focus on detecting faces on a single image first. Detecting faces on a stream of images is better done in a different way, but we’ll get to that.
For detecting faces in a single image, we’ll use something called a Request. You can send a request to a component, and it replies with a Message. If we send the FaceDetection service a CompressedImage request, it will detect the faces and estimate bounding boxes around them. It then returns the coordinates and measurements defining these bounding boxes within a BoundingBoxes Message.
See ‘Messages/Request’ for more information regarding the types of requests.
In code, this looks like this:
img = cv2.imread('/path/to/image/some_image.png')
image_request = CompressedImageRequest(image=img)
boundingboxes = face_rec.request(image_request)
for bbox in boundingboxes.bboxes:
print("Face at", bbox.x, bbox.y)