Autism Spectrum Disorder (ASD) refers to a neurodevelopmental disorder characterized by impaired social interaction, communication difficulties, and restricted and repetitive behaviors and interests. For conditions like autism, there has always been a hope to leverage artificial intelligence, digital technology, and other means to provide more solutions for the assessment, monitoring, and treatment of these disorders.
With the increasing maturity of humanoid robot technology, robots have shown great potential as therapeutic intermediary tools in the field of cognitive impairment. Taking autism as an example, scientific and medical literature since the 1990s indicates that children with autism spectrum disorder have a strong interest in mechanical components, computers, and robots.
Specifically, humanoid robots with anthropomorphic features can "purify" the information received during interactions. They provide predictable and consistent actions, synthetic voices without a pronounced personality in their tone, and possess software that can simulate basic "social and emotional" abilities. These characteristics often reduce anxiety in individuals with autism spectrum disorder and enhance their sensory receptiveness.
For instance, the robot NAO is one of the commonly used robots in the treatment of autism. Studies have shown that the intervention measures provided by the robot NAO can help improve the engagement, joint attention, and social skills of autistic children. (Previous research includes:人形机器人在儿童心理健康评估领域的广阔前景)
Recently, researchers from the University of Calabria in Italy published their study in the Frontiers in Neuropsychology journal, where they combined the Pepper robot with the Open AI system (ChatGPT) to assist in the cognitive development of autistic children. The study simulated two possible interactive scenarios for autistic patients under two different conditions, indicating that the combination of the Pepper robot and ChatGPT technology provided high-quality, adaptive interactive experiences for autistic children.
The technical process of integrating Pepper with Open AI
First, the researchers introduced the key technical specifications involved in the integration process, which included: Pepper's operating system (Naoqi 2.5 Python SDK), Open AI API (GPT3.5 Turbo for text recognition using Whisper), and multiple main Python scripts, each with specific functions. These functions included emotion recognition and communication, communication about Pepper's features and capabilities, and simulation of social interactions through integration with OpenAI.
Among them, the Python 3.11 script used various libraries, including OpenAI, as well as the Whisper library combined with speech recognition and ffmpeg for speech recognition and converting user input into text prompts. It utilized the ChatGPT API to obtain responses, providing contextual information for each query. Additionally, the speech recognition library could recognize specific words to activate the listening phase.
Then, by using the Naoqi library, the text information was converted into readable dialogue files for Pepper. The functions included: connecting to the robot, transmitting dialogue files, and activating Pepper's autonomous living mode, enhancing the robot's ability to interact with users.

The communication process between Open AI and Pepper involves several steps
Ultimately, by connecting the NAOqi library used to manage Pepper's functions with a pre-trained neural network for facial and emotion recognition, the recognition of emotional expressions on faces was achieved. This included recording videos or capturing frames through Pepper's camera, analyzing the images captured by Pepper using facial recognition functions, and invoking the FER (Facial Emotions Recognition) library to identify facial emotions.
Interaction Scenario Simulation
Building on this foundation, the study proposed two simulated interaction scenarios to demonstrate the system's potential applications in different environments. These scenarios are designed to facilitate communication, enhance social skills, and improve problem-solving abilities among individuals with autism spectrum disorder.
In essence, the research has developed a comprehensive curriculum aimed at utilizing the capabilities of ChatGPT and combining them with the use of images to enhance the development of emotional recognition skills and cognitive skills in autistic patients.

The interaction scenarios include informal interaction and structured interaction.
In the informal interaction scenario, the focus is on creating a relaxed and comfortable environment where individuals with autism spectrum disorder can freely interact with the Pepper robot. This informal interaction allows for spontaneous communication and provides a platform for patients to express themselves and build rapport with the robot. Pepper's responsive capabilities, including speech recognition, expressive facial features, and body language, help create a supportive and engaging interactive environment.
In the structured interaction scenario, the focus is on guided activities and problem-solving tasks. Pepper takes on the role of an instructor, presenting specific challenges or scenarios that require critical thinking, decision-making, and collaborative problem-solving. The structured interaction provides a framework for patients, including steps, prompts, and integration with Open AI to enhance the problem-solving process. Pepper guides patients through the different stages of problem-solving, providing feedback and suggestions, and encouraging active participation. This scenario aims to foster cognitive skills, decision-making abilities, and collaboration in a structured and supportive manner.
Overall, both the informal and structured scenarios utilize Pepper robot's capabilities such as speech recognition, speech synthesis, facial expressions, and body language to create dynamic interactive experiences. Additionally, the integration with ChatGPT technologies like natural language understanding and image generation enhances the depth and flexibility of the interactions. These scenarios are designed to meet the unique needs of autistic patients, providing personalized and inclusive environments for communication, social skill development, and problem-solving.
It is important to note that while social robots can serve as supportive tools, they do not intend to replace human interaction and intervention. It is through the careful collaboration of social robots and professionals such as specialized therapists that the best therapeutic outcomes can be achieved for patients.
reference:https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1232177/full#B5
More content:
Follow SoftBank Robotics
Explore the endless possibilities of robots