This is an old revision of the document!

S4 project 2018 : A reinforcement learning approach based on imitation using kinect and Pepper.

The project aims to be the first steps for a development of a more general learn-by-imitation approach for autistic children. It was developed in collaboration with the CHU Brest, more specifically with Dr. Nathalie Collot as the main contact.

The main goal was to simulate a well-known nursery rhyme used by Ms Collot's team to interact with the patients. The song is divided up in several sections, where the entertainer would assume a pre-defined posture that the kids should try to imitate.

The gestures currently used are shown in the image below:

As such, the project is made up of two main pillars, namely:

  • Body pose detection: Windows Kinect V1 was the sensor chosen for this task.
  • Robot control and synchronization.

Body pose detection was entirely done using publicly available software. Thus, the main focus of our work was to familiarize ourselves with the robot itself and synchronize all the tasks, as well as fine-tune everything, taking into account the nature of the project and the target subjects.

The chosen framework was ROS (the only tested version was kinetic), written mainly in C++, with some parts in python.

The code is fully available here. The repo should be clone directly into the src folder of a catkin workspace. The repository above contains two ROS packages at the root folder level:

  • openni_tracker: skeleton tracker for use with kinect. This was included in the repository, as no kinetic version is available in the official repos.
  • pepper_imitation: the system developed by the S4 team.

Dependencies (and even ROS itself if not present in the system), can be installed running in pepper_imitation/

The game sequence can be started by running the roslaunch file pepper_imitation_game.launch. The skeleton tracker and the kinect driver are to be run separately using the file pepper_imitation_kinect.launch.

Body pose checks can be disabled in pepper_imitation_node.launch, setting the “skip_pose_checks” parameter to true. If disabled, the game will assume all the checks to be correct.

The system is made up of the following nodes:

  • pepper_audio_player_node: offers an interface capable of loading, starting and stopping and audio file saved in Pepper's internal computer. It reports back the current playing time as well.
  • pepper_face_tracker_node: enable/disable Pepper's built-in random face tracking.
  • pepper_tts_node: interface to use Pepper's TTS engine. It supports emotional speech as well (e.g: style=joyful).
  • pepper_tablet_node: pops up an input box for the user to type his name before starting the game.
  • pepper_imitation_node: command the different gestures and checks if the detected person's pose - if any - is similar.
  • pepper_teleop_joy_node: allows to control the robot's movement and rotation using a joystick. Keep in mind that the default joystick's values defined in the node have been set up to match those of a wired Xbox controller.

All this nodes can be run separately and commanded by publishing in their respective topics, allowing a Wizard of Oz-ish level of control. The actual game and synchronization is achieved by means of a state machine, defined in “” file. SMACH was the library chosen for this.

  • pepper-imitation/pepper-imitation.1530214018.txt.gz
  • Last modified: 2019/04/25 14:08
  • (external edit)