pepper-imitation:pepper-imitation

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
pepper-imitation:pepper-imitation [2018/06/28 19:57] – s4pepper2018pepper-imitation:pepper-imitation [2018/06/28 20:18] (current) – s4pepper2018
Line 11: Line 11:
 {{:pepper-imitation:gestures.jpg?200|}} {{:pepper-imitation:gestures.jpg?200|}}
  
-As such, the project is made up of two main pillars, namely: +As such, the project is built on top of two main pillars, namely: 
-  * Body pose detection: Windows Kinect V1 was the sensor chosen for this task.  +  * **Body pose detection**: Windows Kinect V1 was the sensor chosen for this task.  
-  * Robot control and synchronization.+  * **Robot control and synchronization**.
  
 Body pose detection was entirely done using publicly available software. Thus, the main focus of our work was to familiarize ourselves with the robot itself and synchronize all the tasks, as well as fine-tune everything, taking into account the nature of the project and the target subjects. Body pose detection was entirely done using publicly available software. Thus, the main focus of our work was to familiarize ourselves with the robot itself and synchronize all the tasks, as well as fine-tune everything, taking into account the nature of the project and the target subjects.
Line 21: Line 21:
 ===== Installation and Set Up ===== ===== Installation and Set Up =====
  
-The code is fully available [[https://github.com/Thordreck/pepper-imitation-pkg|here]]. The repo should be clone directly into the src folder of a catkin workspace.+The code is fully available [[https://github.com/Thordreck/pepper-imitation-pkg|here]]. The repo should be cloned directly into the src folder of a catkin workspace.
 The repository above contains two ROS packages at the root folder level: The repository above contains two ROS packages at the root folder level:
-  * openni_tracker: skeleton tracker for use with kinect. This was included in the repository, as no kinetic version is available in the official repos. +  * **openni_tracker**: skeleton tracker for use with kinect. This was included in the repository, as no kinetic version is available in the official repos. 
-  * pepper_imitation: the system developed by the S4 team.+  * **pepper_imitation**: the system developed by the S4 team.
  
 Dependencies (and even ROS itself if not present in the system), can be installed running installDependencies.sh in pepper_imitation/installDependencies.sh. Dependencies (and even ROS itself if not present in the system), can be installed running installDependencies.sh in pepper_imitation/installDependencies.sh.
Line 38: Line 38:
  
 The system is made up of the following nodes: The system is made up of the following nodes:
-  * pepper_audio_player_node: offers an interface capable of loading, starting and stopping and audio file saved in Pepper's internal computer. It reports back the current playing time as well. +  * **pepper_audio_player_node**: offers an interface capable of loading, starting and stopping and audio file saved in Pepper's internal computer. It reports back the current playing time as well. 
-  * pepper_face_tracker_node: enable/disable Pepper's built-in random face tracking. +  * **pepper_face_tracker_node**: enable/disable Pepper's built-in random face tracking. 
-  * pepper_tts_node: interface to use Pepper's TTS engine. It supports emotional speech as well (e.g: //style=joyful//). +  * **pepper_tts_node**: interface to use Pepper's TTS engine. It supports emotional speech as well (e.g: //style=joyful//). 
-  * pepper_tablet_node: pops up an input box for the user to type his name before starting the game. +  * **pepper_tablet_node**: pops up an input box for the user to type his name before starting the game. 
-  * pepper_imitation_node: command the different gestures and checks if the detected person's pose - if any - is similar. +  * **pepper_imitation_node**: command the different gestures and checks if the detected person's pose - if any - is similar. 
-  * pepper_teleop_joy_node: allows to control the robot's movement and rotation using a joystick. Keep in mind that the default joystick's values defined in the node have been set up to match those of a wired Xbox controller.+  * **pepper_teleop_joy_node**: allows to control the robot's movement and rotation using a joystick. Keep in mind that the default joystick's values defined in the node have been set up to match those of a wired Xbox controller.
  
-All this nodes can be run separately and commanded by publishing in their respective topics, allowing a Wizard of Oz-ish level of control.+All these nodes can be run separately and commanded by publishing in their respective topics, allowing a Wizard of Oz-ish level of control.
 The actual game and synchronization is achieved by means of a state machine, defined in "pepper_imitation_game_node.py" file. SMACH was the library chosen for this. The actual game and synchronization is achieved by means of a state machine, defined in "pepper_imitation_game_node.py" file. SMACH was the library chosen for this.
  
Line 56: Line 56:
   * Pepper prompts the user to do as him, and assumes a given pose sync'd with the music.   * Pepper prompts the user to do as him, and assumes a given pose sync'd with the music.
   * Now, two possible scenarios can follow:   * Now, two possible scenarios can follow:
-     * The player adopts a pose similar enough. Then Pepper will encourage him to keep going, and the son g continues.+     * The player adopts a pose similar enough. Then Pepper will encourage him to keep going, and the song continues.
      * The player does not make a similar pose, or the body pose is not detected properly. In this case, Pepper will ask the player to focus and try again, and the music will go back to the previous part.      * The player does not make a similar pose, or the body pose is not detected properly. In this case, Pepper will ask the player to focus and try again, and the music will go back to the previous part.
      * If no skeleton at all is found, Pepper will tell the user that it cannot find him. Two more additional sub-scenarios are possible here:      * If no skeleton at all is found, Pepper will tell the user that it cannot find him. Two more additional sub-scenarios are possible here:
Line 66: Line 66:
  
   * The synchronization process is hardcoded. As such, correct synchronization will only be achieved if the same file is used (it is available in the robot's internal memory in /home/pepper/resources/audio), The state machine waits until the file is played to a given time (usually the time when a new gesture is started).   * The synchronization process is hardcoded. As such, correct synchronization will only be achieved if the same file is used (it is available in the robot's internal memory in /home/pepper/resources/audio), The state machine waits until the file is played to a given time (usually the time when a new gesture is started).
-  * The way openni_tracker publish the skeleton information (using TF frames following the convention <body_part>_<id>, for example torso_1) poses some problems when people are lost and new ids are assigned. This is due to old TF frames still being listed in by ROS even after a while. An easy solution to this would be to query the TF frames, but keep only those with the highest id value.+  * The way openni_tracker publish the skeleton information (using TF frames following the convention <body_part>_<id>, for example torso_1) poses some problems when people are lost and new ids are assigned. This is due to old TF frames still being listed by ROS API even after a while. An easy solution to this would be to query the TF frames, but keep only those with the highest id values.
   * As of now, only one person is tracked. However, adding multiple person's body poses verification should be rather trivial.   * As of now, only one person is tracked. However, adding multiple person's body poses verification should be rather trivial.
  
 ===== Future Work ===== ===== Future Work =====
  
 +  * The detection provided by the kinect and the openni libraries is not enough. The calibration phase is a limiting factor when interacting with the subjects. A possible solution to this would be to adopt body pose detection based on RGB data alone. An example of this would be [[https://github.com/CMU-Perceptual-Computing-Lab/openpose|OpenPose]]. This particular game could be reproduced entirely using only joints angles data in 2D detections, even allowing us to get rid of the external kinect completely, and using the built-in cameras. More complex scenarios and gestures may need visual and depth data fusion. For a real time application, a GPU should be used.
 +  * The concept of this game can be generalized a bit, using some simple config files. These files could define the audio file to be used and the time when each posture should be adopted. Postures may be defined here as well.
 +  * Furthermore, these files could be generated using a user-friendly GUI, where the user could set an audio file and set up the sync times and the robot poses.
 +
 +===== Contact =====
 +
 +Álvaro Páez Guerra
 +
 +paezguerraalvaro@gmail.com
  • pepper-imitation/pepper-imitation.1530215869.txt.gz
  • Last modified: 2019/04/25 14:08
  • (external edit)