Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
poppy-kine:poppy-kine-2015-s5 [2016/03/09 11:37] vngo |
poppy-kine:poppy-kine-2015-s5 [2019/04/25 14:08] (current) |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | {{tag>Poppy Project}} | + | {{tag>Poppy Project poppy-kine Kinect}} |
| - | ====== Poppy-Kine project ====== | + | ====== Poppy-Kine : S5 project 2015-2016 ====== |
| ===== Overview ===== | ===== Overview ===== | ||
| + | {{http://breddydotorg.files.wordpress.com/2013/07/shoulder-flexion-and-extension.png?100}} | ||
| The main objective of this project is to help patients with **functional reeducation** exercices. | The main objective of this project is to help patients with **functional reeducation** exercices. | ||
| Patient A's kinesitherapist will show him/her some movements that A will need to repeat more or less regularly, //at home and without the doctor//. | Patient A's kinesitherapist will show him/her some movements that A will need to repeat more or less regularly, //at home and without the doctor//. | ||
| - | \\ Therefore, for better memorization, the movement will be done by a **Poppy Robot** (see [[http://poppy-project.org/|Poppy project]] and [[poppy|Poppy robot wiki page]]). | + | \\ Therefore, for better memorization, the movement will be done by a **Poppy Robot** (see [[http://poppy-project.org/|Poppy project]] and [[:poppy|Poppy robot wiki page]]). |
| \\ First, the moves are done in front of a **Kinect camera** (v2). | \\ First, the moves are done in front of a **Kinect camera** (v2). | ||
| \\ The movements (captured by position of the Skeleton) are saved, and replayed by the robot. | \\ The movements (captured by position of the Skeleton) are saved, and replayed by the robot. | ||
| Line 19: | Line 19: | ||
| - First step : capturing the Skeleton (Kinect v2) as **cartesian** coordinates (x,y,z) | - First step : capturing the Skeleton (Kinect v2) as **cartesian** coordinates (x,y,z) | ||
| - Second step : converting the Skeleton data into Poppy robot's referential (**angular**) | - Second step : converting the Skeleton data into Poppy robot's referential (**angular**) | ||
| + | \\ __**NB : **__ | ||
| + | * If you want to **make Poppy rigid**, we wrote a program that puts it to the //zero position// (ie all the motors' angles are set to zero): **Init_poppy.py** | ||
| + | * Don't forget to **plug Poppy** ! | ||
| ==== First step: getting the Skeleton ==== | ==== First step: getting the Skeleton ==== | ||
| + | {{:kinect_skeleton.png?200 |}} | ||
| * Plug the Kinect camera to the computer **before** turning it on (or you will probably need to restart the computer) | * Plug the Kinect camera to the computer **before** turning it on (or you will probably need to restart the computer) | ||
| * Connect the Kinect camera | * Connect the Kinect camera | ||
| + | \\ | ||
| + | \\ | ||
| + | \\ | ||
| + | The Kinect recovers the position of the following joints: | ||
| + | * Unordered List Item | ||
| + | {{https://naokinect.files.wordpress.com/2012/02/image_thumb_6f4828ec1.png}} | ||
| ==== Second step: the movement ==== | ==== Second step: the movement ==== | ||
| - | All the programs we wrote are in **projets5/poppy_humanoid/software/poppy_humanoid/** folder | + | All the programs we wrote are in **C:\Users\ihsev\Documents\2015projetS5PoppyKine-VirginieNgoGuillaumeVillette\poppy-humanoid\software\poppy_humanoid/** folder |
| === First option: manually save, convert, then play the movement=== | === First option: manually save, convert, then play the movement=== | ||
| * Save the movement | * Save the movement | ||
| <hidden> Click **save.bat** | <hidden> Click **save.bat** | ||
| - | \\ Enter the name of the exercise as asked by the shell. | + | \\ Enter the //name of the exercise// as asked by the shell. |
| \\ A new window should appear, with the video captured by the Kinect camera, and the skeleton of the person in front of it (The first time, it can take a while... be patient). | \\ A new window should appear, with the video captured by the Kinect camera, and the skeleton of the person in front of it (The first time, it can take a while... be patient). | ||
| \\ Movements are separated by //space key// pressures. | \\ Movements are separated by //space key// pressures. | ||
| Line 42: | Line 51: | ||
| \\ x starts from zero (first movement). | \\ x starts from zero (first movement). | ||
| \\ To finish capturing the movements, just close the window (click on the red cross..). | \\ To finish capturing the movements, just close the window (click on the red cross..). | ||
| + | |||
| + | You can also use save_movement_and_video.bat to save the video too (info to be checked). | ||
| + | |||
| </hidden> | </hidden> | ||
| * Convert the angles | * Convert the angles | ||
| <hidden> Click **convert.bat** | <hidden> Click **convert.bat** | ||
| - | \\ Enter the name of the exercise as asked by the shell | + | \\ Enter the name of the exercise (name of the folder) as asked by the shell |
| \\ The program will convert all the movements saved in the corresponding folder | \\ The program will convert all the movements saved in the corresponding folder | ||
| \\ The cartesian coordinates and the quaternions captured by the Kinect are saved in .txt files | \\ The cartesian coordinates and the quaternions captured by the Kinect are saved in .txt files | ||
| Line 81: | Line 93: | ||
| ---- | ---- | ||
| ===== How we proceeded ===== | ===== How we proceeded ===== | ||
| - | All the programs we wrote are stored in **projets5/poppy_humanoid/software/poppy_humanoid/** folder | + | All the programs we wrote are stored in **2015projetS5PoppyKine-VirginieNgoGuillaumeVillette/poppy_humanoid/software/poppy_humanoid/** folder |
| ==== First step: getting the Skeleton ==== | ==== First step: getting the Skeleton ==== | ||
| We used the //pykinect2// library in Python. | We used the //pykinect2// library in Python. | ||
| \\ The code is written in **poppy_kinect_save_mouvement.py** file | \\ The code is written in **poppy_kinect_save_mouvement.py** file | ||
| \\ For each movement, a .txt file is saved in **⁄exercices⁄<name_of_the_exercise>/** folder, in <name_of_the_exercise_x>.txt file (x starts from zero (first movement)). | \\ For each movement, a .txt file is saved in **⁄exercices⁄<name_of_the_exercise>/** folder, in <name_of_the_exercise_x>.txt file (x starts from zero (first movement)). | ||
| - | \\ A .txt file is a //Python dictionnary// stored with //json// (in order to save it as a dictionnary and not just some text, and treat the data for the conversion) | + | \\ A .txt file is a //Python dictionnary// stored with //json// format (in order to save it as a dictionnary and not just some text, and treat the data for the conversion) |
| \\ | \\ | ||
| \\ The **cartesian coordinates** are written in the subdictionnary **"positions"**, and each time unit (depends on the framerate) has its own dictionnary | \\ The **cartesian coordinates** are written in the subdictionnary **"positions"**, and each time unit (depends on the framerate) has its own dictionnary | ||
| Line 106: | Line 118: | ||
| \\ | \\ | ||
| \\ Well, you need to **convert the cartesian coordinates into angles**. How ?! | \\ Well, you need to **convert the cartesian coordinates into angles**. How ?! | ||
| - | \\ Take 4 points: we can define 2 vectors. You can calculate the angles between 2 vectors thanks to Euclidean geometry (the functions needed are defined in **calculangles.py** file). | + | <hidden> |
| - | \\ Here, the main difficulty is not to calculate the angles; but to find __which__ vectors to choose. Indeed, we need to consider all the possible configurations in 3D space and find the good formula. | + | {{:anglesbras.png?200 |}} |
| + | \\ Take 4 points: we can define 2 vectors. | ||
| + | \\ You can then calculate the angles between 2 vectors thanks to Euclidean geometry (the python functions needed are defined in **calculangles.py** file). | ||
| + | \\ Here, the main difficulty is not to calculate the angles; but to find __which__ vectors to choose. Indeed, we need to consider all the possible configurations in 3D space and find the good formula.</hidden> | ||
| \\ You can find our approximations in **Kinect_to_Poppy_angles.py** file. | \\ You can find our approximations in **Kinect_to_Poppy_angles.py** file. | ||
| \\ | \\ | ||
| \\ For each movement, a .txt file is saved in //⁄exercices⁄<name_of_the_exercise>/// folder, in **<name_of_the_exercise_x>_poppy.txt** file (x starts from zero (first movement)). | \\ For each movement, a .txt file is saved in //⁄exercices⁄<name_of_the_exercise>/// folder, in **<name_of_the_exercise_x>_poppy.txt** file (x starts from zero (first movement)). | ||
| - | \\ A .txt file is a //Python dictionnary// stored with //json// (in order to save it as a dictionnary and not just some text, and treat the data for Poppy to play the movement) | + | \\ A .txt file is a //Python dictionnary// stored with //json// format (in order to save it as a dictionnary and not just some text, and treat the data for Poppy to play the movement) |
| \\ Basically, the structure of a .txt file (ie the dictionnary that stores the movement) is: | \\ Basically, the structure of a .txt file (ie the dictionnary that stores the movement) is: | ||
| <code> | <code> | ||
| Line 128: | Line 143: | ||
| ===== For further improvements ===== | ===== For further improvements ===== | ||
| The angles we calculated are unfortunately not accurate enough for complex movements. That is why using quaternions can be another way to improve the conversion. | The angles we calculated are unfortunately not accurate enough for complex movements. That is why using quaternions can be another way to improve the conversion. | ||
| - | Moreover, Gaussian mixture models can be used to get one movement from several samples of the same movement; Gaussian mixture regressions can also be considered for the recognition phase (when the human repeats Poppy's movement | + | \\ Moreover, GMM (Gaussian mixture models) and GMR (Gaussian mixture regressions) can be used to synthetize samples of the same movement, and make Poppy then the patient reproduce it. |
| - | Moreover, GMM (Gaussian mixture models) and GMR (Gaussian mixture regressions) can be used to synthetize samples of the same movement, and make Poppy then the patient reproduce it. | + | |