rosnode - rgbd_hand_gesture_recognition.py - parameter

[thomaspeyrucain]

Parameters need to be consistent with other tools with agparser

I am using opendr installed on my computer on the develop branch, I am feeding the rgb camera topic and the depth_image like this one:

I cannot get an output from the /opendr/gestures topic
Is the depth_image topic different from the one that you are using?

[minhquoc0712]

@thomaspeyrucain , I have created the branch named ros1-fixed-rgbd-hand-gestures-recognition. Can you try to run the corresponding ros node from this branch? Since currently I don't have any kinect device with me to test the fixed ros node.

[thomaspeyrucain]

Hello @minhquoc0712 ,
Sorry for the late reply, I saw that it is already merged in develop, I retried and I cannot succeed to get the output topic, maybe we can quickly meet next week to check it

[minhquoc0712]

Can you describe your problem when running the node?

[thomaspeyrucain]

There are no error messages, I just cannot succeed in getting an output on the /opendr/gestures topic, Is this tool only available for the Kinect? We have a depth image in a level of grey as you can see in the first message that I sent; Could it work with this?

[minhquoc0712]

Are you using the newest version of the node? Since after we merge the fix branch into develop, we also fix some bugs.

[thomaspeyrucain]

Yes

[minhquoc0712]

@thomaspeyrucain, can you try the node I just updated in this branch, and maybe modify the delay parameter if needed?

[thomaspeyrucain]

@minhquoc0712 Yes thanks now I get the output:

---
header: 
  seq: 351
  stamp: 
    secs: 1666088087
    nsecs: 507776260
  frame_id: ''
results: 
  - 
    id: 8
    score: 0.3937682509422302
source_img: 
  header: 
    seq: 0
    stamp: 
      secs: 0
      nsecs:         0
    frame_id: ''
  height: 0
  width: 0
  encoding: ''
  is_bigendian: 0
  step: 0
  data: []
---

I cannot find in the documentation what ID correspond what gesture

[tsampazk]

@minhquoc0712 @thomaspeyrucain do you think we could add the id-gesture correspondence in the learner doc, similar to what i did in semantic segmentation?

[thomaspeyrucain]

@tsampazk Yes would be perfect ^^

[minhquoc0712]

Hi, I am not the person implementing the algorithm, but I think I can use the information from the file opendr/projects/perception/multimodal_human_centric/rgbd_hand_gesture_recognition/gesture_recognition_demo.py in the master branch to create the class correspondence table.

[tsampazk]

Thanks @minhquoc0712, i think you can add it directly in #343.

[minhquoc0712]

Hi @thomaspeyrucain, I have updated id-class information in the document. Can you check if the algorithm working properly?

[thomaspeyrucain]

Hello @minhquoc0712 ,
Would it make sense to add a picture of the gestures from the paper?
I tested it but I am not sure what is the optimal distance for it to work? and if the angle of the camera is important ?

Also, It is often giving a detection of the ID 8 punch with a lot of confidence even if nothing is detected

[minhquoc0712]

Hi @thomaspeyrucain,

I have updated the document with example gestures from the paper.

About the second question, I cannot answer it since I did not implement this algorithm from the beginning. Maybe I will discuss it with my team, or you can bring it up at the technical meeting.

[katerynaCh]

Hi @thomaspeyrucain I am also not the person who implemented the algorithm, but I worked with the provided trained model for the demo last year and I found it to be pretty sensitive wrt distance/position of person/etc. It is due to the fact that the dataset it was trained on is pretty small and does not contain "in-the-wild" images, but only images of several people in a similar lab environment. You can see some examples of the dataset images in https://github.com/opendr-eu/opendr/blob/master/projects/perception/multimodal_human_centric/rgbd_hand_gesture_recognition/ or full dataset in https://data.mendeley.com/datasets/ndrczc35bt/1.

For me the model worked better when making sure the inference setup is similar to the train dataset, i.e., frontal view, person in the center, etc. An example here: https://tuni-my.sharepoint.com/:v:/g/personal/kateryna_chumachenko_tuni_fi/Ef_snAWKevRBl9Z6hIvJwysB2W0WabuL-IjpxOmmnKtb0Q?e=4xe56R