-
Notifications
You must be signed in to change notification settings - Fork 95
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Intent recognition tool #443
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello again @katerynaCh. Thank you for the fixes so far.
I ran some more tests and debugging and added more comments. I also went ahead and applied some changes required for ROS1/2 in files that were previously not changed (it would make it awkward for the review process otherwise).
projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py
Outdated
Show resolved
Hide resolved
projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py
Outdated
Show resolved
Hide resolved
projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py
Outdated
Show resolved
Hide resolved
projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py
Outdated
Show resolved
Hide resolved
projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py
Show resolved
Hide resolved
projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py
Outdated
Show resolved
Hide resolved
projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py
Outdated
Show resolved
Hide resolved
projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py
Outdated
Show resolved
Hide resolved
projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py
Outdated
Show resolved
Hide resolved
…ognition/demo_speech.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ognition/demo_speech.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ognition/demo_speech.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ognition/demo_speech.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ognition/demo_speech.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ion_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I went ahead and added the fix from #465 to the new ROS1 node.
Thank you very much @katerynaCh, everything looks good to me!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
* init commit * update readme * update readme * fix ros node readme * fix ros node readmes * style fixes * style fixes * style fixes * style * fix ros node data type * ros data type fix * fixed unused import * license fix * style fix * style fix * license fix * change the way text backbone is defined in demos and ros nodes * change file path * added missing import * ditto * fixed download paths in python demos * style fix * style fix * fixed ros nodes to be consistent with last changes in speech transcription * bug fix in ros node * bug fix * removed unused function * Added __init__.py to intent recognition test * Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/python/perception/multimodal_human_centric/intent_recognition/README.MD Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update docs/reference/intent-recognition-learner.md Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * updated nltk download path; unittest dataset length; docs; order of methods in learner * fix error in test * fix pandas version-related bug * test * test * Added intent recognition dependency on hri msgs and added node to catkin_install_python * Added intent recognition in ros2 setup.py * Update projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/opendr_ws/src/opendr_perception/README.md Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/opendr_ws_2/src/opendr_perception/README.md Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/opendr_ws_2/src/opendr_perception/README.md Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/opendr_ws/src/opendr_perception/scripts/intent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/opendr_ws/src/opendr_perception/scripts/intent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * test fix * Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com> * Added fix from opendr-eu#465 to new node --------- Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
This PR implements an intent recognition tool that can classify text data into the following intents: [ 'Complain', 'Praise', 'Apologise', 'Thank', 'Criticize', 'Agree', 'Taunt', 'Flaunt', 'Joke', 'Oppose', 'Comfort', 'Care', 'Inform', 'Advise', 'Arrange', 'Introduce', 'Leave', 'Prevent', 'Greet', 'Ask for help' ]. This tool is meant to be used together with the speech transcription tool for speech-to-text conversion and hence the ROS nodes are expected to subscribe to output of speech transcription ROS node. Similarly, python demo is provided with integration of the speech transcription tool.
The model implements an architecture that is trained in a multimodal manner (audio, visual, text) with the goal of improving inference on one of the modalities, in our case, text. The reference to the paper describing the approach will be added in the places where it is missing soon, once the paper/preprint is published. We provide several pretrained models of different sizes, from bert-tiny to bert-base.
This tool can be fully tested once the speech transcription (whisper) tool (#433) is integrated (or by just reusing its code) but I would rather start the initial review sooner if possible.
I am not sure why some tools' tests are failing, but seems unrelated to this PR?