Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Intent recognition tool #443

Merged
merged 72 commits into from
Sep 25, 2023
Merged

Intent recognition tool #443

merged 72 commits into from
Sep 25, 2023

Conversation

katerynaCh
Copy link
Collaborator

@katerynaCh katerynaCh commented Jun 30, 2023

This PR implements an intent recognition tool that can classify text data into the following intents: [ 'Complain', 'Praise', 'Apologise', 'Thank', 'Criticize', 'Agree', 'Taunt', 'Flaunt', 'Joke', 'Oppose', 'Comfort', 'Care', 'Inform', 'Advise', 'Arrange', 'Introduce', 'Leave', 'Prevent', 'Greet', 'Ask for help' ]. This tool is meant to be used together with the speech transcription tool for speech-to-text conversion and hence the ROS nodes are expected to subscribe to output of speech transcription ROS node. Similarly, python demo is provided with integration of the speech transcription tool.

The model implements an architecture that is trained in a multimodal manner (audio, visual, text) with the goal of improving inference on one of the modalities, in our case, text. The reference to the paper describing the approach will be added in the places where it is missing soon, once the paper/preprint is published. We provide several pretrained models of different sizes, from bert-tiny to bert-base.

This tool can be fully tested once the speech transcription (whisper) tool (#433) is integrated (or by just reusing its code) but I would rather start the initial review sooner if possible.

I am not sure why some tools' tests are failing, but seems unrelated to this PR?

@katerynaCh katerynaCh added test sources Run style checks test tools Test the toolkit methods labels Jun 30, 2023
@katerynaCh katerynaCh marked this pull request as ready for review July 21, 2023 14:20
Copy link
Collaborator

@tsampazk tsampazk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello again @katerynaCh. Thank you for the fixes so far.

I ran some more tests and debugging and added more comments. I also went ahead and applied some changes required for ROS1/2 in files that were previously not changed (it would make it awkward for the review process otherwise).

tsampazk and others added 22 commits September 21, 2023 16:40
…ognition/demo_speech.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ognition/demo_speech.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ognition/demo_speech.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ognition/demo_speech.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ognition/demo_speech.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ntent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
…ion_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
Copy link
Collaborator

@tsampazk tsampazk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went ahead and added the fix from #465 to the new ROS1 node.

Thank you very much @katerynaCh, everything looks good to me!

Copy link
Collaborator

@passalis passalis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@tsampazk tsampazk merged commit 30817fe into develop Sep 25, 2023
@tsampazk tsampazk deleted the intent-recognition branch September 25, 2023 09:07
lucamarchionni pushed a commit to lucamarchionni/opendr that referenced this pull request Jun 10, 2024
* init commit

* update readme

* update readme

* fix ros node readme

* fix ros node readmes

* style fixes

* style fixes

* style fixes

* style

* fix ros node data type

* ros data type fix

* fixed unused import

* license fix

* style fix

* style fix

* license fix

* change the way text backbone is defined in demos and ros nodes

* change file path

* added missing import

* ditto

* fixed download paths in python demos

* style fix

* style fix

* fixed ros nodes to be consistent with last changes in speech transcription

* bug fix in ros node

* bug fix

* removed unused function

* Added __init__.py to intent recognition test

* Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/python/perception/multimodal_human_centric/intent_recognition/README.MD

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update docs/reference/intent-recognition-learner.md

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* updated nltk download path; unittest dataset length; docs; order of methods in learner

* fix error in test

* fix pandas version-related bug

* test

* test

* Added intent recognition dependency on hri msgs and added node to catkin_install_python

* Added intent recognition in ros2 setup.py

* Update projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/python/perception/multimodal_human_centric/intent_recognition/demo_speech.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/opendr_ws/src/opendr_perception/README.md

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/opendr_ws_2/src/opendr_perception/README.md

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/opendr_ws_2/src/opendr_perception/README.md

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/opendr_ws/src/opendr_perception/scripts/intent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/opendr_ws/src/opendr_perception/scripts/intent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Update projects/opendr_ws_2/src/opendr_perception/opendr_perception/intent_recognition_node.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* test fix

* Update src/opendr/perception/multimodal_human_centric/intent_recognition_learner/intent_recognition_learner.py

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>

* Added fix from opendr-eu#465 to new node

---------

Co-authored-by: Kostas Tsampazis <27914645+tsampazk@users.noreply.github.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
test sources Run style checks test tools Test the toolkit methods
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants