ModuleNotFoundError when building with --simlink-install #141

BrunoB81HK · 2023-08-22T16:23:18Z

When building the example_python package inside my workspace, I get the ModuleNotFoundError. This happens everytime I try to build with --symlink-install. I first encountered this bug with my package and I decided to try the example directly to see if it happened with it too and it does. here is the traceback:

Traceback (most recent call last):
  File "/home/user/colcon_ws/install/generate_parameter_module_example/lib/generate_parameter_module_example/test_node", line 33, in <module>
    sys.exit(load_entry_point('generate-parameter-module-example', 'console_scripts', 'test_node')())
  File "/home/user/colcon_ws/install/generate_parameter_module_example/lib/generate_parameter_module_example/test_node", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/user/colcon_ws/build/generate_parameter_module_example/generate_parameter_module_example/minimal_publisher.py", line 33, in <module>
    from generate_parameter_module_example.admittance_parameters import (
ModuleNotFoundError: No module named 'generate_parameter_module_example.admittance_parameters'

I also checked my PYTHONPATH and I think that it may be related to name collisions.

I saw that @pac48 was referencing to some side-effects when using --symlink-install in #122, is this related?

The text was updated successfully, but these errors were encountered:

pac48 · 2023-08-22T17:23:50Z

@BrunoB81HK Which version of Ubuntu and ROS 2 are you using? I can build with --symlink-install and run the example on this image no problem.

BrunoB81HK · 2023-08-22T18:03:07Z

I'm using humble with Ubuntu 22.04. We are also using docker and our container is generated from this one.

pac48 · 2023-08-22T18:11:04Z

Are you building with any extra options? When I run colcon build --symlink-install to build and ros2 run generate_parameter_module_example test_node --ros-args --params-file src/generate_parameter_library/example_python/config/implementation.yaml to run, it works even on the image you mentioned.

BrunoB81HK · 2023-08-22T18:33:18Z

OK, that is weird. The only other that we use is the following --cmake-args -DCMAKE_BUILD_TYPE=Release. Maybe the error is on my side.

pac48 · 2023-08-22T18:43:17Z

Okay, that is strange. The only other thing I'd suggest is to ensure that rosdep is run: rosdep install --from-paths src --ignore-src -y before building and running.

BrunoB81HK · 2023-08-23T13:26:34Z

rosdep install --from-paths src --ignore-src -y is indeed called before building and running.

For my custom package, python can find it, but I think its name collide with another package so that parameters.py is not found.
I compiled the differences between a code built with and without --symink-install below.

In my case, using --symink-install cause a duplicated module in /colcon_ws/build that is added in the python path and, since this module is a symlink to the /colcon_ws/src, it doesn't contain the generated parameters.py.
The generated parameters.py is theoretically reachable from /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages, but is hidden behind the one in /home/user/colcon_ws/build/pkg_name.

The most surprising thing to me is that it happens to me but not to you. @pac48, do you have any idea why this happens and how I could remedy this problem? Is it a problem with my setup? Or a problem with my module naming?

Installed with `--symlink-install`

Executed script (from `file`):

/home/user/colcon_ws/build/pkg_name/pkg_name/script.py

Python path (from `sys.path`):

Paths
`1️⃣ /home/user/colcon_ws/install/pkg_name/lib/pkg_name`
`2️⃣ /home/user/colcon_ws/build/pkg_name`
`3️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages`
`/opt/ros/humble/lib/python3.10/site-packages`
`/opt/ros/humble/local/lib/python3.10/dist-packages`
`/usr/lib/python310.zip`
`/usr/lib/python3.10`
`/usr/lib/python3.10/lib-dynload`
`/usr/local/lib/python3.10/dist-packages`
`/usr/lib/python3/dist-packages`

Reachable modules (from `sys.modules`):

Module	Location
pkg_name.script	`2️⃣ /home/user/colcon_ws/build/pkg_name`
pkg_name	`2️⃣ /home/user/colcon_ws/build/pkg_name`

Tree of the directory:

colcon_ws
├── build
│   ├── COLCON_IGNORE
│   └── 2️⃣ pkg_name
│       ├── pkg_name -> /home/user/colcon_ws/src/pkg_name/pkg_name
│       ├── pkg_name.egg-info
│       │   └── ...
│       ├── package.xml -> /home/user/colcon_ws/src/pkg_name/package.xml
│       ├── resource
│       │   └── pkg_name -> /home/user/colcon_ws/src/pkg_name/resource/pkg_name
│       ├── setup.cfg -> /home/user/colcon_ws/src/pkg_name/setup.cfg
│       ├── setup.py -> /home/user/colcon_ws/src/pkg_name/setup.py
│       ├── ...
│       └── share
│           └── pkg_name
│               └── hook
│                   ├── pythonpath_develop.dsv
│                   ├── pythonpath_develop.ps1
│                   └── pythonpath_develop.sh
├── install
│   ├── COLCON_IGNORE
│   ├── pkg_name
│   │   ├── lib
│   │   │   ├── 1️⃣ pkg_name
│   │   │   │   └── entry_point
│   │   │   └── python3.10
│   │   │       └── 3️⃣ site-packages
│   │   │           ├── pkg-name.egg-link
│   │   │           └── pkg_name
│   │   │               └── parameters.py
│   │   └── share
│   │       ├── ...
│   │       └── colcon-core
│   │           └── packages
│   │               └── pkg_name
│   └── ...
└── src
    └── pkg_name
        ├── pkg_name
        │   ├── __init__.py
        │   ├── script.py
        │   └── parameters.yaml
        ├── package.xml
        ├── resource
        │   └── pkg_name
        ├── setup.cfg
        └── setup.py

Installed without `--symlink-install`

Executed script (from `file`):

/home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages/pkg_name/script.py

Python path (from `sys.path`):

Paths
`1️⃣ /home/user/colcon_ws/install/pkg_name/lib/pkg_name`
`2️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages`
`/opt/ros/humble/lib/python3.10/site-packages`
`/opt/ros/humble/local/lib/python3.10/dist-packages`
`/usr/lib/python310.zip`
`/usr/lib/python3.10`
`/usr/lib/python3.10/lib-dynload`
`/usr/local/lib/python3.10/dist-packages`
`/usr/lib/python3/dist-packages`

Reachable modules (from `sys.modules`):

Module	Location
pkg_name.script	`2️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages`
pkg_name	`2️⃣ /home/user/colcon_ws/install/pkg_name/lib/python3.10/site-packages`

Tree of the directory:

colcon_ws
├── build
│   ├── COLCON_IGNORE
│   └── pkg_name
│       ├── pkg_name
│       │   └── parameters.py
│       ├── pkg_name.egg-info
│       │   └── ...
│       ├── build
│       │   └── lib
│       │       └── pkg_name
│       │           ├── __init__.py
│       │           └── script.py
│       └── ...
├── install
│   ├── COLCON_IGNORE
│   ├── pkg_name
│   │   ├── lib
│   │   │   ├── 1️⃣ pkg_name
│   │   │   │   └── entry_point
│   │   │   └── python3.10
│   │   │       └── 2️⃣ site-packages
│   │   │           ├── pkg_name-0.1.0-py3.10.egg-info
│   │   │           │   └── ...
│   │   │           └── pkg_name
│   │   │               ├── __init__.py
│   │   │               ├── __pycache__
│   │   │               │   └── ...
│   │   │               ├── script.py
│   │   │               └── parameters.py
│   │   └── share
│   │       ├── ...
│   │       └── colcon-core
│   │           └── packages
│   │               └── pkg_name
│   └── ...
└── src
    └── pkg_name
        ├── pkg_name
        │   ├── __init__.py
        │   ├── script.py
        │   └── parameters.yaml
        ├── package.xml
        ├── resource
        │   └── pkg_name
        ├── setup.cfg
        └── setup.py

Difference in the directory structure

  colcon_ws
  ├── build
  │   ├── COLCON_IGNORE
  │   └── pkg_name
- │       ├── pkg_name
- │       │   └── parameters.py
+ │       ├── pkg_name -> /home/user/colcon_ws/src/pkg_name/pkg_name
  │       ├── pkg_name.egg-info
  │       │   └── ...
- │       ├── build
- │       │   └── lib
- │       │       └── pkg_name
- │       │           ├── __init__.py
- │       │           └── script.py
+ │       ├── package.xml -> /home/user/colcon_ws/src/pkg_name/package.xml
+ │       ├── resource
+ │       │   └── pkg_name -> /home/user/colcon_ws/src/pkg_name/resource/pkg_name
+ │       ├── setup.cfg -> /home/user/colcon_ws/src/pkg_name/setup.cfg
+ │       ├── setup.py -> /home/user/colcon_ws/src/pkg_name/setup.py
+ │       └── share
+ │           └── pkg_name
+ │               └── hook
+ │                   ├── pythonpath_develop.dsv
+ │                   ├── pythonpath_develop.ps1
+ │                   └── pythonpath_develop.sh
  ├── install
  │   ├── COLCON_IGNORE
  │   ├── pkg_name
  │   │   ├── lib
  │   │   │   ├── pkg_name
  │   │   │   │   └── entry_point
  │   │   │   └── python3.10
  │   │   │       └── site-packages
- │   │   │           ├── pkg_name-0.1.0-py3.10.egg-info
- │   │   │           │   └── ...
+ │   │   │           ├── pkg-name.egg-link
  │   │   │           └── pkg_name
- │   │   │               ├── __init__.py
- │   │   │               ├── __pycache__
- │   │   │               │   └── ...
- │   │   │               ├── script.py
  │   │   │               └── parameters.py
  │   │   └── share
  │   │       ├── ...
  │   │       └── colcon-core
  │   │           └── packages
  │   │               └── pkg_name
  │   └── ...
  └── src
      └── pkg_name
          ├── pkg_name
          │   ├── __init__.py
          │   ├── script.py
          │   └── parameters.yaml
          ├── package.xml
          ├── resource
          │   └── pkg_name
          ├── setup.cfg
          └── setup.py

BrunoB81HK · 2023-08-29T18:33:26Z

@pac48 is there any way you can share me the Dockerfile you used that worked for you?

pac48 · 2023-08-30T17:57:27Z

@BrunoB81HK this is the image that I used. Its called osrf/ros:humble-desktop-full

BrunoB81HK · 2023-09-18T12:02:03Z

After much investigating, I found out that the probleme lied in my Dockerfile configuration. The build process was not saving any changes made inside the src/ directory. I'll close this issue since it is not a bug. Thanks a lot for your help!

ahi-bplus · 2024-04-15T08:31:44Z

@BrunoB81HK can you share the changes you made to fix the problem? I have exactly the same problem with --symlink-install

BrunoB81HK · 2024-04-16T00:09:23Z

@ahi-bplus Are you building the project inside your Dockerfile? If so, the problem for me was pretty simple.

When using --symlink-install, it creates a new module inside the src/ directory. The module is accessible because, if I remember correctly, the whole python module directory (the one who contains the __init__.py file) is symlinked, not just the files. This is not a problem in itself, but becomes one if you mount your src/ into the container. If you do so, the module inside the install/ directory changes to the one mounted in your src/ directory. For that reason, the parameter module is overwritten.

I can give you some ideas to help with your case:

Avoid using --symlink-install inside the Dockerfile.
Build your project inside the docker container, not the Dockerfile.
If used for development, start using devcontainers.
Find a way to avoid mounting your src/ directory inside the container.

I hope it helps, feel free to ask more questions if you need! :)

ahi-bplus · 2024-04-16T05:16:51Z

@BrunoB81HK Thanks for your quick reply. I already use a devcontainer for development. I mount my src files accordingly. As soon as I build my ROS node which implements generate_parameter_library using colcon build --symlink-install --merge-install, I get the problem with ModuleNotFoundError. Without --symlink-install --merge-install it works.

ahi-bplus · 2024-04-16T07:53:21Z

Okay, I have found the problem. It has nothing to do with --symlink-install or --merge-install. The problem comes from my package structure.

package_name/
├── README.md
├── __init__.py
├── config
│   └── default
│       └── default.yaml
├── models
├── package.xml
├── requirements.txt
├── resource
│   └── package_name
├── setup.cfg
├── setup.py
├── src
│   └── package_name
│       ├── __init__.py
│       ├── node.py
│       ├── parameters.yaml

Unfortunately the package_dir={'': 'src'}, of generate_parameter_library is not considered. If I avoid the subdirecory src/package_name everything works. I think this would be worth an issue?

BrunoB81HK · 2024-04-16T10:04:49Z

If you think this is a bug, you should create a new issue. Even if it's not, that will give others that have the same problem some help too.

BrunoB81HK closed this as completed Sep 18, 2023

imcelroy mentioned this issue Jan 7, 2025

ModuleNotFoundError python unit test #233

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ModuleNotFoundError when building with --simlink-install #141

ModuleNotFoundError when building with --simlink-install #141

BrunoB81HK commented Aug 22, 2023

pac48 commented Aug 22, 2023 •

edited

Loading

BrunoB81HK commented Aug 22, 2023

pac48 commented Aug 22, 2023 •

edited

Loading

BrunoB81HK commented Aug 22, 2023

pac48 commented Aug 22, 2023

BrunoB81HK commented Aug 23, 2023 •

edited

Loading

BrunoB81HK commented Aug 29, 2023

pac48 commented Aug 30, 2023

BrunoB81HK commented Sep 18, 2023

ahi-bplus commented Apr 15, 2024

BrunoB81HK commented Apr 16, 2024

ahi-bplus commented Apr 16, 2024

ahi-bplus commented Apr 16, 2024

BrunoB81HK commented Apr 16, 2024

ModuleNotFoundError when building with --simlink-install #141

ModuleNotFoundError when building with --simlink-install #141

Comments

BrunoB81HK commented Aug 22, 2023

pac48 commented Aug 22, 2023 • edited Loading

BrunoB81HK commented Aug 22, 2023

pac48 commented Aug 22, 2023 • edited Loading

BrunoB81HK commented Aug 22, 2023

pac48 commented Aug 22, 2023

BrunoB81HK commented Aug 23, 2023 • edited Loading

Installed with --symlink-install

Executed script (from __file__):

Python path (from sys.path):

Reachable modules (from sys.modules):

Tree of the directory:

Installed without --symlink-install

Executed script (from __file__):

Python path (from sys.path):

Reachable modules (from sys.modules):

Tree of the directory:

Difference in the directory structure

BrunoB81HK commented Aug 29, 2023

pac48 commented Aug 30, 2023

BrunoB81HK commented Sep 18, 2023

ahi-bplus commented Apr 15, 2024

BrunoB81HK commented Apr 16, 2024

ahi-bplus commented Apr 16, 2024

ahi-bplus commented Apr 16, 2024

BrunoB81HK commented Apr 16, 2024

pac48 commented Aug 22, 2023 •

edited

Loading

pac48 commented Aug 22, 2023 •

edited

Loading

BrunoB81HK commented Aug 23, 2023 •

edited

Loading

Installed with `--symlink-install`

Executed script (from `file`):

Python path (from `sys.path`):

Reachable modules (from `sys.modules`):

Installed without `--symlink-install`

Executed script (from `file`):

Python path (from `sys.path`):

Reachable modules (from `sys.modules`):