support namespace packing #167

mozartilize · 2018-05-31T18:12:01Z

I follow the docs here: https://packaging.python.org/guides/packaging-namespace-packages/
but it does not work, maybe poetry not support yet?

uSpike · 2018-06-28T22:32:39Z

How would the maintainers feel about an additional optional field in pyproject.toml called module. For example:

[tool.poetry]
name = "abc-foo"
module = "abc.foo"
...

corresponds with a layout like:

pyproject.toml
abc/
    foo/
        __init__.py

RaptDept · 2018-08-03T15:04:23Z

See #356 (comment)

dfee · 2018-08-08T22:28:28Z

This isn't an issue for me.

[tool.poetry]
name = "abc.foo"
version = "2018.0.0"
description = "abc foo module"
authors = ["Devin Fee <dev...>"]
packages = [
    { include = "abc" },
]

[tool.poetry.dependencies]
python = "^3.6.5"
"abc.core" = {path = "../abc.core"}  # other namespaced modules work, too.

moigagoo · 2018-08-09T16:17:15Z

@dfee what if the package name is different and doesn't correlate with the dir structure?

dfee · 2018-08-09T16:34:39Z

My directory structure is as follows:

repo/
    abc.core/
        pyproject.toml
        abc/
            core/
                __init__.py
    abc.asdf/
        pyproject.toml
        abc/
            asdf/
                __init__.py
    abc.qwerty/
        pyproject.toml
        abc/
            asdf/
                __init__.py

That should explain this line "abc.core" = {path = "../abc.core"} for everyone else.

@moigagoo is your question about something that looks like the following?

repo/  # warning: not my structure
    abc.core/ # same as above
        pyproject.toml
        abc
            core/
                __init__.py
    abc.asdf/ # still just a package root (irrelevant name as I understand it)
        pyproject.toml
        abc.asdf/ # I don't think this complies with PEP420?
            __init__.py
    abc.qwerty/
        pyproject.toml
        my_qwerty/ # I don't think this complies with PEP420?
            __init__.py

Yeah, I just don't know that anything besides my current structure is compliant with PEP420

sdispater · 2019-03-02T11:16:29Z

Poerty support this use case https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages by explicitly declaring the various namespaced packages with the packages property.

geryogam · 2019-09-12T16:07:39Z

The name of a distribution is declared in the tool.poetry.name property of the pyproject.toml file and appears:

in the name of the dist/{distribution_name}-{version}-py3-none-any.whl Wheel distribution file and in the name of the {distribution_name}-{version}.dist-info/ directory of the Wheel distribution file created when the distribution is built;
in the name of the site-packages/{distribution_name}.egg-link symbolic link created when the distribution is installed with edit mode;
in the name of the site-packages/{distribution_name}-{version}.dist-info/ directory created when the distribution is installed without edit mode;
in the output of pip list (under the misleading "Package" column which should be rather named "Distribution") when the distribution is installed.

The name of a root module of a distribution is declared in the file system and appears:

in the name of the {root_module_name}.py file or {root_module_name}/ directory of the Wheel distribution file created when the distribution is built;
in the name of the site-packages/{root_module_name} source code directory created when the distribution is installed without edit mode.

By default, Poetry looks up the root module to include in a distribution by the distribution name declared in the tool.poetry.name property of the pyproject.toml file, first in the directory of the pyproject.toml file, then in the src/ subdirectory if it exists.

If your root module cannot be found by this default mechanism (the root module is located elsewhere, the root module has a different name than that of the distribution or there are multiple root modules), you have to declare it in the tool.poetry.packages property of the pyproject.toml file, as described in the documentation: add a dictionary item to the list for each root module, with an include property declaring its name as @sdispater said earlier (e.g., foo.py for a non-package module, bar for a package, baz/qux for a package with only its qux subpackage) and, if necessary, a from property declaring the relative path of its parent directory (e.g., src or lib).

For instance, let us say that you need to build three distributions named application_1, application_2 and library with the following repository layouts:

application_1
├── .git
├── src
│   └── organization
│       └── application
│           └── application_1
│               ├── __init__.py
│               └── __main__.py
└── pyproject.toml

application_2
├── .git
├── src
│   └── organization
│       └── application
│           └── application_2
│               ├── __init__.py
│               └── __main__.py
└── pyproject.toml

library
├── .git
├── src
│   └── organization
│       └── library
│           ├── library_1
│           │   └── __init__.py
│           ├── library_2
│           │   └── __init__.py
│           └── library_3
│               └── __init__.py
└── pyproject.toml

where application_1/src/organization/application/application_1/__main__.py is:

from organization.library.library_1 import f
from organization.library.library_2 import g

print(f())
print(g())

and application_2/src/organization/application/application_2/__main__.py is:

from organization.library.library_3 import h

print(h())

and library/src/organization/library/library_1/__init__.py is:

def f():
    return "foo"

and library/src/organization/library/library_2/__init__.py is:

def g():
    return "bar"

and library/src/organization/library/library_3/__init__.py is:

def h():
    return "baz"

In order to build the distributions, application_1/pyproject.toml should be:

[tool.poetry]
name = "application_1"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

and application_2/pyproject.toml should be:

[tool.poetry]
name = "application_2"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

and library/pyproject.toml should be:

[tool.poetry]
name = "library"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

wallrite · 2020-06-11T15:50:32Z

This isn't an issue for me.

[tool.poetry]
name = "abc.foo"
version = "2018.0.0"
description = "abc foo module"
authors = ["Devin Fee <dev...>"]
packages = [
    { include = "abc" },
]

[tool.poetry.dependencies]
python = "^3.6.5"
"abc.core" = {path = "../abc.core"}  # other namespaced modules work, too.

@dfee Do you have any problems building your project? I have the same structure and faced problem with poetry build.

I think I've found related issue #2046

pawamoy · 2020-07-22T09:17:55Z

@maggyero, just to be clear, in

[tool.poetry]
name = "application_1"
version = "..."
description = "..."
authors = ["..."]
packages = [
  { include = "organization", from = "src" }
]

...it is not mandatory to use application_1 as name, right? We could use any name we want, and pip will in any case install the application_1 package in the organization.application namespace'd one.

geryogam · 2020-07-22T10:51:47Z

@pawamoy No it is not mandatory to use "application_1" as name, you can call your distribution as you want. The binding between the distribution name and the top-level Python module name "application_1" will be recorded in the file top_level.txt of the distribution.

For instance you can see that the setuptools distribution has three top-level Python modules:

$ cat /usr/local/lib/python3.8/site-packages/setuptools-49.2.0-py3.8.egg-ino/top_level.txt
easy_install
pkg_resources
setuptools

Indeed:

$ ls -l /usr/local/lib/python3.8/site-packages/
total 16
drwxr-xr-x   4 maggyero  admin   128 18 jul 13:40 __pycache__
-rw-r--r--   1 maggyero  admin   126 13 jul 14:37 easy_install.py
lrwxr-xr-x   1 maggyero  admin    81 18 jul 13:41 homebrew-protobuf.pth -> ../../../Cellar/protobuf/3.12.3/lib/python3.8/site-packages/homebrew-protobuf.pth
drwxr-xr-x   7 maggyero  admin   224 18 jul 13:40 pip
drwxr-xr-x   8 maggyero  admin   256 18 jul 13:40 pip-20.1.1-py3.8.egg-info
drwxr-xr-x   6 maggyero  admin   192 18 jul 13:40 pkg_resources
drwxr-xr-x  45 maggyero  admin  1440 18 jul 13:40 setuptools
drwxr-xr-x   9 maggyero  admin   288 18 jul 13:40 setuptools-49.2.0-py3.8.egg-info
-rw-r--r--   1 maggyero  admin  2081 18 jul 13:40 sitecustomize.py
drwxr-xr-x  14 maggyero  admin   448 18 jul 13:40 wheel
drwxr-xr-x   9 maggyero  admin   288 18 jul 13:40 wheel-0.34.2-py3.8.egg-info

Luttik · 2020-07-27T14:40:36Z

Edit I made a dumb comment here that was addressed in #356. tl;dr
If you want to use a nested namespace like organization.domain.tool you can use nested folders (organization/domain/tool) and use this in your package.toml

packages = [
    { include = "organization", from = "src" }  # only use from="src" if that is how you've structured your code.
]

poetry will automatically know how to interpret the folders nested under organization.

mthaak · 2021-09-21T11:51:56Z

It's important that your organization folders in each package does not contain a __init__.py file. See https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages

wlievens · 2021-12-15T08:46:05Z

I fear I don't quite understand this. My use case is that I want to distribute a python package where the package has a different name from the module inside it. However, I still want my module to have a init.py file in it that ideally I write myself. However poetry (or some other tool in the chain) seems to strip out the file when I install the package as dependency. Can't I opt out of the "native namespace" concept (which I admittedly don't understand yet) and just use the different package/module naming as a strategy on its own?

merwok · 2021-12-15T15:14:28Z

Python distributable packages can contain zero or more modules, zero or more packages, zero or more scripts, and data files. It seems that you are looking for poetry options to control the distribution name independently of the package names. If you have trouble with that, you should open a specific ticket, not comment on this one that’s about namespace packages.

wlievens · 2021-12-15T15:46:25Z

That makes sense, thanks. I managed to resolve my issue, leaving me with more questions than answers as to what actually caused it, but that's an altogether different story.

github-actions · 2024-03-02T17:04:20Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

moigagoo mentioned this issue Jun 7, 2018

Allow to define package source dir in pyproject.toml #182

Closed

nicktimko mentioned this issue Aug 1, 2018

How to generate this setup.py for a namespace subpackage? #356

Closed

2 tasks

tmirzoev mentioned this issue Nov 23, 2018

Multiproject support #667

Closed

2 tasks

sdispater closed this as completed Mar 2, 2019

davenquinn mentioned this issue Oct 1, 2021

Python codebase reorganization EarthCubeGeochron/Sparrow#219

Open

mforbes mentioned this issue Jan 6, 2022

Ownership of sphinxcontrib/__init__.py sphinx-contrib/zopeext#5

Closed

github-actions bot locked as resolved and limited conversation to collaborators Mar 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support namespace packing #167

support namespace packing #167

mozartilize commented May 31, 2018

uSpike commented Jun 28, 2018

RaptDept commented Aug 3, 2018

dfee commented Aug 8, 2018

moigagoo commented Aug 9, 2018

dfee commented Aug 9, 2018

sdispater commented Mar 2, 2019

geryogam commented Sep 12, 2019 •

edited

Loading

wallrite commented Jun 11, 2020 •

edited

Loading

pawamoy commented Jul 22, 2020 •

edited

Loading

geryogam commented Jul 22, 2020 •

edited

Loading

Luttik commented Jul 27, 2020 •

edited

Loading

mthaak commented Sep 21, 2021 •

edited

Loading

wlievens commented Dec 15, 2021

merwok commented Dec 15, 2021

wlievens commented Dec 15, 2021

github-actions bot commented Mar 2, 2024

support namespace packing #167

support namespace packing #167

Comments

mozartilize commented May 31, 2018

uSpike commented Jun 28, 2018

RaptDept commented Aug 3, 2018

dfee commented Aug 8, 2018

moigagoo commented Aug 9, 2018

dfee commented Aug 9, 2018

sdispater commented Mar 2, 2019

geryogam commented Sep 12, 2019 • edited Loading

wallrite commented Jun 11, 2020 • edited Loading

pawamoy commented Jul 22, 2020 • edited Loading

geryogam commented Jul 22, 2020 • edited Loading

Luttik commented Jul 27, 2020 • edited Loading

mthaak commented Sep 21, 2021 • edited Loading

wlievens commented Dec 15, 2021

merwok commented Dec 15, 2021

wlievens commented Dec 15, 2021

github-actions bot commented Mar 2, 2024

geryogam commented Sep 12, 2019 •

edited

Loading

wallrite commented Jun 11, 2020 •

edited

Loading

pawamoy commented Jul 22, 2020 •

edited

Loading

geryogam commented Jul 22, 2020 •

edited

Loading

Luttik commented Jul 27, 2020 •

edited

Loading

mthaak commented Sep 21, 2021 •

edited

Loading