failed to open PyO3 config file at pyo3-cross-compile-config.txt in manylinux2010 docker container

[messense]

Bug Description

error: failed to run custom build command for `pyo3 v0.16.2`

Caused by:
  process didn't exit successfully: `/home/runner/work/maturin-action/maturin-action/maturin/test-crates/pyo3-mixed/target/debug/build/pyo3-241f0dea0e3e2ffc/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at 'failed to parse PyO3 config file: Error { value: "failed to open PyO3 config file at /home/runner/work/maturin-action/maturin-action/maturin/test-crates/pyo3-mixed/target/debug/build/pyo3-build-config-91712567431bbcb4/out/pyo3-cross-compile-config.txt", source: Some(Os { code: 2, kind: NotFound, message: "No such file or directory" }) }', /root/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/pyo3-build-config-0.16.2/src/lib.rs:78:10

After #2224 it still fails with the same kind of error message on Linux.

Steps to Reproduce

https://github.com/messense/maturin-action/runs/5563946271?check_suite_focus=true

Run the following command in manylinux2010 docker image on pyo3-mixed test crate:

export PYO3_CROSS_LIB_DIR=/opt/python/cp38-cp38/lib
maturin build -i python3.8 --no-sdist --manylinux 2010

Backtrace

0: rust_begin_unwind
               at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:498:5
     1: core::panicking::panic_fmt
               at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/panicking.rs:116:14
     2: core::result::unwrap_failed
               at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/result.rs:1690:5
     3: core::result::Result<T,E>::expect
               at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/result.rs:975:23
     4: pyo3_build_config::get::{{closure}}
               at /root/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/pyo3-build-config-0.16.2/src/lib.rs:69:9
     5: once_cell::sync::OnceCell<T>::get_or_init::{{closure}}
               at /root/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/once_cell-1.10.0/src/lib.rs:974:57
     6: once_cell::imp::OnceCell<T>::initialize::{{closure}}
               at /root/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/once_cell-1.10.0/src/imp_std.rs:95:19
     7: once_cell::imp::initialize_inner
               at /root/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/once_cell-1.10.0/src/imp_std.rs:171:31
     8: once_cell::imp::OnceCell<T>::initialize
               at /root/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/once_cell-1.10.0/src/imp_std.rs:93:9
     9: once_cell::sync::OnceCell<T>::get_or_try_init
               at /root/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/once_cell-1.10.0/src/lib.rs:1014:13
    10: once_cell::sync::OnceCell<T>::get_or_init
               at /root/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/once_cell-1.10.0/src/lib.rs:974:19
    11: pyo3_build_config::get
               at /root/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/pyo3-build-config-0.16.2/src/lib.rs:68:5
    12: build_script_build::configure_pyo3
               at ./build.rs:43:30
    13: build_script_build::main
               at ./build.rs:56:21
    14: core::ops::function::FnOnce::call_once
               at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/ops/function.rs:227:5

Your operating system and version

Ubuntu 20.04

Your Python version (python --version)

Python 3.8

Your Rust version (rustc --version)

1.59.0

Your PyO3 version

0.16.2

How did you install python? Did you use a virtualenv?

from manylinux2010 docker image.

Additional Info

No response

[messense]

It seems that the failure only occurs when host is x86_64 and the target is x86_64 or i686.

https://github.com/messense/maturin-action/actions/runs/1990613199

I guess pyo3-build-config determines it's not cross compiling when host == target or when target is 32-bit host architecture, but in some other code paths it determines that it is cross compiling because PYO3_CROSS_LIB_DIR exists.

[messense]

Reproduced in #2230: https://github.com/PyO3/pyo3/runs/5564915289?check_suite_focus=true

[davidhewitt]

Does #2232 fix it? Looks like we should always make the checks only if none of the environment variables are set.

[messense]

Does #2232 fix it? Looks like we should always make the checks only if none of the environment variables are set.

I've pushed the commit to #2230, let's see what happens.

[adamreichold]

I would have expected that the check in https://github.com/PyO3/pyo3/blob/main/pyo3-build-config/src/lib.rs#L73 which is only looking the environment variables would need the same checks for the various target combinations?

[adamreichold]

Looks like we should always make the checks only if none of the environment variables are set.

Ah, this makes sense if setting those variables is supposed to force cross-compilation unconditionally?

[davidhewitt]

I think that check is correct - the cross compile config should always exist when the environment variables are set, yes. The problem is that we seem to be not doing this in some cases.