Linking Python to the latest sqlite on Amazon Linux 2

[AlJohri]

I am trying to install python using the latest version of sqlite. I am doing this because airflow requires a very recent version of SQLite:

airflow.exceptions.AirflowConfigException: error: SQLite C library too old (< 3.15.0). See [https://airflow.apache.org/docs/apache-airflow/2.7.1/howto/set-up-database.html#setting-up-a-sqlite-database](https://airflow.apache.org/docs/apache-airflow/2.7.1/howto/set-up-database.html#setting-up-a-sqlite-database)

I have installed sqlite using the following commands:

wget https://sqlite.org/2023/sqlite-autoconf-3430100.tar.gz
tar -xvf sqlite-autoconf-3430100.tar.gz
cd sqlite-autoconf-3430100
./configure
make clean && make -j 20
sudo make install

Now when I try to install python (using rtx which uses python-build under the hood), I get the following error:

❯ rtx install python 3.11.5
Downloading Python-3.11.5.tar.xz...
-> https://www.python.org/ftp/python/3.11.5/Python-3.11.5.tar.xz
Installing Python-3.11.5...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/aljohri/.local/share/rtx/installs/python/3.11.5/lib/python3.11/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "/home/aljohri/.local/share/rtx/installs/python/3.11.5/lib/python3.11/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?
Installed Python-3.11.5 to /home/aljohri/.local/share/rtx/installs/python/3.11.5
rtx python@3.11.5                                                                       ✓ 58s

In order to get it to work properly, I need to set the following flags and run it again.

❯ rtx uninstall python 3.11.5

❯ export CPPFLAGS="${CPPFLAGS:+$CPPFLAGS }-I/usr/local/include"
❯ export LDFLAGS="${LDFLAGS:+$LDFLAGS }-L/usr/local/lib -Wl,-rpath,/usr/local/lib"

❯ rtx install python 3.11.5
Downloading Python-3.11.5.tar.xz...
-> https://www.python.org/ftp/python/3.11.5/Python-3.11.5.tar.xz
Installing Python-3.11.5...
Installed Python-3.11.5 to /home/aljohri/.local/share/rtx/installs/python/3.11.5
rtx python@3.11.5                                                                       ✓ 54s[WARN] rtx::toolset: Tool not installed: python@3.11.1 (install with: rtx install)

And now we can see that it installed properly and is using the latest sqlite3.

❯ rtx use python@3.11.5
❯ python --version
Python 3.11.5
❯ python -c "import sqlite3; print(sqlite3.sqlite_version)"
3.43.1

Multiple questions:

1. I set the CPPFLAGS and LDFLAGS to a pretty standard location, i.e. /usr/local/include and /usr/local/lib. Is there a way this can automatically be detected? I am using Amazon Linux 2 and I installed sqlite using the default location via sudo make install. I got these flags from the python-build source code itself here which seems to get autoset for FreeBSD systems.

2. I found that setting export LD_RUN_PATH=/usr/local/lib did not work and I needed to manually set CPPFLAGS and LDFLAGS-- any ideas why that might be the case?

[PIoneer2]

the issue and its solution worked for me on Amazon Linux 2 for plain Python3.11 - without "pyenv". Looks like a Python3.9+ issue.

[native-api]

I set the CPPFLAGS and LDFLAGS to a pretty standard location, i.e. /usr/local/include and /usr/local/lib.

Not all distros configure their compilers (and ld -- hence the need for -Wl,-rpath) to search in /usr/local by default. BSD definitely doesn't -- which Ports take advantage of. Maybe Amazon Linux is also one of those.

See https://stackoverflow.com/questions/4980819/what-are-the-gcc-default-include-directories on how to find out GCC's default search directories.