Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Update ubuntu for arm compiler crashes #151

Merged
merged 7 commits into from
Feb 24, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ jobs:
shell: bash
run: |
docker buildx build --platform ${{ matrix.platform }} --output type=docker -t test-runner - < Dockerfile.test-runner
TARGET_DIR=${{ matrix.target_dir }} PLATFORM=${{ matrix.platform }} just test
TARGET_DIR=${{ matrix.target_dir }} PLATFORM=${{ matrix.platform }} just test-ci

# The date/channel/version are expected to be the same on both architectures and are needed for the merge step.
# We store them here since it makes the merge step a bit easier - it doesn't need to figure out which of the
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/stable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ jobs:
shell: bash
run: |
docker buildx build --platform ${{ matrix.platform }} --output type=docker -t test-runner - < Dockerfile.test-runner
TARGET_DIR=${{ matrix.target_dir }} PLATFORM=${{ matrix.platform }} just test
TARGET_DIR=${{ matrix.target_dir }} PLATFORM=${{ matrix.platform }} just test-ci

# The date/channel/version are expected to be the same on both architectures and are needed for the merge step.
# We store them here since it makes the merge step a bit easier - it doesn't need to figure out which of the
Expand Down
30 changes: 15 additions & 15 deletions Dockerfile.arm64
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# syntax=docker/dockerfile:1
FROM ubuntu:jammy
FROM ubuntu:noble
LABEL maintainer="Eirik Albrigtsen <sszynrae@gmail.com>"

# Required packages:
Expand Down Expand Up @@ -54,12 +54,12 @@ RUN chmod a+X /root
# Convenience list of versions and variables for compilation later on
# This helps continuing manually if anything breaks.
ENV SSL_VER="1.1.1w" \
CURL_VER="8.7.1" \
CURL_VER="8.12.1" \
ZLIB_VER="1.3.1" \
PQ_VER="11.12" \
SQLITE_VER="3450300" \
PROTOBUF_VER="25.3" \
SCCACHE_VER="0.8.0" \
SQLITE_VER="3490100" \
PROTOBUF_VER="29.2" \
SCCACHE_VER="0.9.1" \
CC=musl-gcc \
PREFIX=/musl \
PATH=/usr/local/bin:/root/.cargo/bin:$PATH \
Expand Down Expand Up @@ -107,14 +107,14 @@ RUN curl -sSL https://www.openssl.org/source/openssl-$SSL_VER.tar.gz | tar xz &&

# Build curl (needs with-zlib and all this stuff to allow https)
# curl_LDFLAGS needed on stretch to avoid fPIC errors - though not sure from what
RUN curl -sSL https://curl.se/download/curl-$CURL_VER.tar.gz | tar xz && \
cd curl-$CURL_VER && \
CC="musl-gcc -fPIC -pie" LDFLAGS="-L$PREFIX/lib" CFLAGS="-I$PREFIX/include" ./configure \
--enable-shared=no --with-zlib --enable-static=ssl --enable-optimize --prefix=$PREFIX \
--with-ca-path=/etc/ssl/certs/ --with-ca-bundle=/etc/ssl/certs/ca-certificates.crt --without-ca-fallback \
--with-openssl --without-libpsl && \
make -j$(nproc) curl_LDFLAGS="-all-static" && make install && \
cd .. && rm -rf curl-$CURL_VER
# RUN curl -sSL https://curl.se/download/curl-$CURL_VER.tar.gz | tar xz && \
# cd curl-$CURL_VER && \
# CC="musl-gcc -fPIC -pie" LDFLAGS="-L$PREFIX/lib" CFLAGS="-I$PREFIX/include" ./configure \
# --enable-shared=no --with-zlib --enable-static=ssl --enable-optimize --prefix=$PREFIX \
# --with-ca-path=/etc/ssl/certs/ --with-ca-bundle=/etc/ssl/certs/ca-certificates.crt --without-ca-fallback \
# --with-openssl --without-libpsl && \
# make -j$(nproc) curl_LDFLAGS="-all-static" && make install && \
# cd .. && rm -rf curl-$CURL_VER

# Build libpq
RUN curl -sSL https://ftp.postgresql.org/pub/source/v$PQ_VER/postgresql-$PQ_VER.tar.gz | tar xz && \
Expand All @@ -128,11 +128,11 @@ RUN curl -sSL https://ftp.postgresql.org/pub/source/v$PQ_VER/postgresql-$PQ_VER.
cd .. && rm -rf postgresql-$PQ_VER

# Build libsqlite3 using same configuration as the alpine linux main/sqlite package
RUN curl -sSL https://www.sqlite.org/2024/sqlite-autoconf-$SQLITE_VER.tar.gz | tar xz && \
RUN curl -sSL https://www.sqlite.org/2025/sqlite-autoconf-$SQLITE_VER.tar.gz | tar xz && \
cd sqlite-autoconf-$SQLITE_VER && \
CFLAGS="-DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS5 -DSQLITE_ENABLE_COLUMN_METADATA -DSQLITE_SECURE_DELETE -DSQLITE_ENABLE_UNLOCK_NOTIFY -DSQLITE_ENABLE_RTREE -DSQLITE_USE_URI -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_JSON1" \
CC="musl-gcc -fPIC -pie" \
./configure --prefix=$PREFIX --host=x86_64-unknown-linux-musl --enable-threadsafe --enable-dynamic-extensions --disable-shared && \
./configure --prefix=$PREFIX --host=x86_64-unknown-linux-musl --enable-threadsafe --disable-shared && \
make && make install && \
cd .. && rm -rf sqlite-autoconf-$SQLITE_VER

Expand Down
2 changes: 1 addition & 1 deletion Dockerfile.test-runner
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM ubuntu:jammy
FROM ubuntu:noble

RUN apt-get update && apt-get install -y \
ca-certificates \
Expand Down
30 changes: 15 additions & 15 deletions Dockerfile.x86_64
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM ubuntu:jammy
FROM ubuntu:noble
LABEL maintainer="Eirik Albrigtsen <sszynrae@gmail.com>"

# Required packages:
Expand Down Expand Up @@ -53,12 +53,12 @@ RUN chmod a+X /root
# Convenience list of versions and variables for compilation later on
# This helps continuing manually if anything breaks.
ENV SSL_VER="1.1.1w" \
CURL_VER="8.7.1" \
CURL_VER="8.12.1" \
ZLIB_VER="1.3.1" \
PQ_VER="11.12" \
SQLITE_VER="3450300" \
PROTOBUF_VER="25.3" \
SCCACHE_VER="0.8.0" \
SQLITE_VER="3490100" \
PROTOBUF_VER="29.2" \
SCCACHE_VER="0.9.1" \
CC=musl-gcc \
PREFIX=/musl \
PATH=/usr/local/bin:/root/.cargo/bin:$PATH \
Expand Down Expand Up @@ -106,14 +106,14 @@ RUN curl -sSL https://www.openssl.org/source/openssl-$SSL_VER.tar.gz | tar xz &&

# Build curl (needs with-zlib and all this stuff to allow https)
# curl_LDFLAGS needed on stretch to avoid fPIC errors - though not sure from what
RUN curl -sSL https://curl.se/download/curl-$CURL_VER.tar.gz | tar xz && \
cd curl-$CURL_VER && \
CC="musl-gcc -fPIC -pie" LDFLAGS="-L$PREFIX/lib" CFLAGS="-I$PREFIX/include" ./configure \
--enable-shared=no --with-zlib --enable-static=ssl --enable-optimize --prefix=$PREFIX \
--with-ca-path=/etc/ssl/certs/ --with-ca-bundle=/etc/ssl/certs/ca-certificates.crt --without-ca-fallback \
--with-openssl --without-libpsl && \
make -j$(nproc) curl_LDFLAGS="-all-static" && make install && \
cd .. && rm -rf curl-$CURL_VER
# RUN curl -sSL https://curl.se/download/curl-$CURL_VER.tar.gz | tar xz && \
# cd curl-$CURL_VER && \
# CC="musl-gcc -fPIC -pie" LDFLAGS="-L$PREFIX/lib" CFLAGS="-I$PREFIX/include" ./configure \
# --enable-shared=no --with-zlib --enable-static=ssl --enable-optimize --prefix=$PREFIX \
# --with-ca-path=/etc/ssl/certs/ --with-ca-bundle=/etc/ssl/certs/ca-certificates.crt --without-ca-fallback \
# --with-openssl --without-libpsl && \
# make -j$(nproc) curl_LDFLAGS="-all-static" && make install && \
# cd .. && rm -rf curl-$CURL_VER

# Build libpq
RUN curl -sSL https://ftp.postgresql.org/pub/source/v$PQ_VER/postgresql-$PQ_VER.tar.gz | tar xz && \
Expand All @@ -127,11 +127,11 @@ RUN curl -sSL https://ftp.postgresql.org/pub/source/v$PQ_VER/postgresql-$PQ_VER.
cd .. && rm -rf postgresql-$PQ_VER

# Build libsqlite3 using same configuration as the alpine linux main/sqlite package
RUN curl -sSL https://www.sqlite.org/2024/sqlite-autoconf-$SQLITE_VER.tar.gz | tar xz && \
RUN curl -sSL https://www.sqlite.org/2025/sqlite-autoconf-$SQLITE_VER.tar.gz | tar xz && \
cd sqlite-autoconf-$SQLITE_VER && \
CFLAGS="-DSQLITE_ENABLE_FTS4 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS5 -DSQLITE_ENABLE_COLUMN_METADATA -DSQLITE_SECURE_DELETE -DSQLITE_ENABLE_UNLOCK_NOTIFY -DSQLITE_ENABLE_RTREE -DSQLITE_USE_URI -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_JSON1" \
CC="musl-gcc -fPIC -pie" \
./configure --prefix=$PREFIX --host=x86_64-unknown-linux-musl --enable-threadsafe --enable-dynamic-extensions --disable-shared && \
./configure --prefix=$PREFIX --host=x86_64-unknown-linux-musl --enable-threadsafe --disable-shared && \
make && make install && \
cd .. && rm -rf sqlite-autoconf-$SQLITE_VER

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,11 @@ For pinned, or historical builds, see the [available tags on dockerhub](https://

The following system libraries are compiled against `musl-gcc`:

- [x] curl ([curl crate](https://github.com/carllerche/curl-rust))
- [x] openssl ([openssl crate](https://github.com/sfackler/rust-openssl))
- [x] pq ([pq-sys crate](https://github.com/sgrif/pq-sys) used by [diesel](https://github.com/diesel-rs/diesel))
- [x] sqlite3 ([libsqlite3-sys crate](https://github.com/jgallagher/rusqlite/tree/master/libsqlite3-sys) used by [diesel](https://github.com/diesel-rs/diesel))
- [x] zlib (used by pq and openssl)
- [~] curl ([curl crate](https://github.com/carllerche/curl-rust)) - dropped [#96](https://github.com/clux/muslrust/issues/96)

We **[try](https://github.com/clux/muslrust/blob/main/update_libs.py)** to keep these up to date.

Expand All @@ -75,11 +75,11 @@ Before we push a new version of muslrust we [test](https://github.com/clux/muslr
- [x] [serde](https://crates.io/crates/serde)
- [x] [diesel](https://crates.io/crates/diesel)
- [x] [hyper](https://crates.io/crates/hyper)
- [x] [curl](https://crates.io/crates/curl)
- [x] [rustls](https://crates.io/crates/rustls)
- [x] [openssl](https://crates.io/crates/openssl)
- [x] [flate2](https://crates.io/crates/flate2)
- [x] [rand](https://crates.io/crates/rand)
- [~] [curl](https://crates.io/crates/curl) - dropped [#96](https://github.com/clux/muslrust/issues/96)

## Caching

Expand Down
22 changes: 14 additions & 8 deletions justfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# See https://just.systems/man/

[private]
default:
Expand All @@ -19,31 +18,38 @@ build-nightly-arm: (_build "nightly" "arm64" "linux/arm64" "arm64")
run:
docker run -v $PWD/test:/volume -w /volume -it rustmusl-temp /bin/bash

# Build test runner
test-setup:
docker build -t test-runner . -f Dockerfile.test-runner

# Test an individual crate against built container
_t crate:
./test.sh {{crate}}

# when running locally use one of these instead of _t
_t_amd crate:
# Test an individual crate locally using env vars set by _t_amd or t_arm
_ti crate:
# poor man's environment multiplex
just _t_{{ os() }}_{{ arch() }} {{crate}}

# when running locally we can use one of these instead of _t
_t_linux_amd64 crate:
#!/bin/bash
# TODO: make a variant for arm here, or do platform inference
export PLATFORM="linux/amd64"
export TARGET_DIR="x86_64-unknown-linux-musl"
export AR="amd64"
./test.sh {{crate}}
_t_arm crate:
_t_macos_aarch64 crate:
#!/bin/bash
export PLATFORM="linux/arm64"
export TARGET_DIR="aarch64-unknown-linux-musl"
export AR="arm64"
./test.sh {{crate}}


# Test all crates against built container
test: (_t "plain") (_t "ssl") (_t "rustls") (_t "pq") (_t "serde") (_t "curl") (_t "zlib") (_t "hyper") (_t "dieselpg") (_t "dieselsqlite")
# Test all crates against built container locally
test: (_ti "plain") (_ti "ssl") (_ti "rustls") (_ti "pq") (_ti "serde") (_ti "zlib") (_ti "hyper") (_ti "dieselpg") (_ti "dieselsqlite")
# Test all crates against built container in ci (inheriting set PLATFORM/TARGET_DIR/AR vars)
test-ci: (_t "plain") (_t "ssl") (_t "rustls") (_t "pq") (_t "serde") (_t "zlib") (_t "hyper") (_t "dieselpg") (_t "dieselsqlite")
# NB: taken out curl for #96 and build issuse

# Cleanup everything
clean: clean-docker clean-tests
Expand Down
Loading