Error 500 when pushing a docker multiarch image

[Skydust2B]

Description

Ever since I upgraded to gitea 1.23.0-rc0, I am not able to push to the docker registry multi-arch images, with simple image also randomly failing to push.
Everything was working fine before I upgraded the docker image when the gitea instance suddenly started giving me "package version already exists" errors.

I have joined logs of me running the command on demo.gitea.com and my own server. Let me know if you need more infos.

Gitea Version

1.23.0-rc0

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

https://gist.github.com/Skydust2B/cb365129eec75cac93b1d927d26e5ff4

FROM public.ecr.aws/docker/library/node:22-alpine3.21

RUN apk add --no-cache bash git curl jq

# Vulnerability fixes
RUN npm install -g npm@latest

ENTRYPOINT ["/bin/bash"]

$ docker build . -t <serverurl>/skydust/multiarchtest:latest --platform linux/amd64,linux/arm64 --push

...

ERROR: failed commit on ref "manifest-sha256:1201f32e3f851ed5eb8e8f9e8df297e7c6b6aa64c849fe4de75a2f95872fcc34": unexpected status from PUT request to https://<serverurl>//v2/skydust/multiarchtest/manifests/latest: 500 Internal Server Error

...

2024/12/27 20:49:47 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/skydust/multiarchtest/blobs/sha256:ea542b86a0afcd40ccb12aa90fce8a8cf7d8b9798e8ae5ec545087bb
fa70600a for 10.0.0.118:48888, 200 OK in 113.4ms @ container/container.go:501(container.HeadBlob)
2024/12/27 20:49:47 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/skydust/multiarchtest/blobs/sha256:274663461bd74df9f025b6e29c963c0583cb98dbb8641dd609119431
fe17d4a4 for [fd00::46]:35924, 200 OK in 125.7ms @ container/container.go:501(container.HeadBlob)
2024/12/27 20:49:47 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/skydust/multiarchtest/blobs/sha256:796da77e13076b9cd79d52eb61ff3b36816a68ed04bc7f701702679d
c2909442 for 10.0.0.118:48878, 200 OK in 122.9ms @ container/container.go:501(container.HeadBlob)
2024/12/27 20:49:47 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/skydust/multiarchtest/manifests/latest for [fd00::46]:35996, 200 OK in 6.2ms @ container/co
ntainer.go:639(container.HeadManifest)
2024/12/27 20:49:47 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/skydust/multiarchtest/manifests/latest for [fd00::46]:36008, 200 OK in 6.5ms @ container/co
ntainer.go:639(container.HeadManifest)
2024/12/27 20:49:47 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/skydust/multiarchtest/manifests/latest for 10.0.0.118:48916, 200 OK in 27.2ms @ container/c
ontainer.go:639(container.HeadManifest)
2024/12/27 20:49:47 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/skydust/multiarchtest/manifests/latest for 10.0.0.118:48920, 200 OK in 40.5ms @ container/c
ontainer.go:639(container.HeadManifest)
2024/12/27 20:49:48 ...ontainer/manifest.go:364:createPackageAndVersion() [E] Error inserting package: package version already exists
2024/12/27 20:49:48 ...ontainer/manifest.go:364:createPackageAndVersion() [E] Error inserting package: package version already exists
2024/12/27 20:49:48 ...ontainer/manifest.go:364:createPackageAndVersion() [E] Error inserting package: package version already exists
2024/12/27 20:49:48 ...ntainer/container.go:92:apiError() [E] package version already exists
2024/12/27 20:49:48 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/skydust/multiarchtest/manifests/latest for [fd00::46]:36024, 500 Internal Server Error in 52
.7ms @ container/container.go:554(container.UploadManifest)
2024/12/27 20:49:48 ...ntainer/container.go:92:apiError() [E] package version already exists
2024/12/27 20:49:48 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/skydust/multiarchtest/manifests/latest for 10.0.0.118:48928, 500 Internal Server Error in 76
.4ms @ container/container.go:554(container.UploadManifest)
2024/12/27 20:49:48 ...ntainer/container.go:92:apiError() [E] package version already exists
2024/12/27 20:49:48 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/skydust/multiarchtest/manifests/latest for 10.0.0.118:48930, 500 Internal Server Error in 41
.3ms @ container/container.go:554(container.UploadManifest)
2024/12/27 20:49:48 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/skydust/multiarchtest/manifests/latest for [fd00::46]:36010, 201 Created in 90.0ms @ contain
er/container.go:554(container.UploadManifest)

Git Version

2.45.2

Operating System

Docker

How are you running Gitea?

I'm running gitea inside kubernetes using the official gitea/gitea:1.23.0-rc0 container.

Database

PostgreSQL

[wxiaoguang]

Some related:

• Fix container parallel upload bugs #32022
• Use per package global lock for container uploads instead of memory lock #31860
• Workaround for container registry push/pull errors #21862

But not completely fixed

@KN4CK3R

[hiifong]

I recommend you use docker buildx to build multi-architecture images

[Skydust2B]

I recommend you use docker buildx to build multi-architecture images

I should have mentioned, I'm using docker-desktop on a mac and it's using buildkit by default. My CI has the same errors and directly uses docker buildx build.

[hiifong]

Since I can't reproduce this issue locally, it would be great if you could use Wireshark to capture the push network request and provide more detailed logs (trace level).

[Skydust2B]

I tried to capture everything with wireshark, but I'm not sure how useful it will be considering I wasn't able to decrypt the tls packets.
Sorry if this is kind of messy, but here is what I did:
I went on a windows machine, installed docker desktop, set SSLKEYLOGFILE to a usable path. I then enabled containerd, restarted docker desktop, and ran the same command as above for demo.gitea.com (which seemed to have some hiccups with the login token).
I made wireshark follow the path to my log file, disabled my ipv6, and captured all packets going to the ipv4 of demo.gitea.com while building and pushing. (EDIT: Added a better file below, removed the link for clarity)

Because you couldn't reproduce it, I've tried using my mobile network on my mac and pushed again, only to fail to the same error.

EDIT:
I used a proxy, tried a docker push of the same built image, and injected the secrets into the dump file:
capture.pcapng.zip

[wxiaoguang]

I see the problem, will fix it

[wxiaoguang]

could you try 1.23-nightly? I think it should have been fixed there

• https://dl.gitea.com/gitea/1.23-nightly/
• https://hub.docker.com/r/gitea/gitea/tags?name=1.23-nightly

[hiifong]

It works well for me here.

[Skydust2B]

I do not have any issues using the last nightly 👍Thank you for the speedy resolution