Package registry: large packages download with size of 0B

[dorianim]

Description

When uploading a package to the generic registry, the size is correctly stated in the UI:

When downloading it, however, it has a size of 0B:

Gitea logs:
2022/08/01 23:21:08 [62e843c4] router: completed GET /bluerock/-/packages/generic/app/0-preview-2f3f0488da9755d3e1378fd33d95aceec86a1058/files/160 for 46.114.95.2:0, 0 in 25.2ms @ user/package.go:370(user.DownloadPackageFile)

Gitea Version

1.17.0

Can you reproduce the bug on the Gitea demo site?

No, maybe because it is on 1.18.0+dev-198-g4f14c6de1?

How are you running Gitea?

I am using the gitea/gitea docker image on the latest tag

Database

PostgreSQL

[wxiaoguang]

Are you using any reverse-proxy, or can you try to download the file from Gitea HTTP port directly? The content-length header doesn't seem to be generated by Gitea.

[dorianim]

Thanks for your reply! Unfortunately, this does not seem to be the reason.
My instance is indeed running behind a trafik reverse proxy, but when curling it directly, the content length header still exists and is 0:

[wxiaoguang]

I guess the problem is that the uploaded file was empty already. There is a bug when uploading packages.

When a large package is uploaded, the pfci.Data is already EOF before contentStore.Save, so there will an empty file saved in the store.

@KN4CK3R

If I add a pfci.Data.(*packages_module.HashedBuffer).Seek(0, io.SeekStart) before if err := contentStore.Save(packages_module.BlobHash256Key(pb.HashSHA256), pfci.Data, pfci.Data.Size()); err != nil { in addFileToPackageVersion, then the problem is resolved.

[KN4CK3R]

That should not be EOF. My old test upload does not have a problem. I will have a look.

Edit: The error occurs only with larger files which are temporarly stored on the filesystem, so my FileBackedBuffer may be broken.

[KN4CK3R]

Fixed with #20622

[dorianim]

Thanks for the quick fix!
In which version will this be released?

[wxiaoguang]

Now it on 1.18-dev

There will be a backport for 1.17.1

[KN4CK3R]

Backport in #20635

[f403]

I fear, this is not solved, or there is another bug.
When uploading very big files (in my case 1.4G) they do not appear on disk, and downloading results in a 0B size.
Without any warnings during upload/download.

Gitea 1.17.2 (docker)
Steps to reproduce:

$ ll
total 1,4G
drwxrwxr-x  2 someuser someuser 4,0K Okt 11 14:47 .
drwxrwxrwt 41 root     root      12K Okt 11 14:50 ..
-rw-rw-r--  1 someuser someuser 1,4G Aug  9 18:48 ubuntu-22.04.1-live-server-amd64.iso

$ curl --user user:tockentockentockentockentockentockentock --upload-file ubuntu-22.04.1-live-server-amd64.iso https://gitea.host/api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso

$ wget -O downloaded.iso https://gitea.host/user/-/packages/generic/ubuntu-live-server/22.04.1/files/30
--2022-10-11 14:57:57--  https://gitea.host/user/-/packages/generic/ubuntu-live-server/22.04.1/files/30
Resolving gitea.host (gitea.host)... 198.51.100.1
Connecting to gitea.host (gitea.host)|198.51.100.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [application/octet-stream]
Saving to: ‘downloaded.iso’

downloaded.iso                                           [ <=>                                                                                                                  ]       0  --.-KB/s    in 0s      

2022-10-11 14:57:57 (0,00 B/s) - ‘downloaded.iso’ saved [0/0]

$ ll
total 1,4G
drwxrwxr-x  2 someuser someuser 4,0K Okt 11 14:57 .
drwxrwxrwt 41 root     root      12K Okt 11 14:57 ..
-rw-rw-r--  1 someuser someuser    0 Okt 11 14:57 downloaded.iso
-rw-rw-r--  1 someuser someuser 1,4G Aug  9 18:48 ubuntu-22.04.1-live-server-amd64.iso

And the logs at this time:

gitea     | 2022/10/11 12:53:36 [63456750] router: started   PUT /api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso for 172.19.0.3:55322      
gitea     | 2022/10/11 12:53:36 ...rvices/auth/basic.go:67:Verify() [T] [63456750] Basic Authorization: Attempting login for: user                                                                                   
gitea     | 2022/10/11 12:53:36 ...rvices/auth/basic.go:90:Verify() [T] [63456750] Basic Authorization: Valid AccessToken for user[0]                                                                              
gitea     | 2022/10/11 12:53:39 [633ef465-25] router: slow      PUT /api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso for 172.19.0.3:55322, elapsed 3888.4ms @ generic/gener
ic.go:67(generic.UploadPackage)                                                                                                                                                                                    
gitea     | 2022/10/11 12:53:50 ...packages/packages.go:112:createPackageAndVersion() [T] [63456750] Creating package: 3, 3, generic, ubuntu-live-server, 22.04.1, map[], map[], false                             
gitea     | 2022/10/11 12:53:51 ...packages/packages.go:226:addFileToPackageVersion() [T] [63456750] Adding package file: 24, ubuntu-22.04.1-live-server-amd64.iso
gitea     | 2022/10/11 12:53:52 [63456750] router: completed PUT /api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso for 172.19.0.3:55322, 201 Created in 16835.7ms @ generic/
generic.go:67(generic.UploadPackage)  

gitea     | 2022/10/11 12:57:58 [63456856] router: started   GET /user/-/packages/generic/ubuntu-live-server/22.04.1/files/30 for 172.19.0.3:37902
gitea     | 2022/10/11 12:57:58 [63456856] router: completed GET /user/-/packages/generic/ubuntu-live-server/22.04.1/files/30 for 172.19.0.3:37902, 0  in 43.2ms @ user/package.go:370(user.DownloadPackageFile)

And there no new files on disk:

$ touch --date "2022-10-10" /tmp/start
$ find /data/gitea/gitea/packages/ -type f -newer /tmp/start
$ 

[KN4CK3R]

Do you use a reverse proxy?

Tested it with a 1.5gb file and it worked without problems.

[f403]

Yes, but same happens when uploading without proxy.

$ curl --user user:tockentockentockentockentockentockentock --upload-file ubuntu-22.04.1-live-server-amd64.iso http://172.19.0.4:3000/api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso
$ wget -O downloaded.iso http://172.19.0.4:3000/api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso
--2022-10-11 14:03:51--  http://172.19.0.4:3000/api/packages/user/generic/ubuntu-live-server/22.04.1/ubuntu-22.04.1-live-server-amd64.iso
Connecting to 172.19.0.4:3000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [application/octet-stream]
Saving to: ‘downloaded.iso’

downloaded.iso                                           [ <=>                                                                                                                  ]       0  --.-KB/s    in 0s      

2022-10-11 14:03:51 (0.00 B/s) - ‘downloaded.iso’ saved [0/0]

$ ll
total 1.4G
drwxrwxr-x  2 someuser someuser 4.0K Oct 11 14:03 ./
drwxrwxrwt 15 root     root     4.0K Oct 11 13:53 ../
-rw-rw-r--  1 someuser someuser    0 Oct 11 14:03 downloaded.iso
-rw-rw-r--  1 someuser someuser 1.4G Aug  9 16:48 ubuntu-22.04.1-live-server-amd64.iso

[f403]

Hmm...
It worked in a fresh container on my laptop.

Can you, please, give me a hint, where to look for the uploaded files (temp path, disk path for a certain package file)?

[KN4CK3R]

When uploading files larger 32mb the content gets temporarly stored in %temp%/gitea-buffer-xxx with %temp% whatever is the temporary directory of the current user and xxx some random characters.

And the file is then stored in data/packages/xxx by default. This can be changed in the app.ini. xxx is a path depeding of the file sha256 hash.

[f403]

Thank you!

There was an empty file under data/packages/xxx created before updating to 1.17.2.
Deleting the file and restarting Gitea helped. (And running "Cleanup expired packages", not sure if it was needed)

(Not overwriting existing files is correct, otherwise there is a possibility for a hash-collision attack)