cabal upload throwing curl errors

[hasufell]

Describe the bug

+ cabal upload --publish --publish -d dist-docs.nENRAe/file-io-0.1.3-docs.tar.gz
hackage.haskell.org username: maerwald
hackage.haskell.org password:
Uploading documentation dist-docs.nENRAe/file-io-0.1.3-docs.tar.gz...
Error: cabal: '/usr/bin/curl' exited with an error:
curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding

Happens reproducibly.

To Reproduce

1. cabal v2-haddock --haddock-for-hackage --enable-doc --haddock-options=--quickjump
2. cabal upload --publish -d dist-newstyle/file-io-0.1.3-docs.tar.gz

Expected behavior

cabal upload should work or have actual error handling.

System information

• Operating system: Fedora 39
• cabal 3.12.1.0, ghc 9.8.2

[ulysses4ever]

Thanks for the report! Does it reproduce with 3.10?

[hasufell]

Thanks for the report! Does it reproduce with 3.10?

yes

[ulysses4ever]

Good. We should be catching more errors from auxiliary tools, for sure. But it's hard to do anytime meaningful if we don't understand the underlying issue.

Are you behind any kind of proxy? Cf. curl/curl#6760

If we find what curl command cabal tried to execute, we could experiment further. Perhaps, running cabal with -v (-v3?) would reveal the exact command.

If we know the command, running the same command with --trace log.txt as suggested in the ticket above could bring an insight...

[hasufell]

Are you behind any kind of proxy?

No.

[sgraf812]

I just hit the same error while trying to upload documentation for a candidate release of happy-lib here.

FWIW, for me the error was that I hadn't yet uploaded the regular sdist of the candidate package. This does not seem related; see below

[geekosaur]

Possibly relatedly, my Ubuntu system just received updates to both curl and wget. Since I don't do much uploading to Hackage (and in fact don't even have an account), I can't check to see if this introduced a new bug.

BTW, curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding is not an error we can do anything about (wtf does it even mean? That's a curl internal burp), so I don't see the issue: we can't do anything other than report it to the user and bail.

[sgraf812]

I think in my case it would have helped to say "Hint: This curl error can occur when uploading haddocks for a package before uploading the package."

[geekosaur]

TBH I think the technically correct thing here is to see if the server outputs error text before it runs its side of the transaction — but I'm not sure that's practical unless it is changed to always output text of some kind (which will of course break every older version of cabal-install). So this strikes me as a poorly-designed Hackage protocol which can't sensibly be dealt with except by the kind of post facto heuristics you suggested, which will always be dubious.

[sgraf812]

I triggered the error again this morning. See haskell/hackage-server#1070. I think it would be great to sort this out; I don't think I made an error while typing in my password because it came from the clipboard (from my password manager), and that same password worked while uploading the code of the package candidate.

[hasufell]

Yes, this is a major UX issue. Can we expedite this?

[ulysses4ever]

FWIW it just reproduced fully for me locally when trying to upload a candidate for one of my packages:

❯ cabal upload -v -d /home/artem/tmp/BNFC-meta/dist-newstyle/alex-meta-0.3.0.14-docs.
tar.gz
Warning: this is a debug build of cabal-install with assertions enabled.
hackage.haskell.org username: ArtemPelenitsyn
hackage.haskell.org password: 
Uploading documentation
/home/artem/tmp/BNFC-meta/dist-newstyle/alex-meta-0.3.0.14-docs.tar.gz...
Running: /run/current-system/sw/bin/curl --config - 'https://hackage.haskell.org/package/alex-meta-0.3.0.14/candidate/docs' --request PUT --data-binary '@/home/artem/tmp/BNFC-meta/dist-newstyle/alex-meta-0.3.0.14-docs.tar.gz' --write-out '
%{http_code}' --user-agent 'cabal-install/3.15.0.0 (linux; x86_64)' --silent --show-error --location --header 'Accept: text/plain' --header 'Content-Type: application/x-tar' --header 'Content-Encoding: gzip'
CallStack (from HasCallStack):
  withMetadata, called at src/Distribution/Simple/Utils.hs:447:12 in Cabal-3.15.0.0-inplace:Distribution.Simple.Utils
Error: cabal: '/run/current-system/sw/bin/curl' exited with an error:
curl: (56) chunk hex-length char not a hex digit: 0xd

(the curl error is phrased differently but the error code is the same, so I assume it just depends on the curl version)

[sgraf812]

When I googled this error earlier, it often appeared in conjunction with newlines in some request header. 0xd (\r) seems to support that corellation

[geekosaur]

It may also depend on what's in any error message hackage-server might be throwing. It would be interesting to see the output of that curl command with --verbose.

[ulysses4ever]

@geekosaur I fail to run the copy-pasted curl command because of the --config - bit, which says that it will get configuration from stdin afaiu: it just hangs and waits for input.

If I remove the --config - bit, it fails:

❯ /run/current-system/sw/bin/curl 'https://hackage.haskell.org/package/alex-meta-0.3.
0.14/candidate/docs' --request PUT --data-binary '@/home/artem/tmp/BNFC-meta/dist-new
style/alex-meta-0.3.0.14-docs.tar.gz' --write-out '
  %{http_code}' --user-agent 'cabal-install/3.15.0.0 (linux; x86_64)' --silent --show
-error --location --header 'Accept: text/plain' --header 'Content-Type: application/x
-tar' --header 'Content-Encoding: gzip'
Error: No authorization provided


curl: (56) chunk hex-length char not a hex digit: 0xd

401⏎                                                                                 

[ulysses4ever]

When I run cabal with -v3 it gives away the stdin it provides curl with:

--anyauth
--user ArtemPelenitsyn:<my password>

[ulysses4ever]

Running curl with -v doesn't help me understand the problem, sadly, here's the gist with the log.

[sgraf812]

Well, at least it seems like the server rejects the request as unauthorised, and curl reports multiple issues, one of which is the one that I got:

* Need to rewind upload for next request
* Ignoring the response-body
* chunk hex-length char not a hex digit: 0xd
* Illegal or missing hexadecimal sequence in chunked-encoding   <------- This is the one I got
* Closing connection
* TLSv1.2 (OUT), TLS alert, close notify (256):

It seems a bit weird that there is a newline in

--write-out '
  %{http_code}'

. can you try without? Edit: Oh, that may just be the width of your terminal emulator window...

[ulysses4ever]

@sgraf812

it seems like the server rejects the request as unauthorised

mm, which line says that?

weird that there is a newline

very true: my terminal is a little junky when it comes to breaking up lines so copy-pasting with it does not always work nicely.

I updated the gist after running without the spurious newline as you suggest.

[geekosaur]

@ulysses4ever, https://gist.github.com/ulysses4ever/602014e347df797f2f16a14edc0a9272#file-gistfile1-txt-L46. Note however that it's just doing an HTTP POST, so indeed it's not going to get that response until after the server receives it. The weird part there is that curl tries it without authentication first and then retries with auth (and there's where it tripped up).

[ulysses4ever]

@sgraf812 re: the newline in http code: it does come from how HttpTransport is implemented:

cabal/cabal-install/src/Distribution/Client/HttpUtils.hs

Line 553 in 5d5110a

, "\n%{http_code}"

of course, it's not clear to me why it's there. But at least it's not an artifact of copy-pasting...

[tusharad]

Getting the same issue. Any solutions?

[tusharad]

Meanwhile this command can be used:

curl -X PUT \
     --header "Authorization: X-ApiKey <API Token>" \
     -H "Content-Type: application/x-tar" \
     -H 'Content-Encoding: gzip' \
     --data-binary "@dist-newstyle/docs/XXXX-0.1.0.0-docs.tar.gz" \
     https://hackage.haskell.org/package/XXXX-0.1.0.0/docs

[edsko]

The workaround from @tusharad works also with --user Username:Password instead of using an API token.

[Mikolaj]

And the following mutation of the workaround works for package candidates (sometimes?):

curl -X PUT \
  --user MikolajKonarski:admin123 \
  -H "Content-Type: application/x-tar" \
  -H 'Content-Encoding: gzip' \
  --data-binary "@dist-newstyle/Cabal-3.14.2.0-docs.tar.gz" \
  https://hackage.haskell.org/package/Cabal-3.14.2.0/candidate/docs

You may need

curl -X PURGE https://hackage.haskell.org/package/Cabal-3.14.2.0/candidate

to see the docs without waiting forever.

[theGhostJW]

Same issue here @Mikolaj work around worked

[ulysses4ever]

We should really open a hackage-server issue. The one referenced above (haskell/hackage-server#1070) has nothing to do with what we see here (that one is about .tar vs. .tgz mishmash, which also was fixed recently).

[gbaz]

if its an issue with how the server parses newlines in headers, then likely the non-cabal fix will have to be not in hackage itself, but upstream in happstack.

[gbaz]

This isn't going to be fixable on hackage-server -- its about curl args that don't affect the actual communication with the server.

looks like the newline dates to here: 9591d95

The --write-out is to capture the resultant http code. Curl likely changed its behavior here. The issue the patch addressed was the http code didn't always have a newline. I suggest that we remove the newline before http_code (which new curl doesn't parse) and just fixup the processing in reading in the file with the resultant http status code.