lego in combination with pebble fails to obtain certificate

[svennis94]

Hi!

We are integrating the go-acme library into our application to allow users to generate certificates for their domains. For local development we use pebble in a docker-compose environment. Using pebble in combination with go-acme is posing challenges. I was hoping to maybe get some feedback, maybe we are doing it wrong 🙂

Some things that we are running into are in combination with pebble:

The first thing; Pebble by default returns a urn:ietf:params:acme:error:badNonce error for 5% of the requests. When using go-acme lib we noticed that we could see this error being logged in the log of our application, but the returned error of go-acme of the Obtain method did not return this error at all which is quite confusing. Instead it returned the following error:

ERROR [2025-02-06 10:05:19.999023] ACME Error: one or more domains had a problem: [test1.se] authorization[get]: empty URL

Subsequent requests towards pebble orderz endpoint just "hang", this might be a pebble thing or be related to the fact that the order was not retried (this still needs to be implemented on our side). Is there a way to get better error messages from the go-acme library that show the error that is returned from pebble? For now we reduced the badNonce error to 0% which we thought would give us a consistent way of generating test certificates under development. But this is when the second issue came to light.

The second issue is a bit more complex. We are getting again the same error from go-acme, but the challenge seemed to be valid. So go-acme is returning an error where the certificate order is actually valid and with a different library acmez this works but not with the go-acme lib. My suspicion is that the certificate was marked as VALID after the last authz request (see pebble logs). Therefore go-acme sees that the certificate order is still not done and returns the error as it did not have authorization urls in the response from pebble. But I might be wrong here. This is very consistent when you configure pebble with the following env variables:

      - PEBBLE_VA_NOSLEEP=1
      - PEBBLE_WFE_NONCEREJECT=0
      - PEBBLE_AUTHZREUSE=0

When you have the PEBBLE_AUTHZREUSE set to a different value, sometimes the certificate generation will succeed, but not always. If you set this variable to 100 the first 2 certificate orders will fail with the error in the application log (empty URL) and the subsequent requests succeed. I have tested this with pebble version 2.7.0 and 2.5.2 with the latest version of go-acme. The latest version of acmez works without any issues with the pebble configuration listed above. The HTTP01 challenge is succeeding as expected, the host test1.se is being pointed towards our local application in the host file of the pebble container.

PEBBLE:

pebble               | Pebble 2025/02/06 10:05:14 GET /dir -> calling handler()
pebble               | Pebble 2025/02/06 10:05:14 HEAD /nonce-plz -> calling handler()
pebble               | Pebble 2025/02/06 10:05:14 POST /sign-me-up -> calling handler()
pebble               | Pebble 2025/02/06 10:05:14 There are now 1 accounts in memory
pebble               | Pebble 2025/02/06 10:05:19 GET /dir -> calling handler()
pebble               | Pebble 2025/02/06 10:05:19 HEAD /nonce-plz -> calling handler()
pebble               | Pebble 2025/02/06 10:05:19 POST /order-plz -> calling handler()
pebble               | Pebble 2025/02/06 10:05:19 There are now 1 authorizations in the db
pebble               | Pebble 2025/02/06 10:05:19 Added order "C7naozp0NGPhowOW6-W4dYG9b8oLxyGHcr9yGYlW3Mw" to the db
pebble               | Pebble 2025/02/06 10:05:19 There are now 1 orders in the db
pebble               | Pebble 2025/02/06 10:05:19 POST /authZ/ -> calling handler()
pebble               | Pebble 2025/02/06 10:05:19 POST /chalZ/ -> calling handler()
pebble               | Pebble 2025/02/06 10:05:19 Pulled a task from the Tasks queue: &va.vaTask{Identifier:acme.Identifier{Type:"dns", Value:"test1.se"}, Challenge:(*core.Challenge)(0xc0001d20a0), Account:(*core.Account)(0xc000158300), AccountURL:"https://pebble:14000/my-account/5228848032d2101d", Wildcard:false}
pebble               | Pebble 2025/02/06 10:05:19 Starting 3 validations.
pebble               | Pebble 2025/02/06 10:05:19 Attempting to validate w/ HTTP: http://test1.se:6081/.well-known/acme-challenge/Xqi1jfMRVXgxJzlQqS80c82Z30n9J5OgPk9Dg74dhk0
pebble               | Pebble 2025/02/06 10:05:19 Attempting to validate w/ HTTP: http://test1.se:6081/.well-known/acme-challenge/Xqi1jfMRVXgxJzlQqS80c82Z30n9J5OgPk9Dg74dhk0
pebble               | Pebble 2025/02/06 10:05:19 Attempting to validate w/ HTTP: http://test1.se:6081/.well-known/acme-challenge/Xqi1jfMRVXgxJzlQqS80c82Z30n9J5OgPk9Dg74dhk0
pebble               | Pebble 2025/02/06 10:05:19 POST /authZ/ -> calling handler()
pebble               | Pebble 2025/02/06 10:05:19 authz fM8nC4kjROdvougXLIFRWZuNKIISZ1C49u7OmmiTqOo set VALID by completed challenge dwWlV8fLeiG2o8kKSjSGWnyuiD-WFQUmHERp3_43ReQ

APPLICATION:

ERROR [2025-02-06 10:05:19.999023] ACME Error: one or more domains had a problem: [test1.se] authorization[get]: empty URL

PEBBLE REQUEST LOG for the last authZ request:

DEBUG [2025-02-06 10:05:19.998974] ACME HTTP: 200 map[string]string{"Cache-Control":"public, max-age=0, no-cache", "Content-Length":"431", "Content-Type":"application/json; charset=utf-8", "Date":"Thu, 06 Feb 2025 10:05:19 GMT", "Link":"<https://pebble:14000/dir>;rel=\"index\"", "Replay-Nonce":"C_j0xEbJcl_AbKnUgtdkUw"}
{
    "status": "valid",
    "identifier": {
       "type": "dns",
       "value": "test1.se"
    },
    "challenges": [
       {
          "type": "http-01",
          "url": "https://pebble:14000/chalZ/dwWlV8fLeiG2o8kKSjSGWnyuiD-WFQUmHERp3_43ReQ",
          "token": "Xqi1jfMRVXgxJzlQqS80c82Z30n9J5OgPk9Dg74dhk0",
          "status": "valid",
          "validated": "2025-02-06T10:05:19Z"
       }
    ],
    "expires": "2025-02-06T11:05:19Z"
 }

Has anyone experienced the same with pebble and maybe can help us resolve this issue with the go-acme library?

[ldez]

Hello,

the project is lego, not go-acme (this is the name of our organization).

FYI, all our integration tests are based on pebble: https://github.com/go-acme/lego/tree/master/e2e

[ldez]

lego retries automatically on urn:ietf:params:acme:error:badNonce (this is why there are logs), I don't understand how you get an error at the end.

Our tests use a 20% reject (PEBBLE_WFE_NONCEREJECT=20), and it works as expected, with no flaky behavior.

[ldez]

I run our tests with:

- PEBBLE_VA_NOSLEEP=1
- PEBBLE_WFE_NONCEREJECT=0
- PEBBLE_AUTHZREUSE=100

and I have 0 errors.

I need more information about your current usage of lego.

[svennis94]

Ah sorry about the naming!

I'll have a look at the e2e tests and see how it differs to our env. We use a custom HTTP client to send HTTP requests over our messages bus to a node that has access to outgoing internet, so it might be that it is related to that as the rest seems to be the same. I'll do some more tests and come back with the results.

Pebble version: v2.7.0
lego version: v4.21.0

Thanks for the swift response.

[svennis94]

@ldez I did some further testing, with the default http.Client from lego it works without any issues. There were issues when I re-introduced our own http.Client implementation. There was a transformation happening on the http.Response.Header from a type http.Header to a type map[string]string, this was then transformed back. Removing this transformation resolved all issues with lego.

Thanks for the help! The e2e dir is very useful.

[ldez]

I hope you enjoyed my work, please consider donating or asking your company to do so.
This will be appreciated, thank you ❤️

https://donate.ldez.dev