Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

reverseproxy dynamic upstream race #5736

Closed
kkroo opened this issue Aug 12, 2023 · 12 comments · Fixed by #5752
Closed

reverseproxy dynamic upstream race #5736

kkroo opened this issue Aug 12, 2023 · 12 comments · Fixed by #5752
Labels
bug 🐞 Something isn't working needs info 📭 Requires more information
Milestone
v2.7.4

Comments

@kkroo
Copy link
Contributor

kkroo commented Aug 12, 2023 •
edited
Loading

Running off of master (d813550)
I am running into write-write race conditions with:

| WARNING: DATA RACE | Write at 0x00c00015d768 by goroutine 1181: | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream() | /root/go/src/github.com/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1072 +0x79 | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration() | /root/go/src/github.com/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21b7 | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP() | /root/go/src/github.com/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac | github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1() | /root/go/src/github.com/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8 | github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

| WARNING: DATA RACE | Write at 0x00c000798820 by goroutine 852: | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).fillHost() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:133 +0x15c | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1068 +0x56 | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7 | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac | github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8 | github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e
@mholt mholt added the bug 🐞 Something isn't working label Aug 12, 2023
@mholt mholt added this to the v2.7.4 milestone Aug 12, 2023
@kkroo
Copy link
Contributor Author

kkroo commented Aug 12, 2023 •
edited
Loading

| WARNING: DATA RACE | Read at 0x00c0005abe60 by goroutine 654: | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Healthy() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:83 +0x6e | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Available() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:75 +0x44 | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.selectRandomHost() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:750 +0xe4 | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.RandomSelection.Select() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:66 +0x66 | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*RandomSelection).Select() | <autogenerated>:1 +0x106 | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:478 +0x5c1 | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac | github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8 | github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

| Previous write at 0x00c0005abe60 by goroutine 655: | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1092 +0x2fb | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21b7 | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac | github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8 | github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

@kkroo
Copy link
Contributor Author

kkroo commented Aug 12, 2023 •
edited
Loading

| Previous read at 0x00c0005abe68 by goroutine 659: | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).reverseProxy() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:809 +0x15f1 | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:530 +0x173d | github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac | github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8 | github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP() | /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

@mohammed90
Copy link
Member

Did you customize Caddy in any way? The referenced line 1072 in github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1072 is an empty line. Please share the information listed below.

## 1. Environment

### 1a. Operating system and version

```
paste here
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```


### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```


## 2. Reproducer

### 2.1 Caddy configuration (Caddyfile or JSON)

```
paste here
```

@kkroo
Copy link
Contributor Author

kkroo commented Aug 13, 2023 •
edited
Loading

oh my bad
yes i tried to fix this race condition, and it just exposed another one.
the point is these methods are not threadsafe
its part of a larger config, but the important part is its a simple reverse proxy with a AUpstreams with an A record
i am in the airport can post it later

diff --git a/modules/caddyhttp/reverseproxy/healthchecks.go b/modules/caddyhttp/reverseproxy/healthchecks.go
index 80b635ad..a40d1d9d 100644
--- a/modules/caddyhttp/reverseproxy/healthchecks.go
+++ b/modules/caddyhttp/reverseproxy/healthchecks.go
@@ -455,8 +455,11 @@ func (h *Handler) countFailure(upstream *Upstream) {
 		return
 	}
 
+	// load host
+	host := upstream.Host()
+
 	// count failure immediately
-	err := upstream.Host.countFail(1)
+	err := host.countFail(1)
 	if err != nil {
 		h.HealthChecks.Passive.logger.Error("could not count failure",
 			zap.String("host", upstream.Dial),
@@ -487,5 +490,5 @@ func (h *Handler) countFailure(upstream *Upstream) {
 				zap.String("host", upstream.Dial),
 				zap.Error(err))
 		}
-	}(upstream.Host, failDuration)
+	}(host, failDuration)
 }
diff --git a/modules/caddyhttp/reverseproxy/hosts.go b/modules/caddyhttp/reverseproxy/hosts.go
index a23715ec..62a7475e 100644
--- a/modules/caddyhttp/reverseproxy/hosts.go
+++ b/modules/caddyhttp/reverseproxy/hosts.go
@@ -33,8 +33,6 @@ type UpstreamPool []*Upstream
 // state of the backend host it is correlated with.
 // Upstream values must not be copied.
 type Upstream struct {
-	*Host `json:"-"`
-
 	// The [network address](/docs/conventions#network-addresses)
 	// to dial to connect to the upstream. Must represent precisely
 	// one socket (i.e. no port ranges). A valid network address
@@ -83,7 +81,7 @@ func (u *Upstream) Available() bool {
 func (u *Upstream) Healthy() bool {
 	healthy := u.healthy()
 	if healthy && u.healthCheckPolicy != nil {
-		healthy = u.Host.Fails() < u.healthCheckPolicy.MaxFails
+		healthy = u.Host().Fails() < u.healthCheckPolicy.MaxFails
 	}
 	if healthy && u.cb != nil {
 		healthy = u.cb.OK()
@@ -94,7 +92,7 @@ func (u *Upstream) Healthy() bool {
 // Full returns true if the remote host
 // cannot receive more requests at this time.
 func (u *Upstream) Full() bool {
-	return u.MaxRequests > 0 && u.Host.NumRequests() >= u.MaxRequests
+	return u.MaxRequests > 0 && u.Host().NumRequests() >= u.MaxRequests
 }
 
 // fillDialInfo returns a filled DialInfo for upstream u, using the request
@@ -124,13 +122,9 @@ func (u *Upstream) fillDialInfo(r *http.Request) (DialInfo, error) {
 	}, nil
 }
 
-func (u *Upstream) fillHost() {
-	host := new(Host)
-	existingHost, loaded := hosts.LoadOrStore(u.String(), host)
-	if loaded {
-		host = existingHost
-	}
-	u.Host = host
+func (u *Upstream) Host() *Host {
+	host, _ := hosts.LoadOrStore(u.String(), new(Host))
+	return host
 }
 
 // Host is the basic, in-memory representation of the state of a remote host.
diff --git a/modules/caddyhttp/reverseproxy/reverseproxy.go b/modules/caddyhttp/reverseproxy/reverseproxy.go
index b65dce72..03274009 100644
--- a/modules/caddyhttp/reverseproxy/reverseproxy.go
+++ b/modules/caddyhttp/reverseproxy/reverseproxy.go
@@ -455,14 +455,11 @@ func (h *Handler) proxyLoopIteration(r *http.Request, origReq *http.Request, w h
 	// get the updated list of upstreams
 	upstreams := h.Upstreams
 	if h.DynamicUpstreams != nil {
-		dUpstreams, err := h.DynamicUpstreams.GetUpstreams(r)
+		dUpstreams, err := h.DynamicUpstreams.GetUpstreams(r, h.provisionUpstream)
 		if err != nil {
 			h.logger.Error("failed getting dynamic upstreams; falling back to static upstreams", zap.Error(err))
 		} else {
 			upstreams = dUpstreams
-			for _, dUp := range dUpstreams {
-				h.provisionUpstream(dUp)
-			}
 			h.logger.Debug("provisioned dynamic upstreams", zap.Int("count", len(dUpstreams)))
 			defer func() {
 				// these upstreams are dynamic, so they are only used for this iteration
@@ -503,14 +500,17 @@ func (h *Handler) proxyLoopIteration(r *http.Request, origReq *http.Request, w h
 	// or satisfactorily represented in a URL
 	caddyhttp.SetVar(r.Context(), dialInfoVarKey, dialInfo)
 
+	// host info
+	host := upstream.Host()
+
 	// set placeholders with information about this upstream
 	repl.Set("http.reverse_proxy.upstream.address", dialInfo.String())
 	repl.Set("http.reverse_proxy.upstream.hostport", dialInfo.Address)
 	repl.Set("http.reverse_proxy.upstream.host", dialInfo.Host)
 	repl.Set("http.reverse_proxy.upstream.port", dialInfo.Port)
-	repl.Set("http.reverse_proxy.upstream.requests", upstream.Host.NumRequests())
+	repl.Set("http.reverse_proxy.upstream.requests", host.NumRequests())
 	repl.Set("http.reverse_proxy.upstream.max_requests", upstream.MaxRequests)
-	repl.Set("http.reverse_proxy.upstream.fails", upstream.Host.Fails())
+	repl.Set("http.reverse_proxy.upstream.fails", host.Fails())
 
 	// mutate request headers according to this upstream;
 	// because we're in a retry loop, we have to copy
@@ -736,9 +736,10 @@ func (h Handler) addForwardedHeaders(req *http.Request) error {
 // (This method is mostly the beginning of what was borrowed from the net/http/httputil package in the
 // Go standard library which was used as the foundation.)
 func (h *Handler) reverseProxy(rw http.ResponseWriter, req *http.Request, origReq *http.Request, repl *caddy.Replacer, di DialInfo, next caddyhttp.Handler) error {
-	_ = di.Upstream.Host.countRequest(1)
+	host := di.Upstream.Host()
+	_ = host.countRequest(1)
 	//nolint:errcheck
-	defer di.Upstream.Host.countRequest(-1)
+	defer host.countRequest(-1)
 
 	// point the request to this upstream
 	h.directRequest(req, di)
@@ -1064,9 +1065,6 @@ func (Handler) directRequest(req *http.Request, di DialInfo) {
 }
 
 func (h Handler) provisionUpstream(upstream *Upstream) {
-	// create or get the host representation for this upstream
-	upstream.fillHost()
-
 	// give it the circuit breaker, if any
 	upstream.cb = h.CB
 
@@ -1300,10 +1298,11 @@ type Selector interface {
 // across successive calls. If the list of upstreams changes or the
 // ordering is not stable, load balancing will suffer. This function
 // may be called during each retry, multiple times per request, and as
-// such, needs to be instantaneous. The returned slice will not be
+// such, needs to be instantaneous. The second argument is a provisioning
+// function to be called with new instances. The returned slice will not be
 // modified.
 type UpstreamSource interface {
-	GetUpstreams(*http.Request) ([]*Upstream, error)
+	GetUpstreams(*http.Request, func(*Upstream)) ([]*Upstream, error)
 }
 
 // Hop-by-hop headers. These are removed when sent to the backend.
diff --git a/modules/caddyhttp/reverseproxy/selectionpolicies.go b/modules/caddyhttp/reverseproxy/selectionpolicies.go
index bc6de351..5f492e85 100644
--- a/modules/caddyhttp/reverseproxy/selectionpolicies.go
+++ b/modules/caddyhttp/reverseproxy/selectionpolicies.go
@@ -255,10 +255,11 @@ func (LeastConnSelection) Select(pool UpstreamPool, _ *http.Request, _ http.Resp
 	var count int
 	leastReqs := -1
 
-	for _, host := range pool {
-		if !host.Available() {
+	for _, upstream := range pool {
+		if !upstream.Available() {
 			continue
 		}
+		host := upstream.Host()
 		numReqs := host.NumRequests()
 		if leastReqs == -1 || numReqs < leastReqs {
 			leastReqs = numReqs
@@ -270,7 +271,7 @@ func (LeastConnSelection) Select(pool UpstreamPool, _ *http.Request, _ http.Resp
 		if numReqs == leastReqs {
 			count++
 			if count > 1 || (weakrand.Int()%count) == 0 { //nolint:gosec
-				bestHost = host
+				bestHost = upstream
 			}
 		}
 	}
@@ -775,7 +776,8 @@ func leastRequests(upstreams []*Upstream) *Upstream {
 		if upstream == nil {
 			continue
 		}
-		reqs := upstream.NumRequests()
+		host := upstream.Host()
+		reqs := host.NumRequests()
 		if reqs == 0 {
 			return upstream
 		}
diff --git a/modules/caddyhttp/reverseproxy/upstreams.go b/modules/caddyhttp/reverseproxy/upstreams.go
index 19e4f3cb..442484fc 100644
--- a/modules/caddyhttp/reverseproxy/upstreams.go
+++ b/modules/caddyhttp/reverseproxy/upstreams.go
@@ -105,7 +105,7 @@ func (su *SRVUpstreams) Provision(ctx caddy.Context) error {
 	return nil
 }
 
-func (su SRVUpstreams) GetUpstreams(r *http.Request) ([]*Upstream, error) {
+func (su SRVUpstreams) GetUpstreams(r *http.Request, provision func(*Upstream)) ([]*Upstream, error) {
 	suAddr, service, proto, name := su.expandedAddr(r)
 
 	// first, use a cheap read-lock to return a cached result quickly
@@ -153,6 +153,7 @@ func (su SRVUpstreams) GetUpstreams(r *http.Request) ([]*Upstream, error) {
 			zap.Uint16("weight", rec.Weight))
 		addr := net.JoinHostPort(rec.Target, strconv.Itoa(int(rec.Port)))
 		upstreams[i] = &Upstream{Dial: addr}
+		provision(upstreams[i])
 	}
 
 	// before adding a new one to the cache (as opposed to replacing stale one), make room if cache is full
@@ -293,7 +294,7 @@ func (au *AUpstreams) Provision(_ caddy.Context) error {
 	return nil
 }
 
-func (au AUpstreams) GetUpstreams(r *http.Request) ([]*Upstream, error) {
+func (au AUpstreams) GetUpstreams(r *http.Request, provision func(*Upstream)) ([]*Upstream, error) {
 	repl := r.Context().Value(caddy.ReplacerCtxKey).(*caddy.Replacer)
 
 	resolveIpv4 := au.Versions.IPv4 == nil || *au.Versions.IPv4
@@ -352,6 +353,7 @@ func (au AUpstreams) GetUpstreams(r *http.Request) ([]*Upstream, error) {
 		upstreams[i] = &Upstream{
 			Dial: net.JoinHostPort(ip.String(), port),
 		}
+		provision(upstreams[i])
 	}
 
 	// before adding a new one to the cache (as opposed to replacing stale one), make room if cache is full
@@ -429,7 +431,7 @@ func (mu *MultiUpstreams) Provision(ctx caddy.Context) error {
 	return nil
 }
 
-func (mu MultiUpstreams) GetUpstreams(r *http.Request) ([]*Upstream, error) {
+func (mu MultiUpstreams) GetUpstreams(r *http.Request, provision func(*Upstream)) ([]*Upstream, error) {
 	var upstreams []*Upstream
 
 	for i, src := range mu.sources {
@@ -439,7 +441,7 @@ func (mu MultiUpstreams) GetUpstreams(r *http.Request) ([]*Upstream, error) {
 		default:
 		}
 
-		up, err := src.GetUpstreams(r)
+		up, err := src.GetUpstreams(r, provision)
 		if err != nil {
 			mu.logger.Error("upstream source returned error",
 				zap.Int("source_idx", i),

@mohammed90 mohammed90 added the needs info 📭 Requires more information label Aug 13, 2023
@mholt
Copy link
Member

mholt commented Aug 13, 2023

@kkroo Can you provide a stack trace using the original code? So we can be sure we get a fix properly.

@kkroo
Copy link
Contributor Author

kkroo commented Aug 13, 2023

stack traces with the original code:

  | ==================
  | WARNING: DATA RACE
  | Read at 0x00c000c64ff0 by goroutine 1022:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Healthy()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:86 +0x9d
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Available()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:77 +0x44
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.selectRandomHost()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:749 +0xe4
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.RandomSelection.Select()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:66 +0x66
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*RandomSelection).Select()
  |       <autogenerated>:1 +0x106
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:478 +0x5c1
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e
  |
  | Previous write at 0x00c000c64ff0 by goroutine 1019:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).fillHost()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:133 +0x15c
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1068 +0x56
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e


  | ==================
  | WARNING: DATA RACE
  | Write at 0x00c000c64ff0 by goroutine 486:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).fillHost()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:133 +0x15c
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1068 +0x56
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e
  |
  |
  | Previous read at 0x00c000c64ff0 by goroutine 483:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).reverseProxy()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:739 +0xa5
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:527 +0x1746
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e


  | ==================
  | WARNING: DATA RACE
  | Write at 0x00c000c64ff0 by goroutine 482:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).fillHost()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:133 +0x15c
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1068 +0x56
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e
  |
  |
  | Previous read at 0x00c000c64ff0 by goroutine 483:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:511 +0x113b
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e


  | ==================
  | WARNING: DATA RACE
  | Write at 0x00c000c65018 by goroutine 467:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1091 +0x326
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e
  |
  |
  | Previous read at 0x00c000c65018 by goroutine 463:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Healthy()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:85 +0x6e
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Available()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:77 +0x44
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.selectRandomHost()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:749 +0xe4
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.RandomSelection.Select()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:66 +0x66
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*RandomSelection).Select()
  |       <autogenerated>:1 +0x106
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:478 +0x5c1
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e



  | ==================
  | WARNING: DATA RACE
  | Write at 0x00c000c65020 by goroutine 467:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1071 +0x8b
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e
  |
  |
  | Previous read at 0x00c000c65020 by goroutine 463:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Healthy()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:88 +0x12b
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Available()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:77 +0x44
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.selectRandomHost()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:749 +0xe4
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.RandomSelection.Select()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:66 +0x66
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*RandomSelection).Select()
  |       <autogenerated>:1 +0x106
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:478 +0x5c1
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e


  | ==================
  | WARNING: DATA RACE
  | Write at 0x00c0000670f8 by goroutine 425:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1080 +0x19b
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e
  |
  |
  | Previous write at 0x00c0000670f8 by goroutine 422:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1080 +0x19b
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e



  | ==================
  | WARNING: DATA RACE
  | Write at 0x00c000c64ff0 by goroutine 425:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).fillHost()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:133 +0x15c
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1068 +0x56
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e
  |
  |
  | Previous read at 0x00c000c64ff0 by goroutine 422:
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).reverseProxy()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:741 +0xf5
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:527 +0x1746
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
  |   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
  |       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

@mholt
Copy link
Member

mholt commented Aug 14, 2023 •
edited
Loading

@kkroo Thanks, that makes more sense. What is your config?

The stack trace (and the issue title, d'oh) indicates that dynamic upstreams are being used. Since that is the case I need to know your config so I can hunt down the solution to the race.

Please post the exact config without changes/redactions so I can accurately create a fix. Thank you!

@kkroo
Copy link
Contributor Author

kkroo commented Aug 14, 2023

Thank you for looking into this @mholt
The config is really large since i am generating them using a custom config provider. here is the reverse proxy handler snippet unchanged:

                {
                  "dynamic_upstreams": {
                    "name": "service.upstream.dev",
                    "port": "80",
                    "source": "a",
                    "versions": {
                      "ipv4": true,
                      "ipv6": true
                    }
                  },
                  "handler": "reverse_proxy",
                  "headers": {
                    "request": {
                      "set": {
                        "Host": [
                          "service.upstream.dev:80"
                        ],
                        "X-Forwarded-For": [
                          "{http.request.remote}"
                        ],
                        "X-Forwarded-Host": [
                          "{http.request.hostport}"
                        ],
                        "X-Forwarded-Proto": [
                          "http"
                        ],
                        "X-Real-Ip": [
                          "{http.request.remote}"
                        ]
                      }
                    }
                  },
                  "health_checks": {
                    "passive": {
                      "fail_duration": 30000000000,
                      "max_fails": 5,
                      "unhealthy_status": [
                        501,
                        503
                      ]
                    }
                  },
                  "load_balancing": {
                    "retries": 5,
                    "try_duration": 30000000000
                  },
                  "transport": {
                    "keep_alive": {
                      "idle_timeout": 15000000000,
                      "max_idle_conns_per_host": 1000,
                      "probe_interval": 1000000000
                    },
                    "protocol": "http",
                    "read_timeout": 30000000000
                  },
                  "trusted_proxies": [
                    "::1/128",
                    "127.0.0.1/32",
                    "10.0.0.0/8",
                    "172.16.0.0/12",
                    "192.168.0.0/16"
                  ]
                }

@mholt
Copy link
Member

mholt commented Aug 14, 2023

Thanks. That helps too.

We're actually creating new *Upstream values for every call to GetUpstreams, which is correct. It can't be a race then because different requests see different *Upstream values.

Except if they're cached.

Do you have a moment to test if the race still occurs if you comment out these lines:

caddy/modules/caddyhttp/reverseproxy/upstreams.go

Lines 323 to 329 in d6f86cc

// first, use a cheap read-lock to return a cached result quickly
aAaaaMu.RLock()
cached := aAaaa[auStr]
aAaaaMu.RUnlock()
if cached.isFresh() {
return cached.upstreams, nil
}

i.e. if we don't use a cached result from another goroutine, maybe the race goes away.

And if so, then I think a better solution is to use Upstream instead of *Upstream, if possible.

@kkroo
Copy link
Contributor Author

kkroo commented Aug 14, 2023

I applied the following diff:

diff --git a/modules/caddyhttp/reverseproxy/upstreams.go b/modules/caddyhttp/reverseproxy/upstreams.go
index 19e4f3cb..4c9f5c1b 100644
--- a/modules/caddyhttp/reverseproxy/upstreams.go
+++ b/modules/caddyhttp/reverseproxy/upstreams.go
@@ -320,12 +320,12 @@ func (au AUpstreams) GetUpstreams(r *http.Request) ([]*Upstream, error) {
 	auStr := repl.ReplaceAll(au.String()+ipVersion, "")
 
 	// first, use a cheap read-lock to return a cached result quickly
-	aAaaaMu.RLock()
-	cached := aAaaa[auStr]
-	aAaaaMu.RUnlock()
-	if cached.isFresh() {
-		return cached.upstreams, nil
-	}
+	//aAaaaMu.RLock()
+	//cached := aAaaa[auStr]
+	//aAaaaMu.RUnlock()
+	//if cached.isFresh() {
+	//	return cached.upstreams, nil
+	//}
 
 	// otherwise, obtain a write-lock to update the cached value
 	aAaaaMu.Lock()
@@ -334,10 +334,10 @@ func (au AUpstreams) GetUpstreams(r *http.Request) ([]*Upstream, error) {
 	// check to see if it's still stale, since we're now in a different
 	// lock from when we first checked freshness; another goroutine might
 	// have refreshed it in the meantime before we re-obtained our lock
-	cached = aAaaa[auStr]
-	if cached.isFresh() {
-		return cached.upstreams, nil
-	}
+	//cached = aAaaa[auStr]
+	//if cached.isFresh() {
+	//	return cached.upstreams, nil
+	//}
 
 	name := repl.ReplaceAll(au.Name, "")
 	port := repl.ReplaceAll(au.Port, "")
@@ -355,12 +355,12 @@ func (au AUpstreams) GetUpstreams(r *http.Request) ([]*Upstream, error) {
 	}
 
 	// before adding a new one to the cache (as opposed to replacing stale one), make room if cache is full
-	if cached.freshness.IsZero() && len(srvs) >= 100 {
-		for randomKey := range aAaaa {
-			delete(aAaaa, randomKey)
-			break
-		}
-	}
+	//if cached.freshness.IsZero() && len(srvs) >= 100 {
+	//	for randomKey := range aAaaa {
+	//		delete(aAaaa, randomKey)
+	//		break
+	//	}
+	//}
 
 	aAaaa[auStr] = aLookup{
 		aUpstreams: au,

These are the races I am still getting:

| ==================
| WARNING: DATA RACE
| Write at 0x00c0009f8690 by goroutine 436:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).fillHost()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:133 +0x15c
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1068 +0x56
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

|
|
| Previous read at 0x00c0009f8690 by goroutine 427:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).reverseProxy()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:741 +0xf5
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:527 +0x1746
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

|
| ==================
| WARNING: DATA RACE
| Write at 0x00c0009f86c0 by goroutine 436:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1071 +0x8b
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

|
|
| Previous write at 0x00c0009f86c0 by goroutine 427:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1071 +0x8b
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

| ==================
| ==================
| WARNING: DATA RACE
| Write at 0x00c000067db8 by goroutine 436:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1080 +0x19b
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

|
|
| Previous write at 0x00c000067db8 by goroutine 427:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1080 +0x19b
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e
| ==================
| ==================
| WARNING: DATA RACE
| Write at 0x00c0009f86b8 by goroutine 436:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1091 +0x326
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

|
|
| Previous write at 0x00c0009f86b8 by goroutine 427:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1091 +0x326
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

| ==================
| WARNING: DATA RACE
| Write at 0x00c0009f86c0 by goroutine 438:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1071 +0x8b
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

|
|
| Previous read at 0x00c0009f86c0 by goroutine 436:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Healthy()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:88 +0x12b
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Available()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:77 +0x44
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.selectRandomHost()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:749 +0xe4
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.RandomSelection.Select()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:66 +0x66
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*RandomSelection).Select()
|       <autogenerated>:1 +0x106
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:478 +0x5c1
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

| ==================
| ==================
| WARNING: DATA RACE
| Write at 0x00c0009f86b8 by goroutine 438:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1091 +0x326
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

|
|
| Previous read at 0x00c0009f86b8 by goroutine 436:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Healthy()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:85 +0x6e
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).Available()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:77 +0x44
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.selectRandomHost()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:749 +0xe4
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.RandomSelection.Select()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/selectionpolicies.go:66 +0x66
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*RandomSelection).Select()
|       <autogenerated>:1 +0x106
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:478 +0x5c1
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

| ==================
| ==================
| WARNING: DATA RACE
| Write at 0x00c0009f8690 by goroutine 456:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).fillHost()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:133 +0x15c
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1068 +0x56
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

|
|
| Previous read at 0x00c0009f8690 by goroutine 443:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:511 +0x113b
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

| ==================
| ==================
| WARNING: DATA RACE
| Write at 0x00c0009f8690 by goroutine 668:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).fillHost()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:133 +0x15c
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1068 +0x56
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

|
|
| Previous write at 0x00c0009f8690 by goroutine 665:
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Upstream).fillHost()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/hosts.go:133 +0x15c
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.provisionUpstream()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:1068 +0x56
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:464 +0x21d7
|   github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/reverseproxy.go:434 +0x9ac
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/routes.go:331 +0xe8
|   github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP()
|       /root/go/src/github.com/caddyserver/caddy/v2/modules/caddyhttp/caddyhttp.go:58 +0x7e

@kkroo
Copy link
Contributor Author

kkroo commented Aug 15, 2023

i take that back. was building from a vendored path so these changes didn't get picked up. i updated sources with the patch and that does seem to fix the race

@mholt
Copy link
Member

mholt commented Aug 15, 2023

Phew 🙏

After I saw your update last night, I was struggling to figure out how it could possibly still be racey. I'm really glad it actually does fix it 😅

Ok, so we know a solution; I will try to get it fixed with minimal intrusion ASAP. Thank you!

mholt added a commit that referenced this issue Aug 16, 2023
@mholt
reverseproxy: Always return new upstreams (fix #5736)
be299bb
@mholt mholt mentioned this issue Aug 16, 2023
Merged
mholt added a commit that referenced this issue Aug 17, 2023
@mholt
reverseproxy: Always return new upstreams (fix #5736) (#5752)
936ee91 
* reverseproxy: Always return new upstreams (fix #5736)

* Fix healthcheck logger race
tianze0926 added a commit to tianze0926/caddy that referenced this issue Dec 20, 2023
@tianze0926 @mholt
@faddat @shyim @AaronDewes @francislavoie @pistasjis @testwill @mohammed90 @singhalkarun @marten-seemann @WeidiDeng @pauljeannot @hainenber @evandam @FossoresLP @glowinthedark @dependabot @dunglas @pkoenig10 @ThanmayNath @Christoph-D @mcfedr @bwesterb @Forza-tng @BattleRattle @perhapsmaple @ddl-ebrown @maraino @dlorenc @ankon @bmarwell @armadi1809 @jum @ddemoss222 @tgeoghegan
update (#1)
b750efd 
* templates: Fix httpInclude (fix caddyserver#5698)

Allowable during feature freeze because this is a simple, non-invasive
bug fix only.

* ci: Use gofumpt to format code (caddyserver#5707)

* go.mod: Upgrade golang.org/x/net to 0.14.0 (caddyserver#5718)

* ci: Add riscv64 (64-bit RISC-V) to goreleaser (caddyserver#5720)

This will add 64-bit RISC-V Linux prebuilts for Caddy.

* ci: Update to Go 1.21 (caddyserver#5719)

* ci: Update to Go 1.21

* Bump quic-go to v0.37.4

* Check EnableFullDuplex err

* Linter bug suppression

See timakin/bodyclose#52

---------

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* fileserver: Don't repeat error for invalid method inside error context (caddyserver#5705)

* caddytls: Update docs for on-demand config

* Fix tests

I thought Go ordered JSON objects when marshaling, but I guess not.

* cmd: Require config for caddy validate (fix caddyserver#5612) (caddyserver#5614)

* Require config for caddy validate - fixes caddyserver#5612

Signed-off-by: Pistasj <hi@pistasjis.net>

* Try making adjacent Caddyfile check its own function

Signed-off-by: Pistasj <hi@pistasjis.net>

* add Francis' suggestion

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* Refactor

* Fix borked commit, sigh

---------

Signed-off-by: Pistasj <hi@pistasjis.net>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>

* fileserver: Slightly more fitting icons

* ci: use gci linter (caddyserver#5708)

* use gofmput to format code

* use gci to format imports

* reconfigure gci

* linter autofixes

* rearrange imports a little

* export GOOS=windows golangci-lint run ./... --fix

* reverseproxy: Always return new upstreams (fix caddyserver#5736) (caddyserver#5752)

* reverseproxy: Always return new upstreams (fix caddyserver#5736)

* Fix healthcheck logger race

* go.mod: Upgrade CertMagic and quic-go

* fix package typo (caddyserver#5764)

Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>

* fileserver: docs: clarify the ability to produce JSON array with `browse` (caddyserver#5751)

* caddyfile: Loosen heredoc parsing (caddyserver#5761)

* httpcaddyfile: Stricter errors for site and upstream address schemes (caddyserver#5757)

Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* update quic-go to v0.37.6 (caddyserver#5767)

* caddyfile: Adjust error formatting (caddyserver#5765)

* replacer: change timezone to UTC for "time.now.http" placeholders (caddyserver#5774)

* chore: Appease gosec linter (caddyserver#5777)

These happen to be harmless memory aliasing
but I guess the linter can't know that and we
can't really prove it in general.

* go.mod: Update quic-go to v0.38.0 (caddyserver#5772)

* go.mod: Update quic-go to v0.38.0

* run "go mod tidy"

---------

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* caddyfile: Fix case where heredoc marker is empty after newline (caddyserver#5769)

Fixes `panic: runtime error: slice bounds out of range [:3] with capacity 2`

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* ci: ensure short-sha is exported correctly on all platforms (caddyserver#5781)

* fileserver: Export BrowseTemplate

This allows programs embedding Caddy to customize the browse template.

* logging: Clone array on log filters, prevent side-effects (caddyserver#5786)

Fixes https://caddy.community/t/is-caddy-mutating-header-content-from-logging-settings/20947

* logging: query filter for array of strings (caddyserver#5779)

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* ci: Run govulncheck (caddyserver#5790)

* feat(ci): check vuln Go mods in CI

* fix(ci): correct directive for govulncheck

* refactor(ci): move govulncheck to lint.yml

* refactor(lint): move govulncheck to different job

* cmd: Prevent overwriting existing env vars with `--envfile` (caddyserver#5803)

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* httpcaddyfile: fix placeholder shorthands in named routes (caddyserver#5791)

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* reverseproxy: fix nil pointer dereference in AUpstreams.GetUpstreams (caddyserver#5811)

fix a nil pointer dereference in AUpstreams.GetUpstreams when AUpstreams.Versions is not set (fixes caddyserver#5809)

Signed-off-by: Pascal Vorwerk <info@fossores.de>

* fileserver: browse template SVG icons and UI tweaks (caddyserver#5812)

* fileserver browse.html UI tweaks: folder-symlink icon, search

fileserver browse.html UI tweaks: folder-symlink icon, search

- ui - add folder-symlink SVG icon
- search: use `<input type="search">` instead of `text`
- fix npe with `sizebar.style.width` = null in grid mode

* tabify whitespace

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

---------

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* caddyhttp: Use LimitedReader for HTTPRedirectListener

* build(deps): bump actions/checkout from 3 to 4 (caddyserver#5846)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* build(deps): bump goreleaser/goreleaser-action from 4 to 5 (caddyserver#5847)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix: caddytest.AssertResponseCode error message (caddyserver#5853)

* reverseproxy: Allow fallthrough for response handlers without routes (caddyserver#5780)

* templates: Add dummy `RemoteAddr` to `httpInclude` request, proxy compatibility (caddyserver#5845)

* Enhancement: Allow X-Forwarded-For Header in httpInclude Virtual Requests

The goal of this enhancement is to modify the funcHTTPInclude function in the Caddy codebase to include the X-Forwarded-For header in the virtual request. This change will enable reverse proxies to set the X-Forwarded-For header, ensuring that the client's IP address is correctly provided to the target endpoint. This modification is essential for applications that depend on the X-Forwarded-For header for various functionalities, such as authentication, logging, or content customization.

* Updated tplcontext.go - set `virtReq.RemoteAddr = "127.0.0.1"`

i have made the suggested changes

* Apply suggestions from code review

* Update modules/caddyhttp/templates/tplcontext.go

---------

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* go.mod: Upgrade dependencies incl. x/net/http

Possibly important for the HTTP/2 Rapid Reset issue.

* fileserver: Add command shortcuts `-l` and `-a` (caddyserver#5854)

* encode: Add `application/wasm*` to the default content types (caddyserver#5869)

* httpcaddyfile: Enable TLS for catch-all site if `tls` directive is specified (caddyserver#5808)

* reverseproxy: Fix retries on "upstreams unavailable" error (caddyserver#5841)

* reverseproxy: fix parsing Caddyfile fails for unlimited request/response buffers (caddyserver#5828)

* cmd: Fix exiting with custom status code, add `caddy -v` (caddyserver#5874)

* Simplify variables for commands

* Add --envfile support for adapt command

* Carry custom status code for commands to os.Exit()

* cmd: add `-v` and `--version` to root caddy command

* Add `--envfile` to `caddy environ`, extract flag parsing to func

---------

Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>

* httpcaddyfile: Sort TLS SNI matcher for deterministic JSON output (caddyserver#5860)

* httpcaddyfile: Sort TLS SNI matcher, for deterministic adapt output

* Update caddyconfig/httpcaddyfile/httptype.go

---------

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* reverseproxy: Replace health header placeholders (caddyserver#5861)

* reverseproxy: Add logging for dynamic A upstreams (caddyserver#5857)

* reverseproxy: Fix `least_conn` policy regression (caddyserver#5862)

* reverseproxy: Add more debug logs (caddyserver#5793)

* reverseproxy: Add more debug logs

This makes debug logging very noisy when reverse proxying, but I guess
that's the point.

This has shown to be useful in troubleshooting infrastructure issues.

* Update modules/caddyhttp/reverseproxy/streaming.go

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* Update modules/caddyhttp/reverseproxy/streaming.go

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* Add opt-in `trace_logs` option

* Rename to VerboseLogs

---------

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* tls: Add X25519Kyber768Draft00 PQ "curve" behind build tag (caddyserver#5852)

… when compiled with cfgo (https://github.com/cloudflare/go).

* fileserver: Set canonical URL on browse template (caddyserver#5867)

* Browse.html: Add canonical URL and home-link

When contents are equal, but maybe just a sort order is different, it is good to add `<link rel="canonical" href="base-path/" />`. This helps search engines propeely index the page.

I also added a link to the home page with the name of `{{.Host}}` just above the bread crumbs to make the page clearer.

https://paste.tnonline.net/files/28Wun5CQZiqA_Screenshot_20231007_134435_Opera.png

* Update browse.html

* ci: Force the Go version for govulncheck (caddyserver#5879)

* admin: Respond with 4xx on non-existing config path (caddyserver#5870)

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* caddyfile: Fix variadic placeholder false positive when token contains `:` (caddyserver#5883)

* cmd: upgrade: resolve symlink of the executable (caddyserver#5891)

* httpcaddyfile: Fix TLS automation policy merging with get_certificate (caddyserver#5896)

* templates: Clarify `include` args docs, add `.ClientIP` (caddyserver#5898)

* core: quic listener will manage the underlying socket by itself (caddyserver#5749)

* core: quic listener will manage the underlying socket by itself.

* format code

* rename sharedQUICTLSConfig to sharedQUICState, and it will now manage the number of active requests

* add comment

* strict unwrap type

* fix unwrap

* remove comment

* cmd: Add newline character to version string in CLI output (caddyserver#5895)

* caddyhttp: Use sync.Pool to reduce lengthReader allocations (caddyserver#5848)

* Use sync.Pool to reduce lengthReader allocations

Signed-off-by: Harish Shan <140232061+perhapsmaple@users.noreply.github.com>

* Add defer putLengthReader to prevent leak

Signed-off-by: Harish Shan <140232061+perhapsmaple@users.noreply.github.com>

* Cleanup in putLengthReader

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

---------

Signed-off-by: Harish Shan <140232061+perhapsmaple@users.noreply.github.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* core: Apply SO_REUSEPORT to UDP sockets (caddyserver#5725)

* core: Apply SO_REUSEPORT to UDP sockets

For some reason, 10 months ago when I implemented SO_REUSEPORT
for TCP, I didn't realize, or forgot, that it can be used for UDP too. It is a
much better solution than using deadline hacks to reuse a socket, at
least for TCP.

Then mholt/caddy-l4#132 was posted,
in which we see that UDP servers never actually stopped when the
L4 app was stopped. I verified this using this command:

    $ nc -u 127.0.0.1 55353

combined with POSTing configs to the /load admin endpoint (which
alternated between an echo server and a proxy server so I could tell
which config was being used).

I refactored the code to use SO_REUSEPORT for UDP, but of course
we still need graceful reloads on all platforms, not just Unix, so I
also implemented a deadline hack similar to what we used for
TCP before. That implementation for TCP was not perfect, possibly
having a logical (not data) race condition; but for UDP so far it
seems to be working. Verified the same way I verified that SO_REUSEPORT
works.

I think this code is slightly cleaner and I'm fairly confident this code
is effective.

* Check error

* Fix return

* Fix var name

* implement Unwrap interface and clean up

* move unix packet conn to platform specific file

* implement Unwrap for unix packet conn

* Move sharedPacketConn into proper file

* Fix Windows

* move sharedPacketConn and fakeClosePacketConn to proper file

---------

Co-authored-by: Weidi Deng <weidi_deng@icloud.com>

* httpcaddyfile: Remove port from logger names (caddyserver#5881)

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* templates: Delete headers on `httpError` to reset to clean slate (caddyserver#5905)

* go.mod: CVE-2023-45142 Update opentelemetry (caddyserver#5908)

* go.mod: Upgrade quic-go to v0.39.1

* caddyhttp: Adjust `scheme` placeholder docs (caddyserver#5910)

* Upgrade acmeserver to github.com/go-chi/chi/v5 (caddyserver#5913)

This commit upgrades the router used in the acmeserver to
github.com/go-chi/chi/v5. In the latest release of step-ca, the router
used by certificates was upgraded to that version.

Fixes caddyserver#5911

Signed-off-by: Mariano Cano <mariano.cano@gmail.com>

* test: acmeserver: add smoke test for the ACME server directory (caddyserver#5914)

* chore: Fix usage pool comment (caddyserver#5916)

* update quic-go to v0.39.3 (caddyserver#5918)

* go.mod: update quic-go version to v0.40.0 (caddyserver#5922)

* Revert "caddyhttp: Use sync.Pool to reduce lengthReader allocations (caddyserver#5848)" (caddyserver#5924)

* fileserver: Add .m4v for browse template icon

* httpredirectlistener: Only set read limit for when request is HTTP (caddyserver#5917)

* chore: Bump otel to v1.21.0. (caddyserver#5949)

Signed-off-by: Dan Lorenc <dlorenc@chainguard.dev>

* panic when reading from backend failed to propagate stream error (caddyserver#5952)

* http2 uses new round-robin scheduler (caddyserver#5946)

* templates: Offically make templates extensible (caddyserver#5939)

* templates: Offically make templates extensible

This supercedes caddyserver#4757 (and caddyserver#4568) by making template extensions
configurable.

The previous implementation was never documented AFAIK and had only
1 consumer, which I'll notify as a courtesy.

* templates: Add 'maybe' function for optional components

* Try to fix lint error

* tls: accept placeholders in string values of certificate loaders (caddyserver#5963)

* tls: loader: accept placeholders in string values

* appease the linter

* caddytls: Context to DecisionFunc (caddyserver#5923)

See caddyserver/certmagic#255

* caddytls: Sync distributed storage cleaning (caddyserver#5940)

* caddytls: Log out remote addr to detect abuse

* caddytls: Sync distributed storage cleaning

* Handle errors

* Update certmagic to fix tiny bug

* Split off port when logging remote IP

* Upgrade CertMagic

* chore: cross-build for AIX (caddyserver#5971)

* core: Always make AppDataDir for InstanceID (caddyserver#5976)

* cmd: Preserve LastModified date when exporting storage (caddyserver#5968)

* proxyprotocol: use github.com/pires/go-proxyproto (caddyserver#5915)

* proxyprotocol: use github.com/pires/go-proxyproto

* Fix typo: r/generelly/generally

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* add config options for `Deny` CIDR and fallback policy

* use `netip` package & trust unix sockets

---------

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* caddyhttp: Add `uuid` to access logs when used (caddyserver#5859)

* fileserver: New --precompressed flag (caddyserver#5880)

exposes the file_server precompressed functionality to be used with the
file-server command

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* fileserver: Enable compression for command by default (caddyserver#5855)

* feat: enable compression for file-server

* refactor

* const

* Update help text

* Update modules/caddyhttp/fileserver/command.go

---------

Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* go.mod: Updated quic-go to v0.40.1 (caddyserver#5983)

* metrics: Record request metrics on HTTP errors (caddyserver#5979)

* httpcaddyfile: Sort skip_hosts for deterministic JSON (caddyserver#5990)

* httpcaddyfile: Sort skip_hosts for deterministic JSON

* Update caddyconfig/httpcaddyfile/httptype.go

Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>

* Fix test

* Bah

---------

Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>

* logging: Add `zap.Option` support (caddyserver#5944)

* cmd: use automaxprocs for better perf in containers (caddyserver#5711)

* feat: use automaxprocs for better perf in containers

* better logs

* cs

* build(deps): bump golang.org/x/crypto from 0.16.0 to 0.17.0 (caddyserver#5994)

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.16.0 to 0.17.0.
- [Commits](golang/crypto@v0.16.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: Pistasj <hi@pistasjis.net>
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
Signed-off-by: Pascal Vorwerk <info@fossores.de>
Signed-off-by: Harish Shan <140232061+perhapsmaple@users.noreply.github.com>
Signed-off-by: Mariano Cano <mariano.cano@gmail.com>
Signed-off-by: Dan Lorenc <dlorenc@chainguard.dev>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
Co-authored-by: Jacob Gadikian <jacobgadikian@gmail.com>
Co-authored-by: Shyim <github@shyim.de>
Co-authored-by: Aaron Dewes <aaron@runcitadel.space>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: pistasjis <57069715+pistasjis@users.noreply.github.com>
Co-authored-by: guangwu <guoguangwu@magic-shield.com>
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Co-authored-by: Karun Agarwal <113603846+singhalkarun@users.noreply.github.com>
Co-authored-by: Marten Seemann <martenseemann@gmail.com>
Co-authored-by: WeidiDeng <weidi_deng@icloud.com>
Co-authored-by: Paul Jeannot <paul.jeannot95@gmail.com>
Co-authored-by: Đỗ Trọng Hải <41283691+hainenber@users.noreply.github.com>
Co-authored-by: Evan Van Dam <evandam92@gmail.com>
Co-authored-by: Pascal Vorwerk <info@fossores.de>
Co-authored-by: glowinthedark <48893368+glowinthedark@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kévin Dunglas <kevin@dunglas.fr>
Co-authored-by: Patrick Koenig <pkoenig10@gmail.com>
Co-authored-by: Thanmay Nath <110758050+ThanmayNath@users.noreply.github.com>
Co-authored-by: Christoph <github@yozora.eu>
Co-authored-by: Fred Cox <mcfedr@gmail.com>
Co-authored-by: Bas Westerbaan <bas@westerbaan.name>
Co-authored-by: Forza <68693597+Forza-tng@users.noreply.github.com>
Co-authored-by: Norman Soetbeer <norman.soetbeer@gmail.com>
Co-authored-by: Harish Shan <140232061+perhapsmaple@users.noreply.github.com>
Co-authored-by: Ethan Brown (Domino) <111539728+ddl-ebrown@users.noreply.github.com>
Co-authored-by: Mariano Cano <mariano.cano@gmail.com>
Co-authored-by: dlorenc <lorenc.d@gmail.com>
Co-authored-by: Andreas Kohn <andreas.kohn@gmail.com>
Co-authored-by: Benjamin Marwell <bmarwell@apache.org>
Co-authored-by: Aziz Rmadi <46684200+armadi1809@users.noreply.github.com>
Co-authored-by: Jens-Uwe Mager <jum@anubis.han.de>
Co-authored-by: David DeMoss <ddemoss222@gmail.com>
Co-authored-by: Tim Geoghegan <timgeog@gmail.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
bug 🐞 Something isn't working needs info 📭 Requires more information
Projects
None yet
Milestone
v2.7.4
Development

Successfully merging a pull request may close this issue.

reverseproxy: Always return new upstreams (fix #5736) caddyserver/caddy
3 participants
@mholt @kkroo @mohammed90