add warmup duration secs api

[ramaraochavali]

API For istio/istio#21228

[howardjohn]

A few questions

• If a pod is removed and then latter added back (say, from a blip in readiness probe) does it re-warm?
• If we use create a subset and are just adding more endpoints to that subset, are they warmed?

[ramaraochavali]

If a pod is removed and then latter added back (say, from a blip in readiness probe) does it re-warm?

Yes. As long as Envoy sees the membership change it warms the new pods

If we use create a subset and are just adding more endpoints to that subset, are they warmed?

Depends on subset traffic policy. If we enable this in subset traffic policy it warms subset endpoints also.

[howardjohn]

I understand why it would warm when we have a readiness blip but it does seem like unexpected behavior. Tbh in this scenario we should probably send the endpoint as UNHEALTHY for other reasons as well (locality lb),which would fix this issue? wdyt?

…

On Mon, Nov 15, 2021, 8:09 PM Rama Chavali ***@***.***> wrote: If a pod is removed and then latter added back (say, from a blip in readiness probe) does it re-warm? Yes. As long as Envoy sees the membership change it warms the new pods If we use create a subset and are just adding more endpoints to that subset, are they warmed? Depends on subset traffic policy. If we enable this in subset traffic policy it warms subset endpoints also. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#2153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXMNFXDWCYKYMTZ72I3UMHKQVANCNFSM5IA4RLOA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[ramaraochavali]

I understand why it would warm when we have a readiness blip but it does
seem like unexpected behavior.

Agree but I think it would not create too much of an issue because it is just 30-60s which is the typical configuration used.

Tbh in this scenario we should probably send the endpoint as UNHEALTHY for
other reasons as well (locality lb),which would fix this issue? wdyt?

I agree. I remember we discussed this some time back not sure why we did not change?

[ramaraochavali]

@howardjohn WDYT about this?

[howardjohn]

I am a bit concerned about the behaviors of the warming occurring at times other than pod startup. Maybe users requesting this like @Stono and @BradErz can give some feedback on this?

[ramaraochavali]

@howardjohn While I share the same concern about possibility of unnecessary warmups, I also think it is safe change because

• The unnecessary warmups could be a rare scenario (affecting less endpoint?) and in the grand scheme of may not matter much and the requests wont be failing any ways
• It is opt-in. People can try and if it creates issues they can turn it off

[hzxuzhonghu]

Does it rewarm when eds reload when the other pods down/up?

[ramaraochavali]

If endpoint IP changes the new endpoint will be warmed

[Stono]

👋
Firstly, thanks so much for working on this, it'll be a great feature. And I totally appreciate this is a first pass implementation so my comments below are based on utopian goals.

I understand why it would warm when we have a readiness blip but it does seem like unexpected behavior.

Yes, we certainly wouldn't want to re warm in the following scenarios:

• Readiness Probe going unready
• Outlier Detection triggering

We would want to warm:

• When the pod first starts
• If the pod restarts (crash, liveness probe, preemption)

From my perspective the purpose of this feature is to facilitate warming of JIT runtimes such as the JVM, so that only needs to happen on startup.

We would not use this feature if it warmed in readiness probe failures etc, that would increase our MTR during issues (for example, service 1 depends on some external service, it fails it readinessProbe because some external service is down, as soon as some external service is up, it becomes ready again - we wouldn't want to delay recovery of the service by 60s to facilitate warming).

[ramaraochavali]

Readiness Probe going unready
Outlier Detection triggering

Readiness probe I think currently Istio removes and adds the endpoint - which we should fix as @howardjohn mentioned here #2153 (comment). I will fix that.

For outlier detection - Envoy does not warm.

[howardjohn]

Tricky thing is readiness probe vs liveness probe cannot (reasonably) be determined by Istiod. Likely we will need to do the same for both of those cases unfortunately.

…

On Thu, Nov 25, 2021 at 2:52 AM Rama Chavali ***@***.***> wrote: Readiness Probe going unready Outlier Detection triggering Readiness probe I think currently Istio removes and adds the endpoint - which we should fix as @howardjohn <https://github.com/howardjohn> mentioned here #2153 (comment) <#2153 (comment)>. I will fix that. For outlier detection - Envoy does not warm. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXLQZ6ISAGFL72KTKK3UNYINTANCNFSM5IA4RLOA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[Stono]

Tricky thing is readiness probe vs liveness probe cannot (reasonably) be determined by Istiod. Likely we will need to do the same for both of those cases unfortunately.
…
On Thu, Nov 25, 2021 at 2:52 AM Rama Chavali @.***> wrote: Readiness Probe going unready Outlier Detection triggering Readiness probe I think currently Istio removes and adds the endpoint - which we should fix as @howardjohn https://github.com/howardjohn mentioned here #2153 (comment) <#2153 (comment)>. I will fix that. For outlier detection - Envoy does not warm. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2153 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXLQZ6ISAGFL72KTKK3UNYINTANCNFSM5IA4RLOA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Because they're both observed as endpoint changes and either would remove it from the endpoint list?

If the choice was between:

1. Not warming after a liveness probe failure (bad), but also not warming after readiness probe blips (good)
2. Warming after a liveness probe failure (good), but also warmed after readiness probe blips (bad)

Eg, warm once after the pod is created but never again.

[howardjohn]

Yeah exactly. Hypothetically we could try to read to pod spec and reverse engineer that it was killed by a liveness probe but I suspect that would be quite tricky. I think I missed what choice you prefer - choice (1)? That would be the one I lean towards On Mon, Nov 29, 2021 at 9:30 AM Karl Stoney ***@***.***> wrote:

…

Tricky thing is readiness probe vs liveness probe cannot (reasonably) be determined by Istiod. Likely we will need to do the same for both of those cases unfortunately. … <#m_520272831354884788_> On Thu, Nov 25, 2021 at 2:52 AM Rama Chavali *@*.***> wrote: Readiness Probe going unready Outlier Detection triggering Readiness probe I think currently Istio removes and adds the endpoint - which we should fix as @howardjohn <https://github.com/howardjohn> https://github.com/howardjohn mentioned here #2153 <#2153> (comment) <#2153 (comment) <#2153 (comment)>>. I will fix that. For outlier detection - Envoy does not warm. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2153 (comment) <#2153 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXLQZ6ISAGFL72KTKK3UNYINTANCNFSM5IA4RLOA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . Because they're both observed as endpoint changes and either would remove it from the endpoint list? If the choice was between: 1. Not warming after a liveness probe failure (bad), but also not warming after readiness probe blips (good) 2. Warming after a liveness probe failure (good), but also warmed after readiness probe blips (bad) Eg, warm once after the pod is created but never again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXLTWJD43EFXD5AVJ5DUOO2DJANCNFSM5IA4RLOA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[Stono]

Haha yes, sorry, I missed it off, #1!
The logic there being liveiness probe failures are rare as usually indicate bigger issues, readiness probes are more frequent and can be considered somewhat BAU. Happy for a pod not to warm after a liveness failure because i consider warming an optimisation anyway, and then just optimise for the happy path (new pods only).

[jiangshantao-dbg]

can we cherry pick Support slow Start mode in Envoy #13176 to istio/envoy so that we can use EnvoyFilter to use this feature first. @howardjohn

[ramaraochavali]

@jiangshantao-dbg it is already available in master. Are you asking for cherry picking to 1.12 - if yes, we do not generally cherry pick features to release.

[jiangshantao-dbg]

@jiangshantao-dbg it is already available in master. Are you asking for cherry picking to 1.12 - if yes, we do not generally cherry pick features to release.

ok. i got it. thank you!

[ramaraochavali]

@howardjohn Now that we are sending unready endpoints, can we add this field?