Consider defining a helper "health check" utility for distroless scenarios

[mthalman]

The use of the HEALTHCHECK instruction often involves the use of a command that executes an HTTP request to the application in order to verify its health. That command might be curl or wget for Linux containers. This becomes problematic for distroless/chiseled containers that would likely want to exclude such commands. How then would a call be made to verify the health of the app?

One solution is to include a basic .NET application that execute the HTTP request (see #4296). Depending on the ubiquity of such a solution, it might make sense for .NET to provide its own implementation of this application that can be easily injected into .NET distroless/chiseled solutions.

I'll illustrate this with an example. Let's say .NET publishes an httpclient image that contains a .NET app which simply sends an HTTP request to a URL provided as an argument. It returns 0 if the request succeeds and 1 if it fails. A distroless implementation could consume this app like the following:

# Build stage excluded for brevity
...

FROM mcr.microsoft.com/dotnet/nightly/runtime-deps:7.0-jammy-chiseled
WORKDIR /app
COPY --from=build /app .

# Inject the .NET httpclient into the image to use for the health check
COPY --from=mcr.microsoft.com/dotnet/httpclient:7.0 /httpclient.exe /
HEALTHCHECK CMD [ "/httpclient.exe", "http://localhost/healthz" ]

ENTRYPOINT ["./app"]

I think it'd be important to understand the landscape of how customers are using health checks to determine the suitability of a pre-defined tool like this. We obviously don't want to be reimplementing curl in .NET but allowing for a little bit of behavior customization in order to fulfill a large portion of customer scenarios seems reasonable.

[mthalman]

[Triage]
As a first step, we'd like to provide documentation in the repo to help guide customer's on the pattern to implement this on their own in a way that doesn't depend on .NET providing a new tool. This is a low-cost solution and can also be used as a means to gather feedback from customers to determine the optimal solution that would meet their needs if we were to publish our own tool.

[64J0]

Does this affect Kubernetes' probes?

I'm not 100% sure, but I think the probes commands are handled outside the container, so it must work even in a distroless environment. Is it correct?

[richlander]

Does this affect Kubernetes' probes?

Not needed for Kubernetes. This is a docker compose need.

[Kralizek]

As far as I understand, having a way to ping the application from within the container is very important in the AWS space for people (like me) using ECS instead of EKS.

[mu88]

@mthalman: is there any news about this? I have a bunch of distroless ASP.NET Core apps running on my Raspi and I'd like to make use of a simple health check probe to http://localhost:8080/healthz in my docker-compose.yml. Or is there any prototype that I can give a shot so that you get some early feedback and I have a MVP? 🙂

[mthalman]

is there any news about this?

There's been recent discussion on this as part of .NET 10 planning. Nothing is decided yet. But we recognize the interest in this feature.

[mu88]

But for the moment, there is no alternative/hack, correct?

[MichelZ]

We have a mini-healthcheck binary that we include in the container:

namespace container.HealthChecker;

internal class Program
{
    static async Task<int> Main(string[] args)
    {
        var httpClient = new HttpClient();
        httpClient.DefaultRequestHeaders.ConnectionClose = true;

        if (args.Length > 1 && Uri.TryCreate(args[1], UriKind.Absolute, out var uri))
            return await httpClient.GetAsync(uri).ContinueWith(r => r.Result.IsSuccessStatusCode) ? 0 : 1;
        else
            throw new ArgumentNullException("A valid URI must be given as first argument");
    }
}

DOCKERFILE:

# Build the HealthChecker
WORKDIR "/src/src/container.HealthChecker"
RUN dotnet publish -c Release --runtime linux-x64 --no-self-contained "container.HealthChecker.csproj" -o "/app/publish/HealthChecker" 

[...]

HEALTHCHECK CMD [ "dotnet", "/app/HealthChecker/container.HealthChecker.dll", "--", "http://localhost:8080/health" ]

[mu88]

Thx for this hint, maybe that's a viable workaround for me 👍🏻 Since I'm using the .NET SDK Container Building Tools, I'd have to build an intermediary Docker base image (e.g. mu88/dotnet-aspnet:9.0) as my apps do no longer have a Dockerfile.

@mthalman: when discussing this particular feature internally, maybe it's worth taking this container publishing approach without an actual Dockerfile into account, too. For example, I could imagine that MS publishes slightly extended base images (e.g. dotnet/aspnet:9.0-chiseled-health) which contain both the runtime (here ASP.NET Core chiseled) and the "mini health check tool". This way, consumers using the .NET SDK Container Building Tools could simply use this as the base image without building it alone.

CC @baronfel

[Kralizek]

I agree wholeheartedly with @mu88 . I also use the dotnet publish approach and I would hate having to return to the dockerfile approach to leverage this feature.

TBF/OT, the SDK approach needs to be extended to extra customization regardless of this or that approach.

[mthalman]

@baronfel - Does the SDK container publishing even have a means to define HEALTHCHECK, regardless of whether distroless is being used?

[lbussell]

Of note, Healthcheck is only for Docker images and not OCI images. The OCI spec does have the healthcheck config item reserved, but only for compatibility with Docker images: https://github.com/opencontainers/image-spec/blob/5325ec48851022d6ded604199a3566254e72842a/media-types.md#applicationvndociimageconfigv1json

[baronfel]

We do not support HEALTHCHECK in any way in the SDK tooling (our issue discussing it was dotnet/sdk-container-builds#316). We chose not to specifically because OCI images (where the world is moving to) doesn't make use of them.

[mu88]

MichelZ's approach inspired me. However, I decided against my first idea and didn't deploy my little health check tool via my Docker base image. Since I use several different .NET base images (dotnet/aspnet:9, dotnet/aspnet:9-noble-chiseled, dotnet/aspnet:9-noble-chiseled-extra), I'd have to maintain several different custom base images, too. As I'm lazy and don't like Dockerfiles 😆 I ship it as part of my NuGet package mu88.Shared as it already contains some shared logic for all my .NET apps. Here you find the corresponding commit.

It took me a while to deploy both the health check assembly and the necessary runtimeconfig.json properly via NuGet so that these files flow through to the app's output directory 😮‍💨 as I ship this file as a <Content> item, the mu88.HealthCheck.runtimeconfig.json now is displayed in the IDE's Solution Explorer:

That's not really nice, but so far I haven't found a solution for this (setting it to <None> doesn't work) - maybe someone here has a deeper understanding of the underlying NuGet workflows and/or a good idea 🙂

Anyhow, now it works properly with this docker-compose.yml and I can still use the .NET SDK Container Building Tools 🤓🥳

services:
  myservice:
    container_name: mycontainer
    image: mu88/myimage:latest-chiseled
    healthcheck:
      test: [ "CMD", "dotnet", "/app/mu88.HealthCheck.dll", "http://localhost:8080/healthz" ]
      interval: 60s
      timeout: 5s
      retries: 3
      start_period: 30s
      start_interval: 5s

If the community should be interested, I could also ship this component as a dedicated NuGet package. But for the moment, I keep it in my already existing package as I don't expect much interest in this niche feature.

Thx again to @MichelZ for your input 🙏🏻

[baronfel]

Glad you found a way to make things work! I think for cross-cutting tools like this the answer that other container platforms have settled on is making specific container layers with apps/diagnostic tooling/etc and then having the container publishing process slip-stream in those layers.

Since a layer is just a file system tarball, practically speaking this means something like:

• publishing the app/tool
• placing it in a file system layout
• creating a tarball of this layout
• pushing the tarball to a registry to create a referenceable Layer
• instructing container building tools to include this Layer

For arch-specific tools this would mean keeping a (signed and verified) inventory of per-RID Layer digests, etc. The SDK Container tooling would also need to implement dotnet/sdk-container-builds#325 to allow pointing to arbitrary Layers. So there's a decent amount of pre-work to get to this final state.

[alexaka1]

I was also bothered on having to give up on healthchecks when using the chiseled images, so I made a docker image that contains a small AOT app, that can perform healthchecks.
https://hub.docker.com/r/alexaka1/distroless-dotnet-healthchecks (with .NET 10 preview support since alexaka1/distroless-dotnet-healthchecks:1.1.0-next.1)

To overcome CPU architectures, I utilized docker's cross compilation, and I have tested it on a macbook (I don't have access to ARM github runners 😢). It works with any .NET 8+ image (even the minimum runtime-deps:*-chiseled-aot image) and it is zero config out of the box.

Example:

FROM mcr.microsoft.com/dotnet/aspnet:8.0-noble-chiseled AS final
# Get the executable and copy it to /healthchecks
COPY --from=ghcr.io/alexaka1/distroless-dotnet-healthchecks:1 / /healthchecks
# Setup the healthcheck using the EXEC array syntax
HEALTHCHECK CMD ["/healthchecks/Distroless.HealthChecks", "--uri", "http://localhost:8080/healthz"]

# Start your app as normal
WORKDIR /app
ENTRYPOINT ["dotnet", "My.Awesome.Webapp.dll"]