Container id may be incorrect if cgroup v2 is used

[PeterF778]

When running in a cgroup v2 container, the Java agent attempts to discover the container-id by reading /proc/self/mountinfo and adds it to the reported Resources. However, it has been observed that the container id from that file may be incorrect.

Running from within container:

cat /proc/self/mountinfo | grep hostname
1290 1281 254:1 /docker/containers/0c31511deaae072674a6f90929684d8e2d4138d424ea5717d1694da8dbd1796e/hostname /etc/hostname rw,relatime - ext4 /dev/vda1 rw

while the true container id is different

kubectl  describe pod javatest1-685798bc98-vmldn | grep "Container ID"  
Container ID:   docker://1193d58b89b1704f18fbf9ee950b9e652fe5e1212345f9a1c161cc6782ccc0d7

To reproduce, use a recent version of docker (which uses cgroup v2) and kubernetes.

The code which reads /proc/self/mountinfo was added by this PR

[biswajit-nanda]

This is a definite bug. It is affecting my instrumented applications deployed in Azure K8s Service v1.25.6 using containerd runtime as well.

Here is the resource attributes that I see in the Otel Collector logs:

Resource attributes:
-> container.id: Str(26a8ced5e234614425835d9db4b244d509b20a870d4e5ca1f3b5301f19f05475)
-> host.arch: Str(amd64)
-> host.name: Str(java-unresolved-77569448c-fjrcz)
-> k8s.container.name: Str(java-unresolved)
-> k8s.deployment.name: Str(java-unresolved)
-> k8s.namespace.name: Str(apps-unresolved)
-> k8s.node.name: Str(aks-userpool-28746597-vmss000000)
-> k8s.pod.name: Str(java-unresolved-77569448c-fjrcz)
-> k8s.replicaset.name: Str(java-unresolved-77569448c)
-> os.description: Str(Linux 5.15.0-1036-azure)
-> os.type: Str(linux)
-> process.command_line: Str(/usr/local/openjdk-8/jre/bin/java -javaagent:/otel-auto-instrumentation/javaagent.jar -jar /app/java-unresolved-0.0.1-SNAPSHOT.jar)
-> process.executable.path: Str(/usr/local/openjdk-8/jre/bin/java)
-> process.pid: Int(8)
-> process.runtime.description: Str(Oracle Corporation OpenJDK 64-Bit Server VM 25.342-b07)
-> process.runtime.name: Str(OpenJDK Runtime Environment)
-> process.runtime.version: Str(1.8.0_342-b07)
-> service.name: Str(java-unresolved)
-> service.namespace: Str(apps-unresolved)
-> telemetry.auto.version: Str(1.26.0)
-> telemetry.sdk.language: Str(java)
-> telemetry.sdk.name: Str(opentelemetry)
-> telemetry.sdk.version: Str(1.26.0)

Here is the output of kubectl describe pod output that shows the actual containerID as 6de3882f3f5729b1648d9c5ab1f40dbc90bf6acc2ca08dc3c879c35da1c893a4 :

kubectl -n apps-unresolved describe pod java-unresolved-77569448c-fjrcz | grep "Container ID"

Container ID: containerd://6de3882f3f5729b1648d9c5ab1f40dbc90bf6acc2ca08dc3c879c35da1c893a4

Here is the output of another command that shows the same containerID 6de3882f3f5729b1648d9c5ab1f40dbc90bf6acc2ca08dc3c879c35da1c893a4 :

kubectl -n apps-unresolved get pods -o jsonpath='{range .items[]}{"pod:java-unresolved-77569448c-fjrcz"}{.metadata.name}{"\n"}{range .status.containerStatuses[]}{"\tname: "}{.containerID}{"\n\timage: "}{.image}{"\n"}{end}'
pod:java-unresolved-77569448c-fjrczjava-unresolved-77569448c-fjrcz

name: containerd://6de3882f3f5729b1648d9c5ab1f40dbc90bf6acc2ca08dc3c879c35da1c893a4
image: docker.io/biswajitnanda/java-unresolved:latest

Here is the output of the /proc/self/mountinfo from the container:

kubectl -n apps-unresolved exec -it java-unresolved-77569448c-fjrcz -c java-unresolved -- /bin/bash
root@java-unresolved-77569448c-fjrcz:/app# cat /proc/self/mountinfo | grep hostname

2398 2388 8:1 /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/26a8ced5e234614425835d9db4b244d509b20a870d4e5ca1f3b5301f19f05475/hostname /etc/hostname rw,relatime - ext4 /dev/root rw,discard,errors=remount-ro

JavaAgent v1.26 is in deed getting incorrect containerID from /proc/self/mountinfo.

CC: @PeterF778 , @svrnm , @lo-jason

[svrnm]

@biswajit-nanda see my comment on #6694, no matter how we approach this by reading the container id from files, we will always be depending on a hack and not a real solution