ByteBuddyAgent.installExternal hangs forever

[shalom938]

Hello,
We have an issue related to bytebuddy that happens in very specific conditions.probably related to relocating bytebuddy packages in a shaded jar.
for context , it is all part of digma-ai intellij plugin

We have a java agent that uses bytebuddy to inject annotations to specific methods based on predefined criteria. we inject opentelemetry WithSpan and another marker annotation.
this is the transformer code:

new AgentBuilder.Default()
                .type(TypeMatchers.create(Configuration.getInstance()))
                .transform((builder, typeDescription, classLoader, module, protectionDomain) -> {

                    try {

                        AnnotationDescription withSpanAnnotationDescription = getWithSpanAnnotationDescription(classLoader);

                        AnnotationDescription digmaMarkerAnnotationDescription = getDigmaMarkerAnnotationDescription();

                        Log.debug("transforming " + typeDescription.getCanonicalName() + " in class loader " + classLoader);

                        return builder
                                .visit(new MemberAttributeExtension.ForMethod()
                                        .annotateMethod(withSpanAnnotationDescription)
                                        .annotateMethod(digmaMarkerAnnotationDescription)
                                        .on(MethodMatchers.create(typeDescription, Configuration.getInstance())));
                    } catch (Throwable e) {
                        Log.error("got exception in bytebuddy transformer", e);
                        return builder;
                    }

                }).installOn(inst);

The transformer uses a long list of type and method matchers.

The agent is packaged as shaded jar using gradle shadow plugin and relocates bytebuddy packages
relocate("net.bytebuddy", "org.digma.net.bytebuddy")
the agent also uses opentelemetry-javaagent-extension-api because it has some useful matchers and relocates that too
relocate("io.opentelemetry", "org.digma.io.opentelemetry")

on startup the agent checks if opentelemetry WithSpan is available in system class loader and if not injects the opetelemetry api to system classloader search using instrumentation.appendToSystemClassLoaderSearch
it adds the following jars:
opentelemetry-api-1.35.0.jar
opentelemetry-context-1.35.0.jar
opentelemetry-instrumentation-annotations-2.1.0.jar

we add the agent to application with JAVA_TOOL_OPTIONS together with opentelemetry java agent, something like that:
JAVA_TOOL_OPTIONS="-javaagent:/tmp/digma-agent.jar -javaagent:/tmp/opentelemetry-javaagent.jar -Dotel.javaagent.extensions=/tmp/digma-otel-agent-extension.jar -Dotel.exporter.otlp.traces.endpoint=https://localhost:5051 -Dotel.service.name=spring-petclinic.test -Dotel.traces.exporter=otlp -Dotel.exporter.otlp.protocol=grpc "

It all works well most of the time. but if the application tries to install a bytebuddy agent by calling ByteBuddyAgent.install the process hangs forever. one example is a unit test with mockito.
ByteBuddyAgent.installExternal tries to start an attache process and waits forever, looks like the process can not start but never fails.
this is a part of thread dump:

"Test worker" #1 prio=5 os_prio=31 cpu=6213.86ms elapsed=47.50s tid=0x00007fbcd7808a00 nid=0x2303 in Object.wait() [0x0000700009209000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@17.0.11/Native Method)
- waiting on <0x00000007e6aef5c8> (a java.lang.ProcessImpl)
at java.lang.Object.wait(java.base@17.0.11/Object.java:338)
at java.lang.ProcessImpl.waitFor(java.base@17.0.11/ProcessImpl.java:434)
- locked <0x00000007e6aef5c8> (a java.lang.ProcessImpl)
at net.bytebuddy.agent.ByteBuddyAgent.installExternal(ByteBuddyAgent.java:705)
at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:636)
at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:616)
- locked <0x00000007e6a05488> (a java.lang.Class for net.bytebuddy.agent.ByteBuddyAgent)
at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:568)
at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:545)
at org.mockito.internal.creation.bytebuddy.InlineDelegateByteBuddyMockMaker.(InlineDelegateByteBuddyMockMaker.java:133)
at org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.(InlineByteBuddyMockMaker.java:23)

Now, and this is the weird part, it happens only on certain conditions, and we manage to make it work in these condition by adding some vmoptions.
we've tested these condition again and again by different people and its 100% consistent.

• it only happens in macos arm,we tested on macos sonoma arm. It never happens on linux or windows.
• It only happens with jdk 17, with 21 it doesn't happen. we tried various vendors.

with the above condition we manage to make it work by doing any of the following:

• If we add to JAVA_TOOL_OPTIONS -Xmx it doesn't happen anymore , no matter what the value is in Xmx, so with -Xmx64m it doesn't happen.
• if we don't relocate bytebuddy classes it doesn't happen.
• if we don't add jars to system class loader search from our agent it doesn't happen

the condition look weird to me, and i couldn't get to the cause of the issue, we have many mac users with jdk 17 and that will effect many of our users.
I have a feeling that the relocated bytebuddy packages is the cause but i'm not sure because we manage to make it work by adding Xmx or by not adding jars to system class loader search.

I know that opentelemetry java agent also uses relocated bytebuddy packages but i still didn't figure out how they do it, they do something not standard. but IMO just relocation with gradle shadow plugin should work, and it works until the application tries to install a bytebuddy agent using attache api.

maybe someone has an idea ?
maybe @raphw ?

[raphw]

I think you are right. I have observed that if two agents attempt to attach to the same JVM from two different class loaders, some VMs hang forever when the second class loader attempts to load the native library that is included with the JVM. This problem was very visible on JDK version 8 but I have not seen it in a while, but I am not using MacOS.

One way to avoid it was to use the emulation natives where the problem did not occur. Another way of doing it was to inject the Instrumentation class into the static field of the ByteBuddyAgentupon load so that a native attach is never attempted on the second class loader.

[shalom938]

@raphw
Thank you for your response.
I understand your explanation of the issue, I think that's what happens.
I understand your proposed solution but i can't implement it because the applications are not ours, we only supply the tools to users packaged in an intellij plugin. maybe i can do the second one but i'd rather not because i don't want to change the flow of user program.
supposing that's what happens, do you have any idea how explain that if i don't relocate bytebuddy packages in our agent the problem will not happen? or if we add -Xmx?
from your experience, is there something special we need to do to relocate bytebuddy packages? currently we don't have any shadow transformers, we just let gradle shadow plugin do it.

[raphw]

I attempted some changes where Byte Buddy attempts to resolve the unshaded "Installer" what would avoid a duplicate attachment. Could you build Byte Buddy from master and try if this solves your problem?

[shalom938]

Yes i can try.
but what do you mean i should use it for? in my agent i don't use bytebuddy agent, i don't even have the byte-buddy-agent dependency, i install directly with installOn(instrumentation).
the application uses byte-buddy-agent ,actually by mockito. so you mean to build master and use it in the application?

[raphw]

Indeed. In this case, try setting you Instrumentation instance to the instrumentation field of the Installer class. This will trigger Byte Buddy no longer trying to attach but reusing the instance. This requires no update. If you use reflection, you can abort this if Byte Buddy is not present on the class path.

[shalom938]

OK, I will try it and write you back

[shalom938]

it will take me some time because i also don't have mac , i'm on linux. i will make the changes and ask a remote colleague to test for me.

[shalom938]

@raphw Hello,
I did what you suggested and it works.
with some details,
I didn't need to use your latest changes to master, what i did in my agent is this:

private static void installInstrumentationOnBytebuddyAgent(Instrumentation myInstrumentation) {
        try {
            Log.debug("Installing Instrumentation on ByteBuddy Installer");
            //need to change the Installer fq name otherwise gradle shadow will relocate it
            Class<?> byteBuddyInstaller = Class.forName("net_bytebuddy_agent_Installer".replaceAll("_", "."), false, ClassLoader.getSystemClassLoader());
            Field instrumentationField = byteBuddyInstaller.getDeclaredField("instrumentation");
            instrumentationField.setAccessible(true);

            Instrumentation instrumentation = (Instrumentation) instrumentationField.get(null);
            if (instrumentation == null) {
                instrumentationField.set(null, myInstrumentation);
            }
            instrumentationField.setAccessible(false);
            Log.debug("Installation of Instrumentation on ByteBuddy Installer succeeded");
        } catch (Exception e) {
            Log.debug("Could not install instrumentation on bytebuddy Installer " + e);
        }
    }

In my agent I don't even have dependency on byte-buddy-agent jar, only byte-buddy.
I don't completely understand how your latest changes with the reveresed class name can help me here, maybe I'm missing something?

what i found is the following behavior, i understand what's happening but i don't understand why.

while my agent is added in command line with -javaagent :
if the application loads on the system class loader, and ByteBuddyAgent is loaded from system class loader, and tries to do external agent install , the problem will happen and the process will hang.
in that case the above solution works, my agent will find Installer in system class loader , will feed it with the Instrumentation, and when the application will call ByteBuddyAgent.install() , bytebuddy will not attempt external install and will use the Instrumentation from the Installer.

if the application loads by a different class loader, for example i tested a spring boot app running with java -jar,
ByteBuddyAgent and Installer are loaded by spring's class loader org.springframework.boot.loader.launch.LaunchedClassLoader.
in that case my agent will not find net.bytebuddy.agent.Installer in system class loader. but, in this case the problem doesn't happen, the external install succeeds.

So I think the problem is only when my agent ,which always loads on system class loader, installs a bytebuddy transformer, and the application is also loaded on system class loader and calls ByteBuddyAgent.install(), only then the process hangs and the above trick solves it.

I don't understand why it behaves like that, and I don't understand why if i don't relocate bytebuddy packages in my agent there is no problem. but I tested the above again and again and its very consistent.
So I think the above solution covers our problem, we don't see this issue in any other conditions, we tested again today on linux and windows and it never happens, and on mac with jdk 21 never happens. only on mac with jdk 17. I think I will run the above method only on mac+jdk17 because I don't want to do it in all other conditions where there is no issue.

if you have more suggestion I'll be happy to hear.

Thank you very much for your help!

[raphw]

With the changes in Byte Buddy, Byte Buddy would now also discover the class from a shaded location, so your work around will be more robust. You just have to make sure to initialize this common dispatcher.

Of course, if another library also does a self-attach, you would need to "preload" that library, too, otherwise you'd run into the same problem.

[shalom938]

OK.
Thank you!

[breun]

I encountered this issue of the external ByteBuddy agent install hanging with a Mockito test in one of my projects. Interestingly enough the issue is 100% reproducible on Azul Zulu OpenJDK 17.0.12, but on Azul Zulu OpenJDK 21.0.4 there is no issue. I've decided to not investigate this further, because this particular project will be abandoned within a month, but I thought it might be an interesting data point.