Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

When Opentelemetry performs dynamic injection (using the agentmain method entry point), the injection reports a java.lang.VerifyError #12881

Open
shenlvcheng opened this issue Dec 12, 2024 · 7 comments
Labels
needs author feedback Waiting for additional feedback from the author stale

Comments

@shenlvcheng
Copy link

Describe the bug

When our project starts up, we found that it's not generating trace metrics data. After adding the -Dotel.javaagent.debug=true parameter, we discovered this issue.

[otel.javaagent 2024-12-12 09:39:34:954 +0800] [Attach Listener] DEBUG io.opentelemetry.javaagent.tooling.AgentInstaller$RedefinitionLoggingListener - Exception while retransforming 76 classes: [class org.springframework.core.task.SimpleAsyncTaskExecutor, class org.springframework.beans.factory.support.DisposableBeanAdapter, class org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter, class org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter, class org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter, class org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter, class org.springframework.context.support.AbstractApplicationContext$1, class org.springframework.context.support.ContextTypeMatchClassLoader, class org.springframework.core.DecoratingClassLoader, class org.springframework.web.context.support.XmlWebApplicationContext, class org.springframework.web.context.support.AbstractRefreshableWebApplicationContext, class ch.qos.logback.classic.spi.LoggingEvent, class ch.qos.logback.classic.Logger, interface org.springframework.web.servlet.HandlerAdapter, class org.springframework.web.servlet.DispatcherServlet, class org.springframework.web.servlet.FrameworkServlet, class org.springframework.web.servlet.HttpServletBean, class org.apache.catalina.connector.CoyoteOutputStream, interface javax.servlet.AsyncContext, class org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState, class org.apache.tomcat.util.net.SocketWrapperBase$OperationState, class org.apache.tomcat.util.net.Acceptor, class org.apache.tomcat.util.threads.TaskThread$WrappingRunnable, class org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker, class org.apache.tomcat.util.threads.ThreadPoolExecutor, class org.apache.tomcat.util.threads.TaskThread, class org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor, class org.apache.tomcat.websocket.server.WsFilter, interface javax.servlet.Filter, class org.apache.jasper.servlet.JspServlet, class org.apache.catalina.servlets.DefaultServlet, class javax.servlet.http.HttpServlet, class javax.servlet.GenericServlet, class org.apache.catalina.loader.ParallelWebappClassLoader, class org.apache.catalina.loader.WebappClassLoaderBase, interface javax.servlet.Servlet, class org.apache.catalina.startup.HostConfig$DeployWar, class javax.servlet.ServletOutputStream, class org.apache.catalina.core.ContainerBase$StartChild, class org.apache.catalina.connector.CoyoteAdapter, class org.apache.tomcat.util.net.NioEndpoint$Poller, class org.apache.coyote.AbstractProtocol$AsyncTimeout, class org.apache.catalina.connector.Response, interface javax.servlet.http.HttpServletResponse, class org.apache.catalina.connector.Request, interface javax.servlet.http.HttpServletRequest, interface javax.servlet.ServletRequest, class org.apache.tomcat.util.net.NioEndpoint$SocketProcessor, class org.apache.tomcat.util.net.SocketProcessorBase, class org.apache.catalina.startup.Catalina$CatalinaShutdownHook, class org.apache.juli.AsyncFileHandler$1, class org.apache.juli.ClassLoaderLogManager$Cleaner, class org.apache.juli.ClassLoaderLogManager$RootLogger, class java.util.concurrent.CompletableFuture, class java.util.concurrent.ForkJoinTask, class java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask, interface java.util.concurrent.RunnableScheduledFuture, class java.util.concurrent.ScheduledThreadPoolExecutor, class java.util.concurrent.Executors$RunnableAdapter, class java.util.concurrent.FutureTask, interface java.util.concurrent.RunnableFuture, class java.lang.invoke.InnerClassLambdaMetafactory, class java.util.concurrent.ThreadPoolExecutor$Worker, interface java.util.concurrent.Callable, class java.net.HttpURLConnection, class java.util.logging.LogManager$RootLogger, class java.util.concurrent.ThreadPoolExecutor, class java.util.concurrent.Executors$FinalizableDelegatedExecutorService, class java.util.concurrent.Executors$DelegatedExecutorService, class java.util.concurrent.AbstractExecutorService, class java.util.logging.Logger, class sun.reflect.Reflection, class sun.misc.Launcher$AppClassLoader, class java.net.URLClassLoader, class java.lang.ClassLoader, class java.lang.Class]
java.lang.VerifyError
at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at net.bytebuddy.utility.Invoker$Dispatcher.invoke(Unknown Source)
at net.bytebuddy.utility.dispatcher.JavaDispatcher$Dispatcher$ForNonStaticMethod.invoke(JavaDispatcher.java:1033)
at net.bytebuddy.utility.dispatcher.JavaDispatcher$ProxiedInvocationHandler.invoke(JavaDispatcher.java:1163)
at net.bytebuddy.agent.builder.$Proxy28.retransformClasses(Unknown Source)
at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(AgentBuilder.java:8611)
at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(AgentBuilder.java:8426)
at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy.apply(AgentBuilder.java:6123)
at net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:11737)
at net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:11637)
at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:202)
at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:109)
at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:101)
at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:54)
at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:48)
at io.opentelemetry.javaagent.bootstrap.AgentInitializer.execute(AgentInitializer.java:70)
at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:47)
at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:59)
at io.opentelemetry.javaagent.OpenTelemetryAgent.agentmain(OpenTelemetryAgent.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:411)

Steps to reproduce

No

Expected behavior

The data can be exported

Actual behavior

The trace data cannot be exported

Javaagent or library instrumentation version

2.10.0

Environment

JDK:1.8.
OS: 1.8.0_421
spring framework:4.0.0.RELEASE

Additional context

[
spring-mvc-1.6.zip
](url)

test demo

@shenlvcheng shenlvcheng added bug Something isn't working needs triage New issue that requires triage labels Dec 12, 2024
@steverao
Copy link
Contributor

According to your demo, it seems everything is fine in my environment. It can create spans and print them out.
image

@steverao steverao added needs author feedback Waiting for additional feedback from the author and removed bug Something isn't working needs triage New issue that requires triage labels Dec 13, 2024
@shenlvcheng
Copy link
Author

According to your demo, it seems everything is fine in my environment. It can create spans and print them out. image

Are you using the -javagent method? Don't use this approach. I just tested it again and it won't export the log data. You should use the dynamic approach instead:
VirtualMachine vm = VirtualMachine.attach(pid);
vm.loadAgent(javaagentPath);

@github-actions github-actions bot removed the needs author feedback Waiting for additional feedback from the author label Dec 16, 2024
@steverao
Copy link
Contributor

You should use the dynamic approach instead: VirtualMachine vm = VirtualMachine.attach(pid); vm.loadAgent(javaagentPath);

As far as I know, we don't support it so far.

@shenlvcheng
Copy link
Author

You should use the dynamic approach instead: VirtualMachine vm = VirtualMachine.attach(pid); vm.loadAgent(javaagentPath);

As far as I know, we don't support it so far.

agentmain is the entry point for attach injection, while premain is the entry point for -javaagent injection. The source code clearly provides this entry point, indicating that attach injection is supported

微信截图_20241216135118

@steverao
Copy link
Contributor

steverao commented Dec 16, 2024

agentmain is the entry point for attach injection, while premain is the entry point for -javaagent injection. The source code clearly provides this entry point, indicating that attach injection is supported

We don't support or test it officially, you can refer to previous discussion #2198.
Hi @trask, is it necessary to add some comments on agentmain?

@laurit
Copy link
Contributor

laurit commented Dec 16, 2024

@shenlvcheng if you wish for someone to take a look then please provide a minimal application that reproduces the issue, the sample app that you provided does not include the dynamic attach code as far as I can tell, along with any necessary instructions. Generally our advice is that you perform the dynamic attach as early as possible, from the stack trace you provided it looks like you are not doing this. We are open to reasonable PRs that fix issues when agent is attached later, but we can not commit to fixing issues when attach is not called immediately when the application starts.

@laurit laurit added the needs author feedback Waiting for additional feedback from the author label Dec 16, 2024
Copy link
Contributor

This has been automatically marked as stale because it has been marked as needing author feedback and has not had any activity for 7 days. It will be closed automatically if there is no response from the author within 7 additional days from this comment.

@github-actions github-actions bot added the stale label Dec 23, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
needs author feedback Waiting for additional feedback from the author stale
Projects
None yet
Development

No branches or pull requests

3 participants