Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[BUG] Spark 3.3.1 UI is broken when loading in vertica spark connector 3.3.4 through spark.driver.extraClassPath and spark.executor.extraClassPath #522

Closed
KevinAppelBofa opened this issue Nov 28, 2022 · 2 comments · Fixed by #524
Closed

[BUG] Spark 3.3.1 UI is broken when loading in vertica spark connector 3.3.4 through spark.driver.extraClassPath and spark.executor.extraClassPath #522

KevinAppelBofa opened this issue Nov 28, 2022 · 2 comments · Fixed by #524
Assignees
@ai-bq
Labels
bug Something isn't working High Priority
Milestone
3.3.5

Comments

@KevinAppelBofa
Copy link

Environment

  • Spark version: 3.3.1
  • Hadoop version: 3.3.4 / 3.2.2
  • Vertica version:
  • Vertica Spark Connector version: 3.3.4
  • Java version: 1.8
  • Additional Environment Information:

Problem Description

  • Describe the issue in as much details as possible, so it is possible to reproduce it.
    We are using the spark configs spark.executor.extraClassPath and spark.driver.extraClassPath to load in the various jdbc jars and other jars we need into the spark, when having Spark 3.3.1 and the spark-vertica-connector-all-3.3.4.jar, this is throwing an error in the Spark UI when trying to click on executors tab.
    api: unavailable
    java.lang.IllegalArgumentException
    at jersey.repackaged.org.objectweb.asm.ClassVisitor.(ClassVisitor.java:79)
    ...
    Removing this jar, the Spark ui is functional; it appears there is glassfish code in that jar file that is taking precedent over the similar code within the spark 3.3.1
  1. Steps to reproduce: Include spark-vertica-connector-all-3.3.4.jar as part of spark.executor.extraClassPath and spark.driver.extraClassPath, start spark job and and open the ui, click on executors tab and it is blank and generates stacktrace
  2. Expected behaviour: the executors tab to provide the information
  3. Actual behaviour: blank page and stack trace
  4. Error message/stack trace:
  5. 22/11/28 21:20:13 WARN api: unavailable
    java.lang.IllegalArgumentException
    at jersey.repackaged.org.objectweb.asm.ClassVisitor.(ClassVisitor.java:79)
    at jersey.repackaged.org.objectweb.asm.ClassVisitor.(ClassVisitor.java:64)
    at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener$AnnotatedClassVisitor.(AnnotationAcceptingListener.java:172)
    at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener$AnnotatedClassVisitor.(AnnotationAcceptingListener.java:156)
    at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener.(AnnotationAcceptingListener.java:124)
    at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener.newJaxrsResourceAndProviderListener(AnnotationAcceptingListener.java:96)
    at org.glassfish.jersey.server.ResourceConfig.scanClasses(ResourceConfig.java:902)
    at org.glassfish.jersey.server.ResourceConfig._getClasses(ResourceConfig.java:863)
    at org.glassfish.jersey.server.ResourceConfig.getClasses(ResourceConfig.java:772)
    at org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.(ResourceConfig.java:1213)
    at org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.(ResourceConfig.java:1186)
    at org.glassfish.jersey.server.ResourceConfig.createRuntimeConfig(ResourceConfig.java:1182)
    at org.glassfish.jersey.server.ApplicationHandler$RuntimeConfigConfigurator.init(ApplicationHandler.java:182)
    at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$0(ApplicationHandler.java:290)
    at java.util.Arrays$ArrayList.forEach(Arrays.java:3880)
    at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:290)
    at org.glassfish.jersey.server.ApplicationHandler.(ApplicationHandler.java:259)
    at org.glassfish.jersey.servlet.WebComponent.(WebComponent.java:311)
    at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:154)
    at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:347)
    at javax.servlet.GenericServlet.init(GenericServlet.java:180)
    at org.sparkproject.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:632)
    at org.sparkproject.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:486)
    at org.sparkproject.jetty.servlet.ServletHolder.prepare(ServletHolder.java:759)
    at org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
    at org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    at org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
    at org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    at org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
    at org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
    at org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.sparkproject.jetty.server.Server.handle(Server.java:516)
    at org.sparkproject.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    at org.sparkproject.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    at org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.sparkproject.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    at org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
    at org.sparkproject.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.sparkproject.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.lang.Thread.run(Thread.java:750)
    22/11/28 21:20:13 WARN HttpChannel: /api/v1/applications/application_1666540628437_1325/allexecutors
    javax.servlet.ServletException: org.glassfish.jersey.servlet.ServletContainer-3a1ec476==org.glassfish.jersey.servlet.ServletContainer@ef6d77c0{jsp=null,order=-1,inst=true,async=true,src=EMBEDDED:null,STARTED}
    at org.sparkproject.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:650)
    at org.sparkproject.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:486)
    at org.sparkproject.jetty.servlet.ServletHolder.prepare(ServletHolder.java:759)
    at org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
    at org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    at org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
    at org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    at org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
    at org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
    at org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.sparkproject.jetty.server.Server.handle(Server.java:516)
    at org.sparkproject.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    at org.sparkproject.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    at org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.sparkproject.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    at org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
    at org.sparkproject.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.sparkproject.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.lang.Thread.run(Thread.java:750)
    Caused by: java.lang.IllegalArgumentException
    at jersey.repackaged.org.objectweb.asm.ClassVisitor.(ClassVisitor.java:79)
    at jersey.repackaged.org.objectweb.asm.ClassVisitor.(ClassVisitor.java:64)
    at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener$AnnotatedClassVisitor.(AnnotationAcceptingListener.java:172)
    at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener$AnnotatedClassVisitor.(AnnotationAcceptingListener.java:156)
    at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener.(AnnotationAcceptingListener.java:124)
    at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener.newJaxrsResourceAndProviderListener(AnnotationAcceptingListener.java:96)
    at org.glassfish.jersey.server.ResourceConfig.scanClasses(ResourceConfig.java:902)
    at org.glassfish.jersey.server.ResourceConfig._getClasses(ResourceConfig.java:863)
    at org.glassfish.jersey.server.ResourceConfig.getClasses(ResourceConfig.java:772)
    at org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.(ResourceConfig.java:1213)
    at org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.(ResourceConfig.java:1186)
    at org.glassfish.jersey.server.ResourceConfig.createRuntimeConfig(ResourceConfig.java:1182)
    at org.glassfish.jersey.server.ApplicationHandler$RuntimeConfigConfigurator.init(ApplicationHandler.java:182)
    at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$0(ApplicationHandler.java:290)
    at java.util.Arrays$ArrayList.forEach(Arrays.java:3880)
    at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:290)
    at org.glassfish.jersey.server.ApplicationHandler.(ApplicationHandler.java:259)
    at org.glassfish.jersey.servlet.WebComponent.(WebComponent.java:311)
    at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:154)
    at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:347)
    at javax.servlet.GenericServlet.init(GenericServlet.java:180)
    at org.sparkproject.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:632)
    ... 29 more
  6. Code sample or example on how to reproduce the issue:

Spark Connector Logs

  • Add related logs entries here.
@KevinAppelBofa KevinAppelBofa added the bug Something isn't working label Nov 28, 2022
@ai-bq
Copy link
Collaborator

ai-bq commented Nov 28, 2022

Hi @KevinAppelBofa, thanks for bringing this up. We'll take a look at it.

@jeremyprime jeremyprime added this to the 3.3.5 milestone Dec 1, 2022
@jonathanl-bq jonathanl-bq mentioned this issue Dec 1, 2022
Merged
@ai-bq ai-bq mentioned this issue Dec 6, 2022
Merged
@ai-bq ai-bq closed this as completed in #524 Dec 7, 2022
@jeremyprime
Copy link
Collaborator

@KevinAppelBofa, we made changes to our shaded JAR to ensure there is only one version of Jersey (excluded the older version), and in testing the executors tab no longer throws an exception.

This fix is part of v3.3.5, which has now been released.

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@ai-bq ai-bq

Labels
bug Something isn't working High Priority
Projects
None yet
Milestone
3.3.5
3 participants
@KevinAppelBofa @jeremyprime @ai-bq