stanford-corenlp-3.8.0.jar fail in Spark2 BUT stanford-corenlp-3.7.0.jar works fine (with models in 3.8.0)

[ghost]

This execution fails:
spark2-shell --jars protobuf-java-3.2.0.jar,stanford-corenlp-3.8.0.jar,stanford-corenlp-3.8.0-models.jar
import edu.stanford.nlp.simple.{Document, Sentence}
new Sentence("hello world!")

java.lang.VerifyError: Bad type on operand stack
Exception Details:
Location:
com/google/protobuf/GeneratedMessageV3$ExtendableMessage.getExtensionCount(Lcom/google/protobuf/GeneratedMessage$GeneratedExtension;)I @2: invokevirtual
Reason:
Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
Current Frame:
bci: @2
flags: { }
locals: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
stack: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
Bytecode:
0x0000000: 2a2b b600 22ac

at edu.stanford.nlp.simple.Document.(Document.java:433)
at edu.stanford.nlp.simple.Sentence.(Sentence.java:118)
at edu.stanford.nlp.simple.Sentence.(Sentence.java:126)
... 48 elided

And this works fine:
spark2-shell --jars protobuf-java-3.2.0.jar,stanford-corenlp-3.7.0.jar,stanford-corenlp-3.8.0-models.jar

[J38]

I don't know a lot about the Spark/Stanford CoreNLP process. Is it possible you are pulling in an older/different version of protobuf when running the Spark process? This seems like a protobuf mismatch issue.

[J38]

See this issue: #556

[J38]

I think the problem is the Spark installation uses protobuf 2.4.1 and we use protobuf 3.2.0.
You could try upgrading the protobuf jar the Spark installation uses to 3.2.0 and see if it works, no promises. Technically this is a Spark issue not a CoreNLP issue!

[J38]

Just as an update to anyone looking at this issue, the problem is conflicting protobuf jars. You probably have to manually change the protobuf jar in your setup from 2.5.0 to 3.2.0. More info available on this issue: #556

[JoshuaZe]

i have the same mismatch problem. but why the stanford-corenlp-3.7.0.jar with protobuf-java-3.2.0.jar works fine ?