Cannot load PipelineModel trained on PySpark in Scala

[SidWeng]

I trained and saved a PipelineModel on PySpark, the PipelineModel could be loaded in PySpark.
However, it cannot be loaded in Scala and returned the following message

requirement failed: Error loading metadata: Expected class name org.apache.spark.ml.PipelineModel but found class name pyspark.ml.pipeline.PipelineModel
java.lang.IllegalArgumentException: requirement failed: Error loading metadata: Expected class name org.apache.spark.ml.PipelineModel but found class name pyspark.ml.pipeline.PipelineModel

Version Info:
Spark: 3.3.0
Scala: 2.12
spark-nlp: 4.4.3
PySpark: 3.3.1

[maziyarpanahi]

Could you please share your whole code/pipeline, where are you saving and which environment are you trying to load it back?

[SidWeng]

The environment I run the pipeline and save the model (model is saved at Azure Gen2 storage):
PySpark: 3.3.1
spark-nlp: 4.4.3
Spark: 3.3.0

The environment I load the model:
Scala: 2.12.15
Spark: 3.3.0

Following is my pipeline and ICDOEntityResolver is a Transformer I wrote.

import sparknlp

from sparknlp.base import *
from sparknlp.annotator import SentenceDetector
from sparknlp.base import Pipeline


spark = sparknlp.start(gpu=False)

documentAssembler = DocumentAssembler() \
    .setInputCol("description") \
    .setOutputCol("document")

sentenceDetector = SentenceDetector() \
    .setInputCols(["document"]) \
    .setOutputCol("sentence")

entityResolver = ICDOEntityResolver() \
    .setInputCol("sentence") \
    .setOutputCol("similarity") \
    .setLimit(1) \
    .setEmbeddingsPath("icd_o_128_embeddings.parquet") \
    .setModelPath('abfss://xxxx@xxxx.dfs.core.windows.net/icd_o_sentence_transformer_128_dim_model') \
    .setDevice('cpu')


pipeline = Pipeline(stages = [
    documentAssembler, 
    sentenceDetector, 
    entityResolver])

train = spark.read.table('default.report')

model = pipeline.fit(train)

model.write().save('PATH_TO_AZURE_DATA_LAKE_STORAGE_GEN2')

[SidWeng]

@maziyarpanahi any hint or suggestion ?

[maziyarpanahi]

Hi @SidWeng
A few questions:

• does it work without ICDOEntityResolver? Just to pin point the issue
• does it work locally? (saving in PySpark and loading in Spark/Scala)
• last not list, this import is strange to me:

from sparknlp.base import Pipeline

I think you should import the Pipeline from actual spark.ml rather than our Pipeline, because they are different. All of our examples use from pyspark.ml import Pipeline: https://github.com/search?q=repo%3AJohnSnowLabs%2Fspark-nlp+pipeline&type=code

In fact all you need is:

from sparknlp.base import *
from sparknlp.annotator import *
from pyspark.ml import Pipeline

I would start with this last point first and try to use these 3 imports. If it didn't work (which is very strange if it doesn't), I then follow with point number 1 to remove that

[SidWeng]

It works without ICDOEntityResolver. I only write a Python class for ICDOEntityResolver but a Scala class for ICDOEntityResolver is needed. Currently I use SentenceDetectorDLApproach.scala and sentence_detector_dl.py as example. Any advice for customizing Annotator?

[maziyarpanahi]

thank you, it would be more helpful if we can see the inside of this ICDOEntityResolver as I thought it was a transformer and not an annotator. Writing a custom annotator requires to be part of the actual source code and be compiled entirely within the new JAR, so when you start the session it actually has this annotator. (unlike Transformer which relies on Spark ML itself)