[SOLVED] How to change output annotator type via mapAnnotationsCol

[marlborino]

Hi all,
I'm trying to define my custom pipeline in order to create a custom lemmatizer.
This is my source code:

val documentAssembler = new DocumentAssembler()
  .setInputCol("text")
  .setOutputCol("document")

val sentence = new SentenceDetector()
  .setInputCols("document")
  .setOutputCol("sentence")
  .setExplodeSentences(false)

val tokenizer = new Tokenizer()
  .setInputCols(Array("sentence"))
  .setOutputCol("token")

val posTagger = PerceptronModel.pretrained(name="pos_ud_isdt", lang="it")
  .setInputCols(Array("document", "token"))
  .setOutputCol("pos")

val linguisticPipeline = new Pipeline().setStages(Array(documentAssembler, sentence, tokenizer, posTagger))
val linguisticPipelineModel = linguisticPipeline.fit(testData)
val out_dataset = linguisticPipelineModel.transform(testData)

val mapped = out_dataset.mapAnnotationsCol("pos", "modpos", (annotations: Seq[Annotation]) => {
  annotations.map(x => Annotation(AnnotatorType.TOKEN, x.begin, x.end, x.metadata.get("word").get+"_"+x.result, Map("sentence" -> "0")))
})

val lemmatizer = new Lemmatizer()
  .setInputCols(Array("modpos"))
  .setOutputCol("lemmas")
  .setDictionary("/data001/LemmaDict_Composer/out/final_dict", "->", "\t")

val terminologyPipeline = new Pipeline().setStages(Array(lemmatizer))
val terminologyPipelineModel = terminologyPipeline.fit(mapped)
val final_dataset = linguisticPipelineModel.transform(mapped)

When I run this code I get the error in the title because the "modpod" column still remain of type POS.
This is the complete error:

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Wrong or missing inputCols annotators in LEMMATIZER_808688b1d3ea.

Current inputCols: modpos. Dataset's columns:
(column_name=id,is_nlp_annotator=false)
(column_name=text,is_nlp_annotator=false)
(column_name=document,is_nlp_annotator=true,type=document)
(column_name=sentence,is_nlp_annotator=true,type=document)
(column_name=token,is_nlp_annotator=true,type=token)
(column_name=pos,is_nlp_annotator=true,type=pos)
(column_name=modpos,is_nlp_annotator=true,type=pos).
Make sure such annotators exist in your pipeline, with the right output names and that they have following annotator types: token
        at scala.Predef$.require(Predef.scala:224)
        at com.johnsnowlabs.nlp.AnnotatorApproach.transformSchema(AnnotatorApproach.scala:68)
        at org.apache.spark.ml.Pipeline$$anonfun$transformSchema$4.apply(Pipeline.scala:184)
        at org.apache.spark.ml.Pipeline$$anonfun$transformSchema$4.apply(Pipeline.scala:184)
        at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57)
        at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66)
        at scala.collection.mutable.ArrayOps$ofRef.foldLeft(ArrayOps.scala:186)
        at org.apache.spark.ml.Pipeline.transformSchema(Pipeline.scala:184)
        at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:74)
        at org.apache.spark.ml.Pipeline.fit(Pipeline.scala:136)
        at Main$.main(Main.scala:151)
        at Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I think I have understood the Annotator/AnnotatorType concept (I'm a SParkNLP newbie) and so in the mapAnnotationsCol I create the annotation of type TOKEN but evidently I miss some point.
Can anyone help me please?

Thanks a lot!

[SameekshaS]

Hi, I am a newbie to spark myself but I faced a similar problem. My idea is that you might have to convert lemmas and transform it from the annotation format of Spark NLP to a “human-readable” format (finisher).

[marlborino]

Thank you @SameekshaS for the response.
My problem is related to the creation of the "lemmas" column because I cannot apply the pipeline composed by the lemmatizer.
So far I have found a workaround (as suggested by you) by transforming the "mapped" dataframe with a finisher and then rebuilding a new pipeline starting from the DocumentAssembler up to the lemmatizer. This obviously solves the problem but introduces a large computation overhead.
Analyzing the issue a bit I think I've understood the problem but I don't know exactly how to solve it. The point is that I should change the column type in the dataframe schema metadata .... but I have to figure out how to do it.

[marlborino]

I've solved with this:

    val metadataBuilder: MetadataBuilder = new MetadataBuilder()
    metadataBuilder.putString("annotatorType", AnnotatorType.TOKEN)
    mapped.schema.fields.update(6, StructField("modpos", ArrayType(Annotation.dataType), nullable = false, metadataBuilder.build))

after the "mapAnnotationsCol" function.

I think this annoying behavior could be corrected by simply inheriting the column metadata value from the annotationType metadata of each annotations in the column.
Anybody knows if there is an open backlog to propose changes?

Thank you all

[maziyarpanahi]

I converted this to a discussion so it can continue without being close as it is interesting. Hope it's OK.

[jinxiangfa]

thanks so much , i try too many times to solve lemma with pos ,but no way util see this article