Issue while running the JAR build locally to connect to milvus

[Jeri-jose]

Following the steps to create the JAR

1. Cloned milvus-java-sdk, created the jar file, using mvn clean package -DskipTests
2. The JAR was created at ./milvus-sdk-java/sdk-core/target/milvus-sdk-java-2.5.4-shaded.jar
3. Added the milvus-java-sdk as the systempath in milvus-spark connector and was able to build the JAR file for spark-milvus connector

But when ran the JAR as pyspark --jars /root/spark/test/spark-milvus/target/spark-milvus-1.0.0-SNAPSHOT.jar

getting error at this code snippet as

sample_df.write \
    .mode("append") \
    .option("milvus.host", "hostname") \
    .option("milvus.port", "19530") \
    .option("milvus.collection.name", "hello_spark_milvus") \
    .option("milvus.collection.vectorField", "vec") \
    .option("milvus.collection.vectorDim", "8") \
    .option("milvus.collection.primaryKeyField", "id") \
    .option("milvus.database.name", "default") \
    .option("milvus.username", "username") \
    .option("milvus.password", "password") \
    .format("milvus") \
    .save()

ERROR:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/spark/python/pyspark/sql/readwriter.py", line 966, in save
    self._jwrite.save()
  File "/usr/local/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
  File "/usr/local/spark/python/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/usr/local/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o89.save.
: java.lang.NoClassDefFoundError: io/milvus/client/MilvusServiceClient
        at zilliztech.spark.milvus.MilvusConnection$.acquire(MilvusConnection.scala:31)
        at zilliztech.spark.milvus.Milvus.getTable(Milvus.scala:23)
        at org.apache.spark.sql.DataFrameWriter.getTable$1(DataFrameWriter.scala:278)
        at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:296)
        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: io.milvus.client.MilvusServiceClient
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 17 more

Approach 2
I also tried building the milvus-spark JAR with the dependency given below rather than the locally build milvus-java-sdk JAR

<dependency>
            <groupId>io.milvus</groupId>
            <artifactId>milvus-sdk-java</artifactId>
            <version>2.5.4</version>
            <exclusions>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-databind</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.google.protobuf</groupId>
                    <artifactId>protobuf-java</artifactId>
                </exclusion>
            </exclusions>
   </dependency

then I am getting the error as

25/03/10 21:56:28 ERROR AbstractMilvusGrpcClient: DEADLINE_EXCEEDED: deadline exceeded after 9.669064540s. Name resolution delay 0.000000000 seconds. [closed=[], open=[[wait_for_ready, buffered_nanos=9679145154, waiting_for_connection]]]
25/03/10 21:56:28 ERROR AbstractMilvusGrpcClient: Failed to initialize connection. Error: DEADLINE_EXCEEDED: deadline exceeded after 9.669064540s. Name resolution delay 0.000000000 seconds. [closed=[], open=[[wait_for_ready, buffered_nanos=9679145154, waiting_for_connection]]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/spark/python/pyspark/sql/readwriter.py", line 966, in save
    self._jwrite.save()
  File "/usr/local/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
  File "/usr/local/spark/python/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/usr/local/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o89.save.
: java.lang.RuntimeException: Failed to initialize connection. Error: DEADLINE_EXCEEDED: deadline exceeded after 9.669064540s. Name resolution delay 0.000000000 seconds. [closed=[], open=[[wait_for_ready, buffered_nanos=9679145154, waiting_for_connection]]]
        at io.milvus.client.MilvusServiceClient.<init>(MilvusServiceClient.java:175)
        at zilliztech.spark.milvus.MilvusConnection$.acquire(MilvusConnection.scala:31)
        at zilliztech.spark.milvus.Milvus.getTable(Milvus.scala:23)
        at org.apache.spark.sql.DataFrameWriter.getTable$1(DataFrameWriter.scala:278)
        at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:296)
        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.lang.Thread.run(Thread.java:750)

Moreover I am unable to build the 1.0.1 snapshot JAR for milvus-spark connector ,

THE ABOVE MENTIONED ERROR DOES NOT OCCUR WHILE USING THE SNAPSHOT-1.0.1 jar file downloaded from the git repo

Development Environment
Java version 1.8.0_362
Apache Maven 3.6.3
scala 2.12.15
Spark version 3.3.2

[SimFG]

apply the pom.xml diff
milvus-io/milvus-sdk-java@master...SimFG:milvus-sdk-java:master-build-expr

[Jeri-jose]

sure @SimFG Will check it out

[Jeri-jose]

@SimFG I have tried run the quickstart.py script with the build JAR with the changes you suggested in milvus-java-sdk, still i am getting the error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/spark/python/pyspark/sql/readwriter.py", line 966, in save
    self._jwrite.save()
  File "/usr/local/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
  File "/usr/local/spark/python/pyspark/sql/utils.py", line 190, in deco
    return f(*a, **kw)
  File "/usr/local/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o89.save.
: java.lang.NoClassDefFoundError: io/milvus/client/MilvusServiceClient
        at zilliztech.spark.milvus.MilvusConnection$.acquire(MilvusConnection.scala:31)
        at zilliztech.spark.milvus.Milvus.getTable(Milvus.scala:23)
        at org.apache.spark.sql.DataFrameWriter.getTable$1(DataFrameWriter.scala:278)
        at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:296)
        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
        at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassNotFoundException: io.milvus.client.MilvusServiceClient
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 17 more