Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Need support on Spark 3.2.x, Spark 3.3.x #217

Open
zhiwilliam opened this issue Dec 10, 2022 · 2 comments
Open

Need support on Spark 3.2.x, Spark 3.3.x #217

zhiwilliam opened this issue Dec 10, 2022 · 2 comments

Comments

@zhiwilliam
Copy link

I found currently, ZparkIO Test cannot work with Spark 3.2.x and Spark 3.3.x

I simply created a project from demo (sbt new leobenkel/zparkio.g8), fixed some issues to make it work. I need to read a delta table so I added delta-core in my sbt file.

When I run the application I found delta-core is not working with 3.1.x spark. And if I upgrade to 3.2.x and run again I found the following issue.

[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar: not found: https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/avro-mapred-1.10.2-hadoop2.jar
[error] (ssExtractDependencies) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2**/avro-mapred-1.10.2-hadoop2.ja**r: not found: https://repo1.maven.org/maven2/org/apache/avro/avro-mapred/1.10.2/**avro-mapred-1.10.2-hadoop2.jar**
[error] Total time: 4 s, completed Dec 10, 2022, 9:37:14 AM

@leobenkel
Copy link
Owner

I will look into it

leobenkel added a commit that referenced this issue Dec 19, 2022
leobenkel added a commit that referenced this issue Dec 19, 2022
* Upgrade to latest versions

Address #217 and #213

* Remove bad options for some scala version

* Simplify ci-cd

* Remove other conflicting scoptions

* Better system for the scalaCoptions

* More fixes

* More fixes

* Try again

* Add sbt cache
@leobenkel
Copy link
Owner

Done. Please confirmed that it is working :)

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

No branches or pull requests

2 participants