-
-
Notifications
You must be signed in to change notification settings - Fork 287
Catch Throwable in ExtraProtobufGenerator #1630
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Merged
simuons
merged 1 commit into
bazel-contrib:master
from
mbland:bzlmod-extraprotobufgenerator-catch-throwable
Oct 24, 2024
Merged
Catch Throwable in ExtraProtobufGenerator #1630
simuons
merged 1 commit into
bazel-contrib:master
from
mbland:bzlmod-extraprotobufgenerator-catch-throwable
Oct 24, 2024
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added a `catch` case for `Throwable` in `handleCodeGeneratorRequest`. Part of bazel-contrib#1482. Not catching this caused `ProtoScalaPBRule` workers to hang when building `//test/...` after bumping com_google_protobuf from 25.5 to 27.5. With this change, we can now see the error: ```txt $ USE_BAZEL_VERSION=7.3.2 bazel build //test/... ERROR: .../test/proto/BUILD:165:14: ProtoScalaPBRule test/proto/standalone_proto_strip_import_prefix_package_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //test/proto:standalone_proto_strip_import_prefix_package) bazel-bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$MessageOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedMessageDescriptor.messageOptions(DescriptorImplicits.scala:532) at scalapb.compiler.DescriptorImplicits$ExtendedMessageDescriptor.sealedOneOfExtendsCount(DescriptorImplicits.scala:578) at scalapb.compiler.ProtoValidation.validateMessage(ProtoValidation.scala:107) at scalapb.compiler.ProtoValidation.$anonfun$validateFile$2(ProtoValidation.scala:17) at scalapb.compiler.ProtoValidation.$anonfun$validateFile$2$adapted(ProtoValidation.scala:17) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at scalapb.compiler.ProtoValidation.validateFile(ProtoValidation.scala:17) at scalapb.compiler.ProtoValidation.$anonfun$validateFiles$1(ProtoValidation.scala:10) at scalapb.compiler.ProtoValidation.$anonfun$validateFiles$1$adapted(ProtoValidation.scala:10) at scala.collection.immutable.Stream.foreach(Stream.scala:533) at scalapb.compiler.ProtoValidation.validateFiles(ProtoValidation.scala:10) at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.handleCodeGeneratorRequest(ExtraProtobufGenerator.scala:65) at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:47) at protocbridge.frontend.PluginFrontend$.$anonfun$runWithBytes$1(PluginFrontend.scala:51) at scala.util.Try$.apply(Try.scala:213) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:51) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:121) at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$2(PosixPluginFrontend.scala:40) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87) at scala.concurrent.package$.blocking(package.scala:146) at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$1(PosixPluginFrontend.scala:38) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../test/proto/BUILD:165:14 scala @//test/proto:standalone_proto_strip_import_prefix_package failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //test/proto:standalone_proto_strip_import_prefix_package) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` The fix for this is to bump ScalaPB jar versions to 1.0.0-alpha.1, which will come in a future change.
simuons
approved these changes
Oct 24, 2024
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Oct 29, 2024
This catches errors that happen at any point in `ExtraProtobufGenerator.run`, not just within `handleCodeGeneratorRequest`. This, in turn, prevents more cases of `ProtoScalaPBRule` aspect workers hanging. Part of bazel-contrib#1482. As I started experimenting with updating gRPC and ScalaPB libs, I bumped the ScalaPB libs to 1.0.0-alpha.1 to try to resolve a problem. The `ProtoScalaPBRule` workers then started hanging again. After pulling the `catch ... Throwable` block from bazel-contrib#1630 up into `ExtraProtobufGenerator.run`, I got the following stack trace: ```txt $ bazel build //test/proto/... ERROR: .../external/com_google_protobuf/BUILD.bazel:334:14: ProtoScalaPBRule external/com_google_protobuf/wrappers_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing command (from target @com_google_protobuf//:wrappers_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.IllegalAccessError: class scalapb.options.Scalapb$ScalaPbOptions tried to access method 'com.google.protobuf.LazyStringArrayList com.google.protobuf.LazyStringArrayList.emptyList()' (scalapb.options.Scalapb$ScalaPbOptions and com.google.protobuf.LazyStringArrayList are in unnamed module of loader 'app') at scalapb.options.Scalapb$ScalaPbOptions.<init>(Scalapb.java:5021) at scalapb.options.Scalapb$ScalaPbOptions.<clinit>(Scalapb.java:11165) at scalapb.options.Scalapb.<clinit>(Scalapb.java:24184) at scalapb.options.compiler.Scalapb$.registerAllExtensions(Scalapb.scala:8) at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:48) at protocbridge.frontend.PluginFrontend$.$anonfun$runWithBytes$1(PluginFrontend.scala:51) at scala.util.Try$.apply(Try.scala:213) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:51) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:121) at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$2(PosixPluginFrontend.scala:40) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87) at scala.concurrent.package$.blocking(package.scala:146) at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$1(PosixPluginFrontend.scala:38) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/BUILD.bazel:334:14 scala @com_google_protobuf//:wrappers_proto failed: (Exit 1): scalapb_worker failed: error executing command (from target @com_google_protobuf//:wrappers_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` Bumping to protobuf v28.2 resolved the issue, at the expense of sacrificing Bazel 6 support, as v21.7 was the last to support Bazel 6. But this issue is orthogonal to the original gRPC test failure issue.
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Oct 29, 2024
This catches errors that happen at any point in `ExtraProtobufGenerator.run`, not just within `handleCodeGeneratorRequest`. This, in turn, prevents more cases of `ProtoScalaPBRule` aspect workers hanging. Part of bazel-contrib#1482. As I started experimenting with updating gRPC and ScalaPB libs, I bumped the ScalaPB libs to 1.0.0-alpha.1 to try to resolve a problem. The `ProtoScalaPBRule` workers then started hanging again. After pulling the `catch ... Throwable` block from bazel-contrib#1630 up into `ExtraProtobufGenerator.run`, I got the following stack trace: ```txt $ bazel build //test/proto/... ERROR: .../external/com_google_protobuf/BUILD.bazel:334:14: ProtoScalaPBRule external/com_google_protobuf/wrappers_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing command (from target @com_google_protobuf//:wrappers_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.IllegalAccessError: class scalapb.options.Scalapb$ScalaPbOptions tried to access method 'com.google.protobuf.LazyStringArrayList com.google.protobuf.LazyStringArrayList.emptyList()' (scalapb.options.Scalapb$ScalaPbOptions and com.google.protobuf.LazyStringArrayList are in unnamed module of loader 'app') at scalapb.options.Scalapb$ScalaPbOptions.<init>(Scalapb.java:5021) at scalapb.options.Scalapb$ScalaPbOptions.<clinit>(Scalapb.java:11165) at scalapb.options.Scalapb.<clinit>(Scalapb.java:24184) at scalapb.options.compiler.Scalapb$.registerAllExtensions(Scalapb.scala:8) at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:48) at protocbridge.frontend.PluginFrontend$.$anonfun$runWithBytes$1(PluginFrontend.scala:51) at scala.util.Try$.apply(Try.scala:213) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:51) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:121) at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$2(PosixPluginFrontend.scala:40) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87) at scala.concurrent.package$.blocking(package.scala:146) at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$1(PosixPluginFrontend.scala:38) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/BUILD.bazel:334:14 scala @com_google_protobuf//:wrappers_proto failed: (Exit 1): scalapb_worker failed: error executing command (from target @com_google_protobuf//:wrappers_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` Bumping to protobuf v28.2 resolved the issue, at the expense of sacrificing Bazel 6 support, as v21.7 was the last to support Bazel 6. But this issue is orthogonal to the original gRPC test failure issue, which I'll address in a future change.
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Nov 1, 2024
This catches errors that happen at any point in `ExtraProtobufGenerator.run`, not just within `handleCodeGeneratorRequest`. This, in turn, prevents more cases of `ProtoScalaPBRule` aspect workers hanging. Part of bazel-contrib#1482. As I started experimenting with updating gRPC and ScalaPB libs, I bumped the ScalaPB libs to 1.0.0-alpha.1 to try to resolve a problem. The `ProtoScalaPBRule` workers then started hanging again. After pulling the `catch ... Throwable` block from bazel-contrib#1630 up into `ExtraProtobufGenerator.run`, I got the following stack trace: ```txt $ bazel build //test/proto/... ERROR: .../external/com_google_protobuf/BUILD.bazel:334:14: ProtoScalaPBRule external/com_google_protobuf/wrappers_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing command (from target @com_google_protobuf//:wrappers_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.IllegalAccessError: class scalapb.options.Scalapb$ScalaPbOptions tried to access method 'com.google.protobuf.LazyStringArrayList com.google.protobuf.LazyStringArrayList.emptyList()' (scalapb.options.Scalapb$ScalaPbOptions and com.google.protobuf.LazyStringArrayList are in unnamed module of loader 'app') at scalapb.options.Scalapb$ScalaPbOptions.<init>(Scalapb.java:5021) at scalapb.options.Scalapb$ScalaPbOptions.<clinit>(Scalapb.java:11165) at scalapb.options.Scalapb.<clinit>(Scalapb.java:24184) at scalapb.options.compiler.Scalapb$.registerAllExtensions(Scalapb.scala:8) at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:48) at protocbridge.frontend.PluginFrontend$.$anonfun$runWithBytes$1(PluginFrontend.scala:51) at scala.util.Try$.apply(Try.scala:213) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:51) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:121) at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$2(PosixPluginFrontend.scala:40) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87) at scala.concurrent.package$.blocking(package.scala:146) at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$1(PosixPluginFrontend.scala:38) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/BUILD.bazel:334:14 scala @com_google_protobuf//:wrappers_proto failed: (Exit 1): scalapb_worker failed: error executing command (from target @com_google_protobuf//:wrappers_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` Bumping to protobuf v28.2 resolved the issue, at the expense of sacrificing Bazel 6 support, as v21.7 was the last to support Bazel 6. But this issue is orthogonal to the original gRPC test failure issue.
simuons
pushed a commit
that referenced
this pull request
Nov 4, 2024
This catches errors that happen at any point in `ExtraProtobufGenerator.run`, not just within `handleCodeGeneratorRequest`. This, in turn, prevents more cases of `ProtoScalaPBRule` aspect workers hanging. Part of #1482. As I started experimenting with updating gRPC and ScalaPB libs, I bumped the ScalaPB libs to 1.0.0-alpha.1 to try to resolve a problem. The `ProtoScalaPBRule` workers then started hanging again. After pulling the `catch ... Throwable` block from #1630 up into `ExtraProtobufGenerator.run`, I got the following stack trace: ```txt $ bazel build //test/proto/... ERROR: .../external/com_google_protobuf/BUILD.bazel:334:14: ProtoScalaPBRule external/com_google_protobuf/wrappers_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing command (from target @com_google_protobuf//:wrappers_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.IllegalAccessError: class scalapb.options.Scalapb$ScalaPbOptions tried to access method 'com.google.protobuf.LazyStringArrayList com.google.protobuf.LazyStringArrayList.emptyList()' (scalapb.options.Scalapb$ScalaPbOptions and com.google.protobuf.LazyStringArrayList are in unnamed module of loader 'app') at scalapb.options.Scalapb$ScalaPbOptions.<init>(Scalapb.java:5021) at scalapb.options.Scalapb$ScalaPbOptions.<clinit>(Scalapb.java:11165) at scalapb.options.Scalapb.<clinit>(Scalapb.java:24184) at scalapb.options.compiler.Scalapb$.registerAllExtensions(Scalapb.scala:8) at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:48) at protocbridge.frontend.PluginFrontend$.$anonfun$runWithBytes$1(PluginFrontend.scala:51) at scala.util.Try$.apply(Try.scala:213) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:51) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:121) at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$2(PosixPluginFrontend.scala:40) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87) at scala.concurrent.package$.blocking(package.scala:146) at protocbridge.frontend.PosixPluginFrontend$.$anonfun$prepare$1(PosixPluginFrontend.scala:38) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/BUILD.bazel:334:14 scala @com_google_protobuf//:wrappers_proto failed: (Exit 1): scalapb_worker failed: error executing command (from target @com_google_protobuf//:wrappers_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` Bumping to protobuf v28.2 resolved the issue, at the expense of sacrificing Bazel 6 support, as v21.7 was the last to support Bazel 6. But this issue is orthogonal to the original gRPC test failure issue, which I'll address in a future change.
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Dec 18, 2024
This version contains this update to `protocbridge.frontend.PluginFrontend.runWithInputStream()`: - https://github.com/scalapb/protoc-bridge/blob/c574d50eaee5b800fd54493fe25c4e0eed3b9def/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 This change originally came from: - https://github.com/scalapb/protoc-bridge/blob/d0d56f635d13f7efaa2755ed0d2d66bdef18b588/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 - scalapb/protoc-bridge#367 This should close scalapb/ScalaPB#1771 and obviate the need for the `try`/`catch` blocks added in bazel-contrib#1630 and bazel-contrib#1637, and the `ScalaPBCodeGenerator` implementation added in bazel-contrib#1648. The update requires special handling in `scripts/create_repository.py` to ensure that only Scala 3.3 and up receive the `protoc-bridge_3` version. Scala 3.1 and 3.2 still receive the `protoc-bridge_2.13` version. However, the `protoc-bridge_3` version currently fails with the following (setting `RULES_SCALA_TEST_ONLY="test_scala_version 3.3.4"` fails the same way): ```txt $ RULES_SCALA_TEST_ONLY="test_scala_version 3.6.2" \ ./test_thirdparty_version.sh ERROR: third_party/test/proto/BUILD.bazel:4:14: ProtoScalaPBRule third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --scala_out: java.lang.NoClassDefFoundError: Could not initialize class scalapb.ScalaPbCodeGenerator$ at scripts.ScalaPbCodeGenerator$.process(ScalaPbCodeGeneratorWrapper.scala:8) at protocgen.CodeGenApp.run(CodeGenApp.scala:48) at protocgen.CodeGenApp.run$(CodeGenApp.scala:41) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocgen.CodeGenApp.run(CodeGenApp.scala:33) at protocgen.CodeGenApp.run$(CodeGenApp.scala:32) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$1(SocketBasedPluginFrontend.scala:31) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:37) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:60) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:71) at scala.concurrent.package$.blocking(package.scala:124) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1(SocketBasedPluginFrontend.scala:37) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:38) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:27) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: third_party/test/proto/BUILD.bazel:4:14 Building source jar third_party/test/proto/proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Dec 19, 2024
This required updates to `scripts/create_repository.py` to accommodate comparisons between `_2.13` and `_3` versions of ScalaPB artifacts. The `_3` versions of the most recent ScalaPB artifacts are only available for Scala 3.3 and later, and `compilerplugin_3` currently still depends on `protoc-gen_2.13`. The `MavenCoordinates` dataclass in `scripts/create_repository.py` has new `unversioned_artifact`, `scala_version`, and `artifact_name` members. Updates to `MavenCoordinates.is_newer_than()` now compare objects with matching `artifact_name` values, and take the `scala_version` into account. These changes then precipitated: - Replacing the `artifact_name()` call with `artifact_name` accesses. - Replacing `ArtifactLabelMaker._remove_scala_version_suffix` with `MavenCoordinates.unversioned_artifact`. - Extracting `__compare_versions` from `is_newer_than` to reuse the same code for comparing Scala versions and artifact versions. (Properly reversed the use of `lhs` and `rhs` in the process.) This change also removes an unnecessary `try`/`catch` block from `scripts.ScalaPbCodeGenerator.process()` for Scala 2.12 and above. The block is still required for the Scala 2.11 implementation of `scripts.ScalaPbCodeGenerator.run()` and in `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run()`. Finally, this change adds notes in a few places to indicate where support remains for Scala 2.11. This supporting code can ultimately be removed if we ever decide to drop Scala 2.11 support. --- Tested by setting the `protobuf` dependencies to a versions I know will break `ScalaPB` 0.11.17: - `abseil-cpp`: 20220623.1 => 20240722.0 - `protobuf`: v21.7 => v26.1 I temporarily removed the `try`/`catch` block for `Throwable` from `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`. I then ran `bazel shutdown` to stop persistent `ProtoScalaPBRule` workers. After that, building with Bazel 7.4.1 (since Bazel 6.5.0 can't build `abseil-cpp` 20240722.0 without setting C++14 compiler flags) crashed as expected, instead of hanging. Undoing the `abseil-cpp` and `protobuf` changes and building then produced a working build. ```txt $ USE_BAZEL_VERSION=7.4.1 bazel build //third_party/test/proto:scala [ ...snip... ] ERROR: third_party/test/proto/BUILD.bazel:4:14: ProtoScalaPBRule third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.VerifyError: Cannot inherit from final class at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1022) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:711) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:706) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:719) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at scalapb.options.Scalapb.<clinit>(Scalapb.java:24835) at scalapb.options.compiler.Scalapb$.registerAllExtensions(Scalapb.scala:8) at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:52) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113) at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$2(SocketBasedPluginFrontend.scala:31) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87) at scala.concurrent.package$.blocking(package.scala:146) at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$1(SocketBasedPluginFrontend.scala:23) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) Target //third_party/test/proto:scala failed to build ERROR: third_party/test/proto/BUILD.bazel:4:14 scala @//third_party/test/proto:proto failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` --- The initial motivation for this change was to try to eliminate the need for the custom `scripts.ScalaPbCodeGenerator` implementation by upgrading to `protoc-bridge` 0.9.8. This version contains a change to `catch` any `Throwable` objects raised by a generator implementation in `protocbridge.frontend.PluginFrontend.runWithInputStream()`. - https://github.com/scalapb/protoc-bridge/blob/c574d50eaee5b800fd54493fe25c4e0eed3b9def/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 This change originally came from: - https://github.com/scalapb/protoc-bridge/blob/d0d56f635d13f7efaa2755ed0d2d66bdef18b588/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 - scalapb/protoc-bridge#367 I closed scalapb/ScalaPB#1771 as a result, and reexamined the `try`/`catch` blocks added in bazel-contrib#1630 and bazel-contrib#1637, and the `scripts.ScalaPBCodeGenerator` implementations added in bazel-contrib#1648. However, the last `protoc-bridge` version to support Scala 2.11 is 0.7.14. Scala 2.11 won't be able to use `protoc-bridge` 0.9.8, so we still need to `catch` any `Throwable`s ourselves. This includes the `catch` blocks from the Scala 2.11 `scripts.ScalaPbCodeGenerator` and `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`. --- In `protoc-bridge` 0.9.7 and 0.9.8, `protocbridge.ProtocCodeGenerator` declares a `run()` method with no implementation (and no `try`/`catch` block). - https://github.com/scalapb/protoc-bridge/blob/v0.9.7/bridge/src/main/scala/protocbridge/ProtocCodeGenerator.scala#L5 In `protoc-gen` 0.9.7 and 0.9.8, `protogen.CodeGenApp` implements `protocbridge.ProtocCodeGenerator`, and `CodeGenApp.run()` wraps a call to `CodeGenApp.process()` inside a `try`/`catch` block. - https://github.com/scalapb/protoc-bridge/blob/v0.9.7/protoc-gen/src/main/scala/protocgen/CodeGenApp.scala#L41-L56 `scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17 extends `CodeGenApp` and implements `process()`. For this version of `ScalaPbCodeGenerator`, available in Scala 2.12 and later, there's no need for a `try`/`catch` wrapper on our end. - https://github.com/scalapb/ScalaPB/blob/v0.11.17/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L11 However, the last available `compilerplugin` version for Scala 2.11 is 0.9.8. `scalapb.ScalaPbCodeGenerator` from that version implements `protocbridge.ProtocCodeGenerator` and has a `try`/`catch` in its `run()` method. However, the `Scalapb.registerAllExtensions(registry)` call on line 14 lies outside this block, producing the crash described in bazel-contrib#1648 (commit 23ae356). This is why we need the Scala 2.11 implementation of `scripts.ScalaPbCodeGenerator`. - https://github.com/scalapb/ScalaPB/blob/v0.9.8/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L14 `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator`, also extends `ProtocCodeGenerator` directly, instead of implementing `CodeGenApp` (so it also builds with Scala 2.11). This is why `ExtraProtobufGenerator.run()` hangs when we remove the `try`/`catch` block and run with `protoc-bridge` 0.9.7 and `protobuf` > v25.5, even with later Scala versions. --- Since `scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17 implements `CodeGenApp`, `scripts.ScalaPbCodeGenerator` for Scala 2.12 should be unnecessary. The only reason we need it is to maintain the same interface as the Scala 2.11 implementation. --- Before the `MavenCoordinates` changes, `scripts/create_repository.py` would ScalaPB flip ScalaPB artifacts between their `_2.13` and `_3` versions on subsequent runs (e.g., `protoc-bridge_2.13` vs. `protoc-bridge_3`). See the comments added within `MavenCoordinates.new()` for details. --- Building Scala 3.3 and later versions with `compilerplugin_3` and `protoc-bridge_3` artifacts (instead of `protoc-bridge_2.13` artifacts) produces the following build failure: ```txt $ RULES_SCALA_TEST_ONLY="test_scala_version 3.6.2" \ ./test_thirdparty_version.sh [ ...snip... ] ERROR: third_party/test/proto/BUILD.bazel:4:14: ProtoScalaPBRule third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --scala_out: java.lang.NoClassDefFoundError: Could not initialize class scalapb.ScalaPbCodeGenerator$ at scripts.ScalaPbCodeGenerator$.process(ScalaPbCodeGeneratorWrapper.scala:8) at protocgen.CodeGenApp.run(CodeGenApp.scala:48) at protocgen.CodeGenApp.run$(CodeGenApp.scala:41) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocgen.CodeGenApp.run(CodeGenApp.scala:33) at protocgen.CodeGenApp.run$(CodeGenApp.scala:32) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$1(SocketBasedPluginFrontend.scala:31) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:37) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:60) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:71) at scala.concurrent.package$.blocking(package.scala:124) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1(SocketBasedPluginFrontend.scala:37) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:38) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:27) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: third_party/test/proto/BUILD.bazel:4:14 Building source jar third_party/test/proto/proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Jan 17, 2025
This required updates to `scripts/create_repository.py` to accommodate comparisons between `_2.13` and `_3` versions of ScalaPB artifacts. The `_3` versions of the most recent ScalaPB artifacts are only available for Scala 3.3 and later, and `compilerplugin_3` currently still depends on `protoc-gen_2.13`. The `MavenCoordinates` dataclass in `scripts/create_repository.py` has new `unversioned_artifact`, `scala_version`, and `artifact_name` members. Updates to `MavenCoordinates.is_newer_than()` now compare objects with matching `artifact_name` values, and take the `scala_version` into account. These changes then precipitated: - Replacing the `artifact_name()` call with `artifact_name` accesses. - Replacing `ArtifactLabelMaker._remove_scala_version_suffix` with `MavenCoordinates.unversioned_artifact`. - Extracting `__compare_versions` from `is_newer_than` to reuse the same code for comparing Scala versions and artifact versions. (Properly reversed the use of `lhs` and `rhs` in the process.) This change also removes an unnecessary `try`/`catch` block from `scripts.ScalaPbCodeGenerator.process()` for Scala 2.12 and above. The block is still required for the Scala 2.11 implementation of `scripts.ScalaPbCodeGenerator.run()` and in `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run()`. Finally, this change adds notes in a few places to indicate where support remains for Scala 2.11. This supporting code can ultimately be removed if we ever decide to drop Scala 2.11 support. --- Tested by setting the `protobuf` dependencies to a versions I know will break `ScalaPB` 0.11.17: - `abseil-cpp`: 20220623.1 => 20240722.0 - `protobuf`: v21.7 => v26.1 I temporarily removed the `try`/`catch` block for `Throwable` from `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`. I then ran `bazel shutdown` to stop persistent `ProtoScalaPBRule` workers. After that, building with Bazel 7.4.1 (since Bazel 6.5.0 can't build `abseil-cpp` 20240722.0 without setting C++14 compiler flags) crashed as expected, instead of hanging. Undoing the `abseil-cpp` and `protobuf` changes and building then produced a working build. ```txt $ USE_BAZEL_VERSION=7.4.1 bazel build //third_party/test/proto:scala [ ...snip... ] ERROR: third_party/test/proto/BUILD.bazel:4:14: ProtoScalaPBRule third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.VerifyError: Cannot inherit from final class at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1022) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:711) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:706) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:719) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at scalapb.options.Scalapb.<clinit>(Scalapb.java:24835) at scalapb.options.compiler.Scalapb$.registerAllExtensions(Scalapb.scala:8) at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:52) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113) at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$2(SocketBasedPluginFrontend.scala:31) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87) at scala.concurrent.package$.blocking(package.scala:146) at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$1(SocketBasedPluginFrontend.scala:23) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) Target //third_party/test/proto:scala failed to build ERROR: third_party/test/proto/BUILD.bazel:4:14 scala @//third_party/test/proto:proto failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` --- The initial motivation for this change was to try to eliminate the need for the custom `scripts.ScalaPbCodeGenerator` implementation by upgrading to `protoc-bridge` 0.9.8. This version contains a change to `catch` any `Throwable` objects raised by a generator implementation in `protocbridge.frontend.PluginFrontend.runWithInputStream()`. - https://github.com/scalapb/protoc-bridge/blob/c574d50eaee5b800fd54493fe25c4e0eed3b9def/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 This change originally came from: - https://github.com/scalapb/protoc-bridge/blob/d0d56f635d13f7efaa2755ed0d2d66bdef18b588/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 - scalapb/protoc-bridge#367 I closed scalapb/ScalaPB#1771 as a result, and reexamined the `try`/`catch` blocks added in bazel-contrib#1630 and bazel-contrib#1637, and the `scripts.ScalaPBCodeGenerator` implementations added in bazel-contrib#1648. However, the last `protoc-bridge` version to support Scala 2.11 is 0.7.14. Scala 2.11 won't be able to use `protoc-bridge` 0.9.8, so we still need to `catch` any `Throwable`s ourselves. This includes the `catch` blocks from the Scala 2.11 `scripts.ScalaPbCodeGenerator` and `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`. --- In `protoc-bridge` 0.9.7 and 0.9.8, `protocbridge.ProtocCodeGenerator` declares a `run()` method with no implementation (and no `try`/`catch` block). - https://github.com/scalapb/protoc-bridge/blob/v0.9.7/bridge/src/main/scala/protocbridge/ProtocCodeGenerator.scala#L5 In `protoc-gen` 0.9.7 and 0.9.8, `protogen.CodeGenApp` implements `protocbridge.ProtocCodeGenerator`, and `CodeGenApp.run()` wraps a call to `CodeGenApp.process()` inside a `try`/`catch` block. - https://github.com/scalapb/protoc-bridge/blob/v0.9.7/protoc-gen/src/main/scala/protocgen/CodeGenApp.scala#L41-L56 `scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17 extends `CodeGenApp` and implements `process()`. For this version of `ScalaPbCodeGenerator`, available in Scala 2.12 and later, there's no need for a `try`/`catch` wrapper on our end. - https://github.com/scalapb/ScalaPB/blob/v0.11.17/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L11 However, the last available `compilerplugin` version for Scala 2.11 is 0.9.8. `scalapb.ScalaPbCodeGenerator` from that version implements `protocbridge.ProtocCodeGenerator` and has a `try`/`catch` in its `run()` method. However, the `Scalapb.registerAllExtensions(registry)` call on line 14 lies outside this block, producing the crash described in bazel-contrib#1648 (commit 23ae356). This is why we need the Scala 2.11 implementation of `scripts.ScalaPbCodeGenerator`. - https://github.com/scalapb/ScalaPB/blob/v0.9.8/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L14 `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator`, also extends `ProtocCodeGenerator` directly, instead of implementing `CodeGenApp` (so it also builds with Scala 2.11). This is why `ExtraProtobufGenerator.run()` hangs when we remove the `try`/`catch` block and run with `protoc-bridge` 0.9.7 and `protobuf` > v25.5, even with later Scala versions. --- Since `scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17 implements `CodeGenApp`, `scripts.ScalaPbCodeGenerator` for Scala 2.12 should be unnecessary. The only reason we need it is to maintain the same interface as the Scala 2.11 implementation. --- Before the `MavenCoordinates` changes, `scripts/create_repository.py` would ScalaPB flip ScalaPB artifacts between their `_2.13` and `_3` versions on subsequent runs (e.g., `protoc-bridge_2.13` vs. `protoc-bridge_3`). See the comments added within `MavenCoordinates.new()` for details. --- Building Scala 3.3 and later versions with `compilerplugin_3` and `protoc-bridge_3` artifacts (instead of `protoc-bridge_2.13` artifacts) produces the following build failure: ```txt $ RULES_SCALA_TEST_ONLY="test_scala_version 3.6.2" \ ./test_thirdparty_version.sh [ ...snip... ] ERROR: third_party/test/proto/BUILD.bazel:4:14: ProtoScalaPBRule third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --scala_out: java.lang.NoClassDefFoundError: Could not initialize class scalapb.ScalaPbCodeGenerator$ at scripts.ScalaPbCodeGenerator$.process(ScalaPbCodeGeneratorWrapper.scala:8) at protocgen.CodeGenApp.run(CodeGenApp.scala:48) at protocgen.CodeGenApp.run$(CodeGenApp.scala:41) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocgen.CodeGenApp.run(CodeGenApp.scala:33) at protocgen.CodeGenApp.run$(CodeGenApp.scala:32) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$1(SocketBasedPluginFrontend.scala:31) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:37) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:60) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:71) at scala.concurrent.package$.blocking(package.scala:124) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1(SocketBasedPluginFrontend.scala:37) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:38) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:27) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: third_party/test/proto/BUILD.bazel:4:14 Building source jar third_party/test/proto/proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Jan 27, 2025
This required updates to `scripts/create_repository.py` to accommodate comparisons between `_2.13` and `_3` versions of ScalaPB artifacts. The `_3` versions of the most recent ScalaPB artifacts are only available for Scala 3.3 and later, and `compilerplugin_3` currently still depends on `protoc-gen_2.13`. The `MavenCoordinates` dataclass in `scripts/create_repository.py` has new `unversioned_artifact`, `scala_version`, and `artifact_name` members. Updates to `MavenCoordinates.is_newer_than()` now compare objects with matching `artifact_name` values, and take the `scala_version` into account. These changes then precipitated: - Replacing the `artifact_name()` call with `artifact_name` accesses. - Replacing `ArtifactLabelMaker._remove_scala_version_suffix` with `MavenCoordinates.unversioned_artifact`. - Extracting `__compare_versions` from `is_newer_than` to reuse the same code for comparing Scala versions and artifact versions. (Properly reversed the use of `lhs` and `rhs` in the process.) This change also removes an unnecessary `try`/`catch` block from `scripts.ScalaPbCodeGenerator.process()` for Scala 2.12 and above. The block is still required for the Scala 2.11 implementation of `scripts.ScalaPbCodeGenerator.run()` and in `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run()`. Finally, this change adds notes in a few places to indicate where support remains for Scala 2.11. This supporting code can ultimately be removed if we ever decide to drop Scala 2.11 support. --- Tested by setting the `protobuf` dependencies to a versions I know will break `ScalaPB` 0.11.17: - `abseil-cpp`: 20220623.1 => 20240722.0 - `protobuf`: v21.7 => v26.1 I temporarily removed the `try`/`catch` block for `Throwable` from `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`. I then ran `bazel shutdown` to stop persistent `ProtoScalaPBRule` workers. After that, building with Bazel 7.4.1 (since Bazel 6.5.0 can't build `abseil-cpp` 20240722.0 without setting C++14 compiler flags) crashed as expected, instead of hanging. Undoing the `abseil-cpp` and `protobuf` changes and building then produced a working build. ```txt $ USE_BAZEL_VERSION=7.4.1 bazel build //third_party/test/proto:scala [ ...snip... ] ERROR: third_party/test/proto/BUILD.bazel:4:14: ProtoScalaPBRule third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.VerifyError: Cannot inherit from final class at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1022) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:711) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:706) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:719) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at scalapb.options.Scalapb.<clinit>(Scalapb.java:24835) at scalapb.options.compiler.Scalapb$.registerAllExtensions(Scalapb.scala:8) at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:52) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113) at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$2(SocketBasedPluginFrontend.scala:31) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87) at scala.concurrent.package$.blocking(package.scala:146) at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$1(SocketBasedPluginFrontend.scala:23) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) Target //third_party/test/proto:scala failed to build ERROR: third_party/test/proto/BUILD.bazel:4:14 scala @//third_party/test/proto:proto failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` --- The initial motivation for this change was to try to eliminate the need for the custom `scripts.ScalaPbCodeGenerator` implementation by upgrading to `protoc-bridge` 0.9.8. This version contains a change to `catch` any `Throwable` objects raised by a generator implementation in `protocbridge.frontend.PluginFrontend.runWithInputStream()`. - https://github.com/scalapb/protoc-bridge/blob/c574d50eaee5b800fd54493fe25c4e0eed3b9def/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 This change originally came from: - https://github.com/scalapb/protoc-bridge/blob/d0d56f635d13f7efaa2755ed0d2d66bdef18b588/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 - scalapb/protoc-bridge#367 I closed scalapb/ScalaPB#1771 as a result, and reexamined the `try`/`catch` blocks added in bazel-contrib#1630 and bazel-contrib#1637, and the `scripts.ScalaPBCodeGenerator` implementations added in bazel-contrib#1648. However, the last `protoc-bridge` version to support Scala 2.11 is 0.7.14. Scala 2.11 won't be able to use `protoc-bridge` 0.9.8, so we still need to `catch` any `Throwable`s ourselves. This includes the `catch` blocks from the Scala 2.11 `scripts.ScalaPbCodeGenerator` and `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`. --- In `protoc-bridge` 0.9.7 and 0.9.8, `protocbridge.ProtocCodeGenerator` declares a `run()` method with no implementation (and no `try`/`catch` block). - https://github.com/scalapb/protoc-bridge/blob/v0.9.7/bridge/src/main/scala/protocbridge/ProtocCodeGenerator.scala#L5 In `protoc-gen` 0.9.7 and 0.9.8, `protogen.CodeGenApp` implements `protocbridge.ProtocCodeGenerator`, and `CodeGenApp.run()` wraps a call to `CodeGenApp.process()` inside a `try`/`catch` block. - https://github.com/scalapb/protoc-bridge/blob/v0.9.7/protoc-gen/src/main/scala/protocgen/CodeGenApp.scala#L41-L56 `scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17 extends `CodeGenApp` and implements `process()`. For this version of `ScalaPbCodeGenerator`, available in Scala 2.12 and later, there's no need for a `try`/`catch` wrapper on our end. - https://github.com/scalapb/ScalaPB/blob/v0.11.17/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L11 However, the last available `compilerplugin` version for Scala 2.11 is 0.9.8. `scalapb.ScalaPbCodeGenerator` from that version implements `protocbridge.ProtocCodeGenerator` and has a `try`/`catch` in its `run()` method. However, the `Scalapb.registerAllExtensions(registry)` call on line 14 lies outside this block, producing the crash described in bazel-contrib#1648 (commit 23ae356). This is why we need the Scala 2.11 implementation of `scripts.ScalaPbCodeGenerator`. - https://github.com/scalapb/ScalaPB/blob/v0.9.8/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L14 `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator`, also extends `ProtocCodeGenerator` directly, instead of implementing `CodeGenApp` (so it also builds with Scala 2.11). This is why `ExtraProtobufGenerator.run()` hangs when we remove the `try`/`catch` block and run with `protoc-bridge` 0.9.7 and `protobuf` > v25.5, even with later Scala versions. --- Since `scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17 implements `CodeGenApp`, `scripts.ScalaPbCodeGenerator` for Scala 2.12 should be unnecessary. The only reason we need it is to maintain the same interface as the Scala 2.11 implementation. --- Before the `MavenCoordinates` changes, `scripts/create_repository.py` would ScalaPB flip ScalaPB artifacts between their `_2.13` and `_3` versions on subsequent runs (e.g., `protoc-bridge_2.13` vs. `protoc-bridge_3`). See the comments added within `MavenCoordinates.new()` for details. --- Building Scala 3.3 and later versions with `compilerplugin_3` and `protoc-bridge_3` artifacts (instead of `protoc-bridge_2.13` artifacts) produces the following build failure: ```txt $ RULES_SCALA_TEST_ONLY="test_scala_version 3.6.2" \ ./test_thirdparty_version.sh [ ...snip... ] ERROR: third_party/test/proto/BUILD.bazel:4:14: ProtoScalaPBRule third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --scala_out: java.lang.NoClassDefFoundError: Could not initialize class scalapb.ScalaPbCodeGenerator$ at scripts.ScalaPbCodeGenerator$.process(ScalaPbCodeGeneratorWrapper.scala:8) at protocgen.CodeGenApp.run(CodeGenApp.scala:48) at protocgen.CodeGenApp.run$(CodeGenApp.scala:41) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocgen.CodeGenApp.run(CodeGenApp.scala:33) at protocgen.CodeGenApp.run$(CodeGenApp.scala:32) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$1(SocketBasedPluginFrontend.scala:31) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:37) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:60) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:71) at scala.concurrent.package$.blocking(package.scala:124) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1(SocketBasedPluginFrontend.scala:37) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:38) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:27) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: third_party/test/proto/BUILD.bazel:4:14 Building source jar third_party/test/proto/proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```
mbland
added a commit
to mbland/rules_scala
that referenced
this pull request
Jan 27, 2025
This required updates to `scripts/create_repository.py` to accommodate comparisons between `_2.13` and `_3` versions of ScalaPB artifacts. The `_3` versions of the most recent ScalaPB artifacts are only available for Scala 3.3 and later, and `compilerplugin_3` currently still depends on `protoc-gen_2.13`. The `MavenCoordinates` dataclass in `scripts/create_repository.py` has new `unversioned_artifact`, `scala_version`, and `artifact_name` members. Updates to `MavenCoordinates.is_newer_than()` now compare objects with matching `artifact_name` values, and take the `scala_version` into account. These changes then precipitated: - Replacing the `artifact_name()` call with `artifact_name` accesses. - Replacing `ArtifactLabelMaker._remove_scala_version_suffix` with `MavenCoordinates.unversioned_artifact`. - Extracting `__compare_versions` from `is_newer_than` to reuse the same code for comparing Scala versions and artifact versions. (Properly reversed the use of `lhs` and `rhs` in the process.) This change also removes an unnecessary `try`/`catch` block from `scripts.ScalaPbCodeGenerator.process()` for Scala 2.12 and above. The block is still required for the Scala 2.11 implementation of `scripts.ScalaPbCodeGenerator.run()` and in `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run()`. Finally, this change adds notes in a few places to indicate where support remains for Scala 2.11. This supporting code can ultimately be removed if we ever decide to drop Scala 2.11 support. --- Tested by setting the `protobuf` dependencies to a versions I know will break `ScalaPB` 0.11.17: - `abseil-cpp`: 20220623.1 => 20240722.0 - `protobuf`: v21.7 => v26.1 I temporarily removed the `try`/`catch` block for `Throwable` from `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`. I then ran `bazel shutdown` to stop persistent `ProtoScalaPBRule` workers. After that, building with Bazel 7.4.1 (since Bazel 6.5.0 can't build `abseil-cpp` 20240722.0 without setting C++14 compiler flags) crashed as expected, instead of hanging. Undoing the `abseil-cpp` and `protobuf` changes and building then produced a working build. ```txt $ USE_BAZEL_VERSION=7.4.1 bazel build //third_party/test/proto:scala [ ...snip... ] ERROR: third_party/test/proto/BUILD.bazel:4:14: ProtoScalaPBRule third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.VerifyError: Cannot inherit from final class at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1022) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:711) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:706) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:719) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at scalapb.options.Scalapb.<clinit>(Scalapb.java:24835) at scalapb.options.compiler.Scalapb$.registerAllExtensions(Scalapb.scala:8) at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:52) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113) at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$2(SocketBasedPluginFrontend.scala:31) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87) at scala.concurrent.package$.blocking(package.scala:146) at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$1(SocketBasedPluginFrontend.scala:23) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) Target //third_party/test/proto:scala failed to build ERROR: third_party/test/proto/BUILD.bazel:4:14 scala @//third_party/test/proto:proto failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` --- The initial motivation for this change was to try to eliminate the need for the custom `scripts.ScalaPbCodeGenerator` implementation by upgrading to `protoc-bridge` 0.9.8. This version contains a change to `catch` any `Throwable` objects raised by a generator implementation in `protocbridge.frontend.PluginFrontend.runWithInputStream()`. - https://github.com/scalapb/protoc-bridge/blob/c574d50eaee5b800fd54493fe25c4e0eed3b9def/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 This change originally came from: - https://github.com/scalapb/protoc-bridge/blob/d0d56f635d13f7efaa2755ed0d2d66bdef18b588/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 - scalapb/protoc-bridge#367 I closed scalapb/ScalaPB#1771 as a result, and reexamined the `try`/`catch` blocks added in bazel-contrib#1630 and bazel-contrib#1637, and the `scripts.ScalaPBCodeGenerator` implementations added in bazel-contrib#1648. However, the last `protoc-bridge` version to support Scala 2.11 is 0.7.14. Scala 2.11 won't be able to use `protoc-bridge` 0.9.8, so we still need to `catch` any `Throwable`s ourselves. This includes the `catch` blocks from the Scala 2.11 `scripts.ScalaPbCodeGenerator` and `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`. --- In `protoc-bridge` 0.9.7 and 0.9.8, `protocbridge.ProtocCodeGenerator` declares a `run()` method with no implementation (and no `try`/`catch` block). - https://github.com/scalapb/protoc-bridge/blob/v0.9.7/bridge/src/main/scala/protocbridge/ProtocCodeGenerator.scala#L5 In `protoc-gen` 0.9.7 and 0.9.8, `protogen.CodeGenApp` implements `protocbridge.ProtocCodeGenerator`, and `CodeGenApp.run()` wraps a call to `CodeGenApp.process()` inside a `try`/`catch` block. - https://github.com/scalapb/protoc-bridge/blob/v0.9.7/protoc-gen/src/main/scala/protocgen/CodeGenApp.scala#L41-L56 `scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17 extends `CodeGenApp` and implements `process()`. For this version of `ScalaPbCodeGenerator`, available in Scala 2.12 and later, there's no need for a `try`/`catch` wrapper on our end. - https://github.com/scalapb/ScalaPB/blob/v0.11.17/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L11 However, the last available `compilerplugin` version for Scala 2.11 is 0.9.8. `scalapb.ScalaPbCodeGenerator` from that version implements `protocbridge.ProtocCodeGenerator` and has a `try`/`catch` in its `run()` method. However, the `Scalapb.registerAllExtensions(registry)` call on line 14 lies outside this block, producing the crash described in bazel-contrib#1648 (commit 23ae356). This is why we need the Scala 2.11 implementation of `scripts.ScalaPbCodeGenerator`. - https://github.com/scalapb/ScalaPB/blob/v0.9.8/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L14 `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator`, also extends `ProtocCodeGenerator` directly, instead of implementing `CodeGenApp` (so it also builds with Scala 2.11). This is why `ExtraProtobufGenerator.run()` hangs when we remove the `try`/`catch` block and run with `protoc-bridge` 0.9.7 and `protobuf` > v25.5, even with later Scala versions. --- Since `scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17 implements `CodeGenApp`, `scripts.ScalaPbCodeGenerator` for Scala 2.12 should be unnecessary. The only reason we need it is to maintain the same interface as the Scala 2.11 implementation. --- Before the `MavenCoordinates` changes, `scripts/create_repository.py` would ScalaPB flip ScalaPB artifacts between their `_2.13` and `_3` versions on subsequent runs (e.g., `protoc-bridge_2.13` vs. `protoc-bridge_3`). See the comments added within `MavenCoordinates.new()` for details. --- Building Scala 3.3 and later versions with `compilerplugin_3` and `protoc-bridge_3` artifacts (instead of `protoc-bridge_2.13` artifacts) produces the following build failure: ```txt $ RULES_SCALA_TEST_ONLY="test_scala_version 3.6.2" \ ./test_thirdparty_version.sh [ ...snip... ] ERROR: third_party/test/proto/BUILD.bazel:4:14: ProtoScalaPBRule third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --scala_out: java.lang.NoClassDefFoundError: Could not initialize class scalapb.ScalaPbCodeGenerator$ at scripts.ScalaPbCodeGenerator$.process(ScalaPbCodeGeneratorWrapper.scala:8) at protocgen.CodeGenApp.run(CodeGenApp.scala:48) at protocgen.CodeGenApp.run$(CodeGenApp.scala:41) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocgen.CodeGenApp.run(CodeGenApp.scala:33) at protocgen.CodeGenApp.run$(CodeGenApp.scala:32) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$1(SocketBasedPluginFrontend.scala:31) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:37) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:60) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:71) at scala.concurrent.package$.blocking(package.scala:124) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1(SocketBasedPluginFrontend.scala:37) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:38) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:27) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: third_party/test/proto/BUILD.bazel:4:14 Building source jar third_party/test/proto/proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```
liucijus
pushed a commit
that referenced
this pull request
Feb 3, 2025
This required updates to `scripts/create_repository.py` to accommodate comparisons between `_2.13` and `_3` versions of ScalaPB artifacts. The `_3` versions of the most recent ScalaPB artifacts are only available for Scala 3.3 and later, and `compilerplugin_3` currently still depends on `protoc-gen_2.13`. The `MavenCoordinates` dataclass in `scripts/create_repository.py` has new `unversioned_artifact`, `scala_version`, and `artifact_name` members. Updates to `MavenCoordinates.is_newer_than()` now compare objects with matching `artifact_name` values, and take the `scala_version` into account. These changes then precipitated: - Replacing the `artifact_name()` call with `artifact_name` accesses. - Replacing `ArtifactLabelMaker._remove_scala_version_suffix` with `MavenCoordinates.unversioned_artifact`. - Extracting `__compare_versions` from `is_newer_than` to reuse the same code for comparing Scala versions and artifact versions. (Properly reversed the use of `lhs` and `rhs` in the process.) This change also removes an unnecessary `try`/`catch` block from `scripts.ScalaPbCodeGenerator.process()` for Scala 2.12 and above. The block is still required for the Scala 2.11 implementation of `scripts.ScalaPbCodeGenerator.run()` and in `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run()`. Finally, this change adds notes in a few places to indicate where support remains for Scala 2.11. This supporting code can ultimately be removed if we ever decide to drop Scala 2.11 support. --- Tested by setting the `protobuf` dependencies to a versions I know will break `ScalaPB` 0.11.17: - `abseil-cpp`: 20220623.1 => 20240722.0 - `protobuf`: v21.7 => v26.1 I temporarily removed the `try`/`catch` block for `Throwable` from `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`. I then ran `bazel shutdown` to stop persistent `ProtoScalaPBRule` workers. After that, building with Bazel 7.4.1 (since Bazel 6.5.0 can't build `abseil-cpp` 20240722.0 without setting C++14 compiler flags) crashed as expected, instead of hanging. Undoing the `abseil-cpp` and `protobuf` changes and building then produced a working build. ```txt $ USE_BAZEL_VERSION=7.4.1 bazel build //third_party/test/proto:scala [ ...snip... ] ERROR: third_party/test/proto/BUILD.bazel:4:14: ProtoScalaPBRule third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.VerifyError: Cannot inherit from final class at java.base/java.lang.ClassLoader.defineClass1(Native Method) at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1022) at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:711) at java.base/jdk.internal.loader.BuiltinClassLoader$4.run(BuiltinClassLoader.java:706) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:719) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) at scalapb.options.Scalapb.<clinit>(Scalapb.java:24835) at scalapb.options.compiler.Scalapb$.registerAllExtensions(Scalapb.scala:8) at scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator$.run(ExtraProtobufGenerator.scala:52) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113) at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$2(SocketBasedPluginFrontend.scala:31) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87) at scala.concurrent.package$.blocking(package.scala:146) at protocbridge.frontend.SocketBasedPluginFrontend.$anonfun$prepare$1(SocketBasedPluginFrontend.scala:23) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) Target //third_party/test/proto:scala failed to build ERROR: third_party/test/proto/BUILD.bazel:4:14 scala @//third_party/test/proto:proto failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ``` --- The initial motivation for this change was to try to eliminate the need for the custom `scripts.ScalaPbCodeGenerator` implementation by upgrading to `protoc-bridge` 0.9.8. This version contains a change to `catch` any `Throwable` objects raised by a generator implementation in `protocbridge.frontend.PluginFrontend.runWithInputStream()`. - https://github.com/scalapb/protoc-bridge/blob/c574d50eaee5b800fd54493fe25c4e0eed3b9def/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 This change originally came from: - https://github.com/scalapb/protoc-bridge/blob/d0d56f635d13f7efaa2755ed0d2d66bdef18b588/bridge/src/main/scala/protocbridge/frontend/PluginFrontend.scala#L107-L122 - scalapb/protoc-bridge#367 I closed scalapb/ScalaPB#1771 as a result, and reexamined the `try`/`catch` blocks added in #1630 and #1637, and the `scripts.ScalaPBCodeGenerator` implementations added in #1648. However, the last `protoc-bridge` version to support Scala 2.11 is 0.7.14. Scala 2.11 won't be able to use `protoc-bridge` 0.9.8, so we still need to `catch` any `Throwable`s ourselves. This includes the `catch` blocks from the Scala 2.11 `scripts.ScalaPbCodeGenerator` and `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator.run`. --- In `protoc-bridge` 0.9.7 and 0.9.8, `protocbridge.ProtocCodeGenerator` declares a `run()` method with no implementation (and no `try`/`catch` block). - https://github.com/scalapb/protoc-bridge/blob/v0.9.7/bridge/src/main/scala/protocbridge/ProtocCodeGenerator.scala#L5 In `protoc-gen` 0.9.7 and 0.9.8, `protogen.CodeGenApp` implements `protocbridge.ProtocCodeGenerator`, and `CodeGenApp.run()` wraps a call to `CodeGenApp.process()` inside a `try`/`catch` block. - https://github.com/scalapb/protoc-bridge/blob/v0.9.7/protoc-gen/src/main/scala/protocgen/CodeGenApp.scala#L41-L56 `scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17 extends `CodeGenApp` and implements `process()`. For this version of `ScalaPbCodeGenerator`, available in Scala 2.12 and later, there's no need for a `try`/`catch` wrapper on our end. - https://github.com/scalapb/ScalaPB/blob/v0.11.17/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L11 However, the last available `compilerplugin` version for Scala 2.11 is 0.9.8. `scalapb.ScalaPbCodeGenerator` from that version implements `protocbridge.ProtocCodeGenerator` and has a `try`/`catch` in its `run()` method. However, the `Scalapb.registerAllExtensions(registry)` call on line 14 lies outside this block, producing the crash described in #1648 (commit 23ae356). This is why we need the Scala 2.11 implementation of `scripts.ScalaPbCodeGenerator`. - https://github.com/scalapb/ScalaPB/blob/v0.9.8/compiler-plugin/src/main/scala/scalapb/ScalaPbCodeGenerator.scala#L14 `scalarules.test.extra_protobuf_generator.ExtraProtobufGenerator`, also extends `ProtocCodeGenerator` directly, instead of implementing `CodeGenApp` (so it also builds with Scala 2.11). This is why `ExtraProtobufGenerator.run()` hangs when we remove the `try`/`catch` block and run with `protoc-bridge` 0.9.7 and `protobuf` > v25.5, even with later Scala versions. --- Since `scalapb.ScalaPbCodeGenerator` from `compilerplugin` 0.11.17 implements `CodeGenApp`, `scripts.ScalaPbCodeGenerator` for Scala 2.12 should be unnecessary. The only reason we need it is to maintain the same interface as the Scala 2.11 implementation. --- Before the `MavenCoordinates` changes, `scripts/create_repository.py` would ScalaPB flip ScalaPB artifacts between their `_2.13` and `_3` versions on subsequent runs (e.g., `protoc-bridge_2.13` vs. `protoc-bridge_3`). See the comments added within `MavenCoordinates.new()` for details. --- Building Scala 3.3 and later versions with `compilerplugin_3` and `protoc-bridge_3` artifacts (instead of `protoc-bridge_2.13` artifacts) produces the following build failure: ```txt $ RULES_SCALA_TEST_ONLY="test_scala_version 3.6.2" \ ./test_thirdparty_version.sh [ ...snip... ] ERROR: third_party/test/proto/BUILD.bazel:4:14: ProtoScalaPBRule third_party/test/proto/proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --scala_out: java.lang.NoClassDefFoundError: Could not initialize class scalapb.ScalaPbCodeGenerator$ at scripts.ScalaPbCodeGenerator$.process(ScalaPbCodeGeneratorWrapper.scala:8) at protocgen.CodeGenApp.run(CodeGenApp.scala:48) at protocgen.CodeGenApp.run$(CodeGenApp.scala:41) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocgen.CodeGenApp.run(CodeGenApp.scala:33) at protocgen.CodeGenApp.run$(CodeGenApp.scala:32) at scripts.ScalaPbCodeGenerator$.run(ScalaPbCodeGeneratorWrapper.scala:5) at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:48) at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:113) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$1(SocketBasedPluginFrontend.scala:31) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:37) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:60) at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:71) at scala.concurrent.package$.blocking(package.scala:124) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$1(SocketBasedPluginFrontend.scala:37) at protocbridge.frontend.SocketBasedPluginFrontend.prepare$$anonfun$adapted$1(SocketBasedPluginFrontend.scala:38) at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:687) at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467) at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:27) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: third_party/test/proto/BUILD.bazel:4:14 Building source jar third_party/test/proto/proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing command (from target //third_party/test/proto:proto) bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Added a
catch
case forThrowable
inhandleCodeGeneratorRequest
. Part of #1482.Motivation
Not catching this caused
ProtoScalaPBRule
workers to hang when building//test/...
after bumpingcom_google_protobuf
from 25.5 to 27.5. With this change, we can now see the error:The fix for this is to bump ScalaPB jar versions to 1.0.0-alpha.1, which will come in a future change.