[new release] ocaml-protoc (4 packages) (3.0.0) #24863

c-cube · 2023-11-29T03:02:00Z

Pure OCaml compiler for .proto files

Project page: https://github.com/mransan/ocaml-protoc

CHANGES:

Major version that breaks code for every user. Sorry!

Breaking

Generated code now always fits into 2 files: code
for foo.proto is produced in foo.ml and foo.mli.
This follows a fairly large internal refactor that enables
a plugin architecture internally (which facilitated codegen for
services).
All encoder/decoder functions also have a new name now,
so that they don't collide inside the same file.
CLI flags now use --this-style rather than -this-style
Bump minimal OCaml version to 4.08

Performance

encode probotufs backward, which means we can write directly
into a single buffer (which is not possible in a forward mode
because sub-messages require an unknown amount of prefix space
to write their size as a varint)
add C stubs for pbrt
use Bytes functions to read/write fixed size integers
inlining annotations
more benchmarks, helping optimization overall
reduce allocations drastically, by having the generated code
create fewer closures

Services

add pbrt_services runtime library
generate code for service statements. This is a big feature
for users who want to implement RPC systems using protobuf.
The generated code is agnostic to whatever RPC implementation
will use it, it only packs together RPC method names and path
with the relevant encoders/decoders. Services require
both JSON and binary encoders/decoders to be present.

JSON

migrate pbrt_yojson into the main ocaml-protoc repo
JSON runtime support for empty messages
Add support for bytes in JSON encoding
Add support for encoding and decoding maps in JSON

Other features and fixes

add --make flag to generate make functions that take fewer
optional arguments. This helps preventing the user from
forgetting important arguments when they're encoding to protobuf.
Arguments actually marked as optional in the .proto file
are still optional.
Support maps/lists in options
add Pbrt.Decoder.of_substring
Allow options to be named like (validate.rules).message.required
support code generation for empty messages
Empty proto file is a valid proto as well

Testing

expect-style tests for parser
Tests for option parsing
Test demonstrating parse error printing

c-cube · 2023-11-29T04:55:21Z

This is going to require a lot of new upper bounds on other packages (pbrt < 3.0, ocaml-protoc < 3.0, etc.).

CHANGES: Major version that breaks code for every user. Sorry! ### Breaking - Generated code now always fits into 2 files: code for `foo.proto` is produced in `foo.ml` and `foo.mli`. This follows a fairly large internal refactor that enables a plugin architecture internally (which facilitated codegen for services). All encoder/decoder functions also have a new name now, so that they don't collide inside the same file. - CLI flags now use `--this-style` rather than `-this-style` - Bump minimal OCaml version to 4.08 ### Performance - encode probotufs backward, which means we can write directly into a single buffer (which is not possible in a forward mode because sub-messages require an unknown amount of prefix space to write their size as a varint) - add C stubs for pbrt - use Bytes functions to read/write fixed size integers - inlining annotations - more benchmarks, helping optimization overall - reduce allocations drastically, by having the generated code create fewer closures ### Services - add `pbrt_services` runtime library - generate code for `service` statements. This is a big feature for users who want to implement RPC systems using protobuf. The generated code is agnostic to whatever RPC implementation will use it, it only packs together RPC method names and path with the relevant encoders/decoders. Services require both JSON and binary encoders/decoders to be present. ### JSON - migrate `pbrt_yojson` into the main ocaml-protoc repo - JSON runtime support for empty messages - Add support for bytes in JSON encoding - Add support for encoding and decoding maps in JSON ### Other features and fixes - add `--make` flag to generate `make` functions that take fewer optional arguments. This helps preventing the user from forgetting important arguments when they're encoding to protobuf. Arguments actually marked as `optional` in the .proto file are still optional. - Support maps/lists in options - add Pbrt.Decoder.of_substring - Allow options to be named like `(validate.rules).message.required` - support code generation for empty messages - Empty proto file is a valid proto as well ### Testing - expect-style tests for parser - Tests for option parsing - Test demonstrating parse error printing

haochenx · 2023-12-03T20:53:25Z

This is going to require a lot of new upper bounds on other packages (pbrt < 3.0, ocaml-protoc < 3.0, etc.).

maybe this will do it: c-cube#1

c-cube · 2023-12-04T19:29:38Z

Ah, this looks a lot better!

c-cube · 2023-12-04T19:31:04Z

Is there a way to mark this as incompatible with 32 bit archs?

packages/ocaml-protoc/ocaml-protoc.3.0.0/opam

haochenx · 2023-12-05T01:10:16Z

you probably also need a lower bound for ppx_deriving on the kinetic-client package.

Co-authored-by: Haochen M. Kotoi-Xie <hx@kxc.inc>

c-cube · 2023-12-05T01:45:51Z

I don't know what the kinetic client is, truthfully. Could it instead be a dependency on a recent dune?

haochenx · 2023-12-05T01:55:54Z

Is there a way to mark this as incompatible with 32 bit archs?

Related: ocaml/opam#4633

haochenx · 2023-12-05T01:58:17Z

I don't know what the kinetic client is, truthfully. Could it instead be a dependency on a recent dune?

Ah, I see now that it's just a revdep of your packages. Since it's not directly related to this PR, I don't think you need to bother about it. I can update them later.

Once the CI for the other packages comes green, I think this PR is good to be merged.

haochenx · 2023-12-05T02:51:20Z

Now the only relevant failure is with ppc64. You are free to investigate or simply mark it as unavailable on ppc

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><><><>
Processing  2/3: [ocaml-protoc: dune subst]
+ /home/opam/.opam/opam-init/hooks/sandbox.sh "build" "dune" "subst" (CWD=/home/opam/.opam/5.1/.opam-switch/build/ocaml-protoc.3.0.0)
Processing  2/3: [ocaml-protoc: dune build]
+ /home/opam/.opam/opam-init/hooks/sandbox.sh "build" "dune" "build" "-p" "ocaml-protoc" "-j" "175" "@install" "@runtest" (CWD=/home/opam/.opam/5.1/.opam-switch/build/ocaml-protoc.3.0.0)
- (cd _build/default/src/tests/unit-tests && ./graph_test.exe)
- Graph Test ... Ok
- (cd _build/default/src/tests/unit-tests && ./verify_syntax_invariants.exe)
- Verify syntax invariants ... Ok
- (cd _build/default/src/tests/unit-tests && ./parse_enum.exe)
- Parse Enum ... Ok
- File "src/tests/unit-tests/dune", line 7, characters 14-30:
- 7 |    pbrt_array wrapper_encoding varint)
-                   ^^^^^^^^^^^^^^^^
- (cd _build/default/src/tests/unit-tests && ./wrapper_encoding.exe)
- Command got signal SEGV.
- (cd _build/default/src/tests/unit-tests && ./parse_message.exe)
- Parse Message ... Ok
- Parse Message ... Ok
- (cd _build/default/src/tests/unit-tests && ./parse_field_options.exe)
- Parse Field Options ... Ok
- (cd _build/default/src/tests/unit-tests && ./pbtt_compile_p1.exe)
- Pbtt Compile P1 ... Ok
- File "src/tests/unit-tests/dune", line 7, characters 31-37:
- 7 |    pbrt_array wrapper_encoding varint)
-                                    ^^^^^^
- (cd _build/default/src/tests/unit-tests && ./varint.exe)
- Command got signal SEGV.
- (cd _build/default/src/tests/unit-tests && ./parse_import.exe)
- Parse Import ... Ok
- (cd _build/default/src/tests/unit-tests && ./pbtt_compile_p2.exe)
- Pbtt Compile P2 ... Ok
- (cd _build/default/src/tests/unit-tests && ./parse_file_options.exe)
- Parse File Options ... Ok
- File "src/examples/dune", line 80, characters 7-18:
- 80 |  (name orgchart_ml)
-             ^^^^^^^^^^^
- (cd _build/default/src/examples && ./orgchart_ml.exe)
- Command got signal SEGV.
- (cd _build/default/src/tests/google_unittest && ./google_unittest.exe)
- Google unittest .... OK
[ERROR] The compilation of ocaml-protoc.3.0.0 failed at "dune build -p ocaml-protoc -j 175 @install @runtest".

#=== ERROR while compiling ocaml-protoc.3.0.0 =================================#
# context              2.2.0~alpha4~dev | linux/ppc64 | ocaml-base-compiler.5.1.0 | pinned(https://github.com/mransan/ocaml-protoc/releases/download/v3.0.0/ocaml-protoc-3.0.0.tbz)
# path                 ~/.opam/5.1/.opam-switch/build/ocaml-protoc.3.0.0
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune build -p ocaml-protoc -j 175 @install @runtest
# exit-code            1
# env-file             ~/.opam/log/ocaml-protoc-7-53a35b.env
# output-file          ~/.opam/log/ocaml-protoc-7-53a35b.out
### output ###
# (cd _build/default/src/tests/unit-tests && ./graph_test.exe)
# Graph Test ... Ok
# (cd _build/default/src/tests/unit-tests && ./verify_syntax_invariants.exe)
# Verify syntax invariants ... Ok
# (cd _build/default/src/tests/unit-tests && ./parse_enum.exe)
# Parse Enum ... Ok
# File "src/tests/unit-tests/dune", line 7, characters 14-30:
# 7 |    pbrt_array wrapper_encoding varint)
#                   ^^^^^^^^^^^^^^^^
# (cd _build/default/src/tests/unit-tests && ./wrapper_encoding.exe)
# Command got signal SEGV.
# (cd _build/default/src/tests/unit-tests && ./parse_message.exe)
# Parse Message ... Ok
# Parse Message ... Ok
# (cd _build/default/src/tests/unit-tests && ./parse_field_options.exe)
# Parse Field Options ... Ok
# (cd _build/default/src/tests/unit-tests && ./pbtt_compile_p1.exe)
# Pbtt Compile P1 ... Ok
# File "src/tests/unit-tests/dune", line 7, characters 31-37:
# 7 |    pbrt_array wrapper_encoding varint)
#                                    ^^^^^^
# (cd _build/default/src/tests/unit-tests && ./varint.exe)
# Command got signal SEGV.
# (cd _build/default/src/tests/unit-tests && ./parse_import.exe)
# Parse Import ... Ok
# (cd _build/default/src/tests/unit-tests && ./pbtt_compile_p2.exe)
# Pbtt Compile P2 ... Ok
# (cd _build/default/src/tests/unit-tests && ./parse_file_options.exe)
# Parse File Options ... Ok
# File "src/examples/dune", line 80, characters 7-18:
# 80 |  (name orgchart_ml)
#             ^^^^^^^^^^^
# (cd _build/default/src/examples && ./orgchart_ml.exe)
# Command got signal SEGV.
# (cd _build/default/src/tests/google_unittest && ./google_unittest.exe)
# Google unittest .... OK

packages/ocaml-protoc/ocaml-protoc.3.0.0/opam

Co-authored-by: Haochen M. Kotoi-Xie <hx@kxc.inc>

packages/ocaml-protoc/ocaml-protoc.3.0.0/opam

packages/pbrt/pbrt.3.0.0/opam

packages/pbrt_services/pbrt_services.3.0.0/opam

packages/pbrt_yojson/pbrt_yojson.3.0.0/opam

c-cube · 2023-12-11T14:34:17Z

Seems like all remaining failures are unrelated!

mseri · 2023-12-11T14:40:42Z

Thanks!

c-cube · 2023-12-11T14:48:47Z

Thank you!

c-cube force-pushed the release-ocaml-protoc-v3.0.0 branch 4 times, most recently from 9715af9 to 70bda27 Compare November 30, 2023 02:03

c-cube force-pushed the release-ocaml-protoc-v3.0.0 branch from 70bda27 to 9c18c1a Compare November 30, 2023 02:37

haochenx added the needs reporter action label Dec 3, 2023

add upper bounds for ocaml-protoc revdeps

bf0f01b

github-actions bot removed the needs reporter action label Dec 4, 2023

haochenx reviewed Dec 5, 2023

View reviewed changes

packages/ocaml-protoc/ocaml-protoc.3.0.0/opam Show resolved Hide resolved

haochenx added the needs reporter action label Dec 5, 2023

Update packages/ocaml-protoc/ocaml-protoc.3.0.0/opam

c497de9

Co-authored-by: Haochen M. Kotoi-Xie <hx@kxc.inc>

github-actions bot removed the needs reporter action label Dec 5, 2023

haochenx reviewed Dec 5, 2023

View reviewed changes

packages/ocaml-protoc/ocaml-protoc.3.0.0/opam Outdated Show resolved Hide resolved

haochenx added the needs reporter action label Dec 5, 2023

Update packages/ocaml-protoc/ocaml-protoc.3.0.0/opam

fac1fb7

Co-authored-by: Haochen M. Kotoi-Xie <hx@kxc.inc>

mseri reviewed Dec 6, 2023

View reviewed changes

packages/ocaml-protoc/ocaml-protoc.3.0.0/opam Outdated Show resolved Hide resolved

mseri reviewed Dec 6, 2023

View reviewed changes

packages/pbrt/pbrt.3.0.0/opam Outdated Show resolved Hide resolved

mseri reviewed Dec 6, 2023

View reviewed changes

packages/pbrt_services/pbrt_services.3.0.0/opam Outdated Show resolved Hide resolved

mseri reviewed Dec 6, 2023

View reviewed changes

packages/pbrt_yojson/pbrt_yojson.3.0.0/opam Outdated Show resolved Hide resolved

Apply suggestions from code review

0c81c93

mseri added consider for merge and removed needs reporter action labels Dec 6, 2023

mseri merged commit 18a778e into ocaml:master Dec 11, 2023

c-cube deleted the release-ocaml-protoc-v3.0.0 branch December 11, 2023 14:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[new release] ocaml-protoc (4 packages) (3.0.0) #24863

[new release] ocaml-protoc (4 packages) (3.0.0) #24863

c-cube commented Nov 29, 2023

c-cube commented Nov 29, 2023

haochenx commented Dec 3, 2023

c-cube commented Dec 4, 2023

c-cube commented Dec 4, 2023

haochenx commented Dec 5, 2023

c-cube commented Dec 5, 2023

haochenx commented Dec 5, 2023

haochenx commented Dec 5, 2023

haochenx commented Dec 5, 2023

c-cube commented Dec 11, 2023

mseri commented Dec 11, 2023

c-cube commented Dec 11, 2023

[new release] ocaml-protoc (4 packages) (3.0.0) #24863

[new release] ocaml-protoc (4 packages) (3.0.0) #24863

Conversation

c-cube commented Nov 29, 2023

CHANGES:

Breaking

Performance

Services

JSON

Other features and fixes

Testing

c-cube commented Nov 29, 2023

haochenx commented Dec 3, 2023

c-cube commented Dec 4, 2023

c-cube commented Dec 4, 2023

haochenx commented Dec 5, 2023

c-cube commented Dec 5, 2023

haochenx commented Dec 5, 2023

haochenx commented Dec 5, 2023

haochenx commented Dec 5, 2023

c-cube commented Dec 11, 2023

mseri commented Dec 11, 2023

c-cube commented Dec 11, 2023