Possibility to produce avro encoded message to schema regustry powered topic

[eshepelyuk]

Hello

Some tools allow to produce avro encoded messges to topic using schema from schema registry using JSON file as input message.

For example this https://github.com/tchiotludo/akhq

[eshepelyuk]

Hello @birdayz @fabiojmendes

Unfortunately doesn't work for me.
Trying to implement the simplest test, most of the fields are removed from schema for presentation purpose.

1. My schema, successfully registered in schema registry and I'm able to produce to a topic using AKHQ UI

{
  "type": "record",
  "name": "CreateUserProfileWallet",
  "namespace": "Contracts.WalletManager.Commands",
  "fields": [
    {
      "name": "CurrencyCode",
      "type": "string"
    }
  ]
}

2. The JSON file named my.json

{
  "CurrencyCode": "EUR"
}

3. My kaf version

kaf --version
kaf version 0.1.45 (2abe93c6fcda5d87aab2f23b6f993f20806304da)

4. My CLI for producing, my_topic does exist, 25 is my schema id from schema registry

cat my.json | kaf produce --avro-schema-id 25 my_topic
Failed to encode avro
%!(EXTRA *errors.errorString=cannot decode textual record "Contracts.WalletManager.Commands.CreateUserProfileWallet": short buffer)

[fabiojmendes]

Hi @eshepelyuk, kaf expects one record per line by default and it looks like your input spans a few lines. That's why you're getting the parse error from avro.

You can change this behaviour by passing a full to the input-mode flag

--input-mode          -- Scanning input mode: [line|full]

Or you can reformat your json to fit in one line, which has the benefit of allowing you to pass many inputs in a single file.

[eshepelyuk]

Thanks @fabiojmendes
Works now.
Will try more complicated cases later and report if any bugs appear.

Also, what about making schema id flag optional as I asked in PR ?
Since schema can be retrieved from registry by naming convention, i.e.adding -value prefix to the topic name. So it s possible to introspect schema and make publishing easier.

[fabiojmendes]

I'm not sure about making it optional. Since plain text, json and protobuf are also supported, the flag is what tells the kaf that it should encode as avro.

[eshepelyuk]

I didnt mean to remove flag itself (but it also can be removed if you check schema registry before each publish, to detect format).

I meant rather removal of mandatory schema id passing, and either support latest or similar as a version or just have another flag that wil point kaf to use avro format and retrieve latest version of schema for a topic.

The entire goal of this optimization to avoid going to schema registry to check actual id and make kaf doing it for a user.

[eshepelyuk]

@fabiojmendes @birdayz created #195 as followup.