Support for IAM + AWS Managed Kafka (MSK)

[brizzbuzz]

Neat project 👋 hopefully I'm not missing something obvious here (I checked the examples folder) but it doesn't seem like there is any way to configure kaf to use IAM in order to connect to an AWS managed kafka cluster.

This would be an incredibly useful feature. I am happy to try my best to contribute :) though I have never written golang beyond a couple tutorials

[birdayz]

How does authentication with MSK work? Anything SASL based? Maybe we can make use of the options we support.

[jonatas-barbosa]

+1 I'm also interested in that. Here's the library that AWS provides for accessing its managed cluster using SASL and IAM - https://github.com/aws/aws-msk-iam-auth.

[birdayz]

Hm..i could not find a go library for it :(

https://github.com/aws/aws-msk-iam-auth/tree/main/src/main/java/software/amazon/msk/auth/iam/internals

It guess it would be possible to implement it. We support SASL, and SASL implementations should be pluggable.
If somebody is interested in doing an implementation, i would welcome this. It could live outside kaf, and we just integrate it here.

[birdayz]

FYI:
there is go code for handling aws iam, on franz-go: https://github.com/twmb/franz-go/blob/master/pkg/sasl/aws/aws.go

we use sarama on kaf, but i think it should be possible to build a wrapper around this for use in kaf/sarama. However, i don't have the time for it - if somebody is interested, i'll be happy to accept a PR.

[bpaquet]

Hello, I have a working version in this branch: https://github.com/bpaquet/kaf/tree/msk_iam

This cannot be merged, because this PR need to be merged before in Sarama.

But in the mean time, you can checkout the branch above, build and enjoy :)

[yermulnik]

Hm..i could not find a go library for it :(

FWIW: https://github.com/aws/aws-msk-iam-sasl-signer-go

[Dawnflash]

#313 addresses this

[benjefferies]

Top effort @Dawnflash. I was just starting to investigate a kaf yesterday and this was my first hurdle. In the middle of testing your fork but initial testing shows it's working well 👍

Thanks for your efforts!