Set configuration for a Zk cluster and broker cluster using groups

[rmarting]

This PR is a proposal to fix the #7 including the extension of create the zookeeper (ZK) and broker (BK) cluster successfuly. The current status of the main branch does not include the right configuration in the zookeeper.properties and server.properties file to set up each cluster. These files must include a set of properties to identify each member of the ZK and BK cluster to establish the connectivy between members and then have a complete topology working as expected.

For example, the current status of the main branch defines some properties as standalone, so it is great, when you want to have a one-to-one topology (one ZK instance and a single BK instance):

• broker.id=0 in the server.properties file
• Missing the zookeeper.connect property in the server.properties file
• Missing the server.X properties in the zookeeper.properties (and other properties to create the cluster)

Checking the ZK Admin Server, the instance is defined as standalone:

[rhmw@f38mw01 ~]$ curl http://localhost:8080/commands/stat
{
  "version" : "3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT",
  "read_only" : false,
  "server_stats" : {
    "packets_sent" : 0,
    "packets_received" : 0,
    "fsync_threshold_exceed_count" : 0,
    "client_response_stats" : {
      "last_buffer_size" : -1,
      "min_buffer_size" : -1,
      "max_buffer_size" : -1
    },
    "uptime" : 76287,
    "data_dir_size" : 457,
    "log_dir_size" : 457,
    "provider_null" : false,
    "last_processed_zxid" : 0,
    "outstanding_requests" : 0,
    "server_state" : "standalone",
    "avg_latency" : 0.0,
    "max_latency" : 0,
    "min_latency" : 0,
    "num_alive_client_connections" : 0,
    "auth_failed_count" : 0,
    "non_mtlsremote_conn_count" : 0,
    "non_mtlslocal_conn_count" : 0
  },
  "client_response" : {
    "last_buffer_size" : -1,
    "min_buffer_size" : -1,
    "max_buffer_size" : -1
  },
  "node_count" : 5,
  "connections" : [ ],
  "secure_connections" : [ ],
  "command" : "stats",
  "error" : null
}

Another way to check that it is not a distributed cluster is creating a topic with more replicas. Then the following exception will raise:

Error while executing topic command : Replication factor: 3 larger than available brokers: 1.
[2023-06-08 23:03:21,771] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.
 (kafka.admin.TopicCommand$)

This PR includes a set of configurations and templates for these files allowing to define the different members of the ZK and BK clusters. Using the zookeeper and brokers groups in the inventory the roles will use them to set up everything in the right way.

After seeting that, it is easy to check the status of a ZK instance using the Admin Server endpoint, and identify the status of each member, as leader or follower. Here an example of a leader node:

[rhmw@f38mw01 zookeeper]$ curl http://localhost:8080/commands/stat
{
  "version" : "3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT",
  "read_only" : false,
  "server_stats" : {
    "packets_sent" : 0,
    "packets_received" : 0,
    "fsync_threshold_exceed_count" : 0,
    "client_response_stats" : {
      "last_buffer_size" : -1,
      "min_buffer_size" : -1,
      "max_buffer_size" : -1
    },
    "provider_null" : false,
    "server_state" : "leader",
    "outstanding_requests" : 0,
    "min_latency" : 0,
    "avg_latency" : 0.0,
    "max_latency" : 0,
    "data_dir_size" : 67114438,
    "log_dir_size" : 67114438,
    "last_processed_zxid" : 12884901888,
    "num_alive_client_connections" : 0,
    "auth_failed_count" : 0,
    "non_mtlsremote_conn_count" : 0,
    "non_mtlslocal_conn_count" : 0,
    "uptime" : 3456
  },
  "client_response" : {
    "last_buffer_size" : -1,
    "min_buffer_size" : -1,
    "max_buffer_size" : -1
  },
  "proposal_stats" : {
    "last_buffer_size" : -1,
    "min_buffer_size" : -1,
    "max_buffer_size" : -1
  },
  "node_count" : 26,
  "connections" : [ ],
  "secure_connections" : [ ],
  "command" : "stats",
  "error" : null
}

Bonus track

This PR also includes a set of properties to enable the ZK Admin Server (including the port). Very useful to check the status of the ZK nodes.

[rmarting]

@rpelisse I have been working in this PR to improve the way to deploy a distribute cluster (following the most common patterns). This is an approach from my older Ansible Playbook created many time ago, but as the initial draft of a more complex one. https://github.com/rmarting/amq-streams-rhel-ansible-playbook

That playbook includes some well configurations and just refactored into this repo, but maybe I am not following the best approaches or common standards of Ansible or this collection. Please, feel free to review, update, comment, or improve it.

IMHO the roles must cover the most common deployment topology of Kafka clusters, that it is using more than one instance per component (usually 3 per ZK, and 2,3,4 or more for BK). I tested for a single case, and it is working too.

Looking forward to hearing your feedback and comments.

[rpelisse]

It looks pretty good and worked right away (or almost) on my laptop! Kudos! Honestly, it's gold, thank you so much :)

The only thing is that, right now, the use must have a group of machine called "zookeepers". We probably need to change that. A default value using this approach, that can be overriden, should do the the trick, but I'll do it in later commit.

[rpelisse]

LGTM

[rmarting]

Thank you so much @rpelisse for your comment and feedback! It is moving to a good shape and position this collection to automate a kafka cluster.

[rmarting]

It looks pretty good and worked right away (or almost) on my laptop! Kudos! Honestly, it's gold, thank you so much :)

The only thing is that, right now, the use must have a group of machine called "zookeepers". We probably need to change that. A default value using this approach, that can be overriden, should do the the trick, but I'll do it in later commit.

Looking forward how you are implementing it! (I will learn more things about best Ansible practices) 😄