Application with demo of Apache Kafka infrastructure with Apache-Kafka + Schema Registry (Avro) + with Source Connector (MySQL) and Sink Connector (Elasticsearch) + KStream App that join change-stream from MySql and Produced stream and demo of KSQL querying.

Run the App:

docker-compose up -d

Generate purchase inserts into mysql

docker exec mysql /data/generate_purcases.sh

Run App

spring-boot-maven-plugin:run

Elasticsearch + Kibana:

Kibana log board

Kibana dushboard

Run KSQL queries

SET 'auto.offset.reset' = 'earliest'; SHOW TOPICS;

CREATE STREAM PURCHASE WITH (KAFKA_TOPIC='purchase-detail', VALUE_FORMAT='AVRO'); DESCRIBE PURCHASE;

SELECT ID, FIRST_NAME, SECOND_NAME, PRODUCT, PRICE, OCCURRED_AT FROM PURCHASE WHERE PRODUCT LIKE 'Beer%' or 'Wine%' EMIT CHANGES;

SELECT PRODUCT, COUNT(*) AS PURCHASE_COUNT, SUM(CAST(PRICE AS DECIMAL(13,2))) AS TOTAL_VALUE FROM PURCHASE GROUP BY PRODUCT EMIT CHANGES;