-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
725ca41
commit 5667222
Showing
1 changed file
with
57 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,57 @@ | ||
## Create mocked data | ||
|
||
To start mocking data to log files and publishing them to Spark, once the dev env is set, | ||
call the following commands. They are bash calls, packed as make commands. Calling the make targets | ||
is the same as copying the commands from the [Makefile](../Makefile). | ||
|
||
|
||
### Note about this project's Makefile | ||
|
||
This project's [Makefile](../Makefile) holds necessary commands to run the instructions os this page, | ||
but it starts from dynamically creating and updating a .env file in the root of this repo | ||
with variables and values loaded from the [env.toml](../env.toml) file. | ||
|
||
In case Makefile's commands are triggered manually, remember to create and update the .env | ||
file so that the expected variables are loaded, or include them to the command, as in: | ||
|
||
```shell | ||
PYTHONPATH=src BIND_PORT=9021 poetry run python -m app | ||
``` | ||
|
||
### Note about containers in this project | ||
|
||
Podman is a community option for docker, and is used in this project. We recommend | ||
[installing podman](https://podman.io/docs/installation), but docker can be used instead. | ||
The Makefile targets must be updated or their commands altered on the command line. | ||
|
||
### Start a temporary Spark service (optional) | ||
|
||
In case there is no Spark service available to work with, the following command will create a local one. | ||
It will create a Kafka broker service in a podman container, from [Apache](https://hub.docker.com/u/apache). | ||
|
||
```shell | ||
make test-start-kafka | ||
``` | ||
|
||
### Set up a Kafka UI service (optional) | ||
|
||
A web interface can be set to check and manage Kafka services. The following command will spin a | ||
podman container with [Kafka UI, by Provectus](https://github.com/provectus/kafka-ui). | ||
|
||
```shell | ||
test-start-kafka-ui | ||
``` | ||
|
||
### Create mocked http and app logs | ||
|
||
From the Makefile target, 97-99 logs will be created per run: | ||
```shell | ||
make test-publish-mocked-logs | ||
``` | ||
|
||
### Publish logs to Kafka | ||
|
||
From the Makefile target, run: | ||
```shell | ||
make test-publish-mocked-logs | ||
``` |