Add an Iroha swarm analogue to Iroha 2

[appetrosyan]

The Iroha swarm tool allows generating docker-compose.yml files for specific numbers of peers.

Having such a tool would be useful for @Mingela and the DevOps team.

Issues/PRs that must be solved in order to close this issue:

• [BUG] Unable to build Kagami with cargo install #3603
• [BUG] Load metadata for docker.io/library/archlinux:base-deve error when trying to build docker compose generated from local repository #3604
• [BUG] Error "Please add account_public_key to the configuration" when trying to build docker compose generated Docker image  #3605
  • TODO: this might be caused by outdated image in Dockerhub. If it works on iroha2-dev branch, it should not be a part of this issue.
• [fix] #3473: Make kagami validator runnable from outside iroha dir #3610
• [suggestion] Separate validator and swarm from Kagami #3833

[antonovmike]

Hi!

[AlexStroke]

Someone problem to fix that is we can't run the LTS branch through docker-compose from the box without additional edits to the image name lts instead dev. I've already been stuck here several times to a finding of a problem when pulled the fresh lts.

[AlexStroke]

Existing developments:
https://github.com/antonovmike/docker_swarm

[appetrosyan]

Make it a subcommand of kagami.

[0x009922]

So, what are the requirements for the tool, and what are the exact problems we need to solve?

My understanding so far about the problems:

• docker-compose.yml in the repository is not up-to-date. For example, it refers to iroha2:dev image while being in iroha2-lts branch and using it's configuration in configs/peer. This desynchronisation leads to compatibility issues.
• We have docker-compose.yml, docker-compose-local.yml and docker-compose-single.yml. They all have not many differences, so it's generally a boilerplate which we also need to synchronise (files with each other). Furthermore, they don't cover use cases when the consumer needs some custom configuration with N peers, not 1 or 4.

Could you desribe more with what @Mingela and DevOps struggle with?

My understanding so far about the solution:

1. The tool might be an interactive prompt CLI or a Web application. It depends on what exactly and how we need to build, I don't understand it yet. Also it depends on how the consumer will work with it, what will be more comfortable.
2. I agree with @appetrosyan that this functionality should be a part of kagami. It will keep all Iroha accompanying tooling in a single place, like cargo or rustup for Rust. Also, we have plans to give kagami GUI, so the first point might disappear on its own.
3. It should accept... what parameters? Only amount of peers to configure? Should it generate random keypairs for those peers?

Help me to clarify things please.

[6r1d]

I agree with Dmitry here on having it as a part of Kagami: otherwise, it may get confusing, and Kagami-gui is planned anyway.

The branch and release should be a parameter too, but not necessarily a required one.
We can assume branches like lts or stable and the last version as our defaults.
But at the same time, users should have a way to answer what their release is. Or we could get the release info from the source and avoid that part of the argument.

The tool might be an interactive prompt CLI or a Web application. It depends on what exactly and how we need to build, I don't understand it yet.

We may have fewer options than Fish making the web config unnecessary. @appetrosyan mentioned that concept: https://lwn.net/Articles/832525/, https://fishshell.com/docs/current/cmds/fish_config.html

[Mingela]

Reference for Iroha v1 swarm, jfyi
https://github.com/kuvaldini/iroha-swarm

[0x009922]

The tool will generate a directory with docker-compose.yml and configurations for peers. This directory may also contain a cloned repo, depending on the arguments.

• source - the main parameter. It might be one of:
  • Dockerhub. Then the argument will be the channel, i.e. either lts, dev or stable. It will set image: hyperledger/iroha2:{channel} in a docker compose file, and will download config files from GitHub at iroha2-{channel} branch.
  • Git. It will simply accept the revision in a form iroha2-lts or <long hash>, clone this revision and will specify build: path/to/local/git/clone in a docker compose file. It will also use configuration files from the cloned repo.
    Maybe it also makes sense to provide a shorthand for Git such as git:lts which will be understood as "Clone hyperledger/iroha at iroha2-lts"
  • Local path. It's almost the same as with Git, but it will use a local path.
• dir - where to put the generated configuration. The tool will check whether it is empty or not (and will recursively create if it doesn't exist). If it is not empty, it will print a warning and exit. Additionally, accept an argument like dir-force-rewrite?
• peers - amount of peers to generate.
  • Is there a need to configure each peer in particular? peer[0].torii_api_port form might be used
• torii_api_port, defaults to 8080
• torii_telemetry_port, defaults to 8081

I am not sure yet how to compose it all in a robust CLI interface, will do more research. Also, all of this could be prompted in an interactive mode.

[appetrosyan]

TBH it doesn’t have to be CLI. I would prefer a CLI tool, but that’s only because it’d be easier for us to maintain.

[6r1d]

I am not sure yet how to compose it all in a robust CLI interface, will do more research

What about a ncurses-based wizard, like https://docs.rs/tui/latest/tui/ at the start?
After that, you can copy the well-known settings into a non-interactive version.

I like the idea of making a new directory and not overwriting the config (although I may imagine batch scenarios when the output directory is a part of the configuration, too).

In general, the concept looks reasonable.

[0x009922]

TBH it doesn’t have to be CLI. I would prefer a CLI tool, but that’s only because it’d be easier for us to maintain.

@appetrosyan, I think it would be easier to implement than building a complete GUI for Kagami. At least, for now. Or should I start working in this direction?

What about a ncurses-based wizard, like https://docs.rs/tui/latest/tui/ at the start?

@6r1d, well... it is a UI already. I think TUI is something between CLI and GUI that might not find its target audience. GUI might a complete "friendly" and "nicely-looking" option instead of a CLI, but will TUI worth the effort?

[appetrosyan]

Let's go with a very basic clap-based CLI. We can extend it further when we need to.

[0x009922]

After understanding more details of the task, I came to the conclusion that making kagami swarm a channel-agnostic tool (i.e. it can work with both dev, stable and lts channels) is bad idea. The reasons:

• Kagami is not channel-agnostic itself. kagami config, kagami genesis, kagami schema etc are all channel-specific and closely coupled with the rest of Iroha codebase.
• Iroha from different channels might need channel-specific configuration. This can be workarounded by fetching configs from GitHub at specified revision, though.
• docker-compose.yml configuration might differ from channel to channel, therefore Kagami needs to maintain all of the options within Kagami. It produces difficult questions I don't want to dive into.

Thus, I came to a different understanding of how Kagami should handle different sources:

• Kagami will not fetch configs & genesis from anywhere, but generate defaults on its own. If the user wants some other options (like when calling kagami genesis --syntetic to generate custom genesis), they might pass --no-default-configuration option and then fill the target directory with custom configs on their own.
  • Generated config & genesis will have null in places when values are passed through ENV in docker-compose.yml
• When dockerhub is specified as a source, dockerhub image is used. The channel of the image is the channel Kagami is built on (I should research for possibility to use some proc-macro to get that info from git during the build).
• When github is specified as a source, the repo from the revision Kagami was built from is cloned and used as a build source for Docker images
• When path is specified as a source, local path to Iroha repo is used for building docker images.

@appetrosyan @6r1d, please let me know what you think.

[6r1d]

a channel-agnostic tool

So, channels are equal to branches, as we've just discussed.

Kagami will not fetch configs & genesis from anywhere but generate defaults on its own

Sounds reasonable.

Generated config & genesis will have null in places when values are passed through ENV in docker-compose.yml

Suppose our goal is to save time for the user by providing defaults. In that case, I expect this to break things further: users won't know anything about environment variables, and Iroha will stop working.

[timofeevmd]

The basic functionality works:
directory with configuration files created
config files copied
docker compose created

but bugs were found @0x009922
#3603
#3604
#3605

[0x009922]

There are bugs and other tasks as separated issues, but this issue might be closed, I think.

upd: so we can close it only after verification, I see..

[appetrosyan]

We also need to merge #3610

[timofeevmd]

#3603 - @0x009922 close with comment
With @appetrosyan and @ilchu, we decided that we don't have a goal to make Kagami runnable outside of the repo root.

[timofeevmd]

new bugs related
#3828
#3829

[0x009922]

After some research and discussions, I came up with the following thoughts.

Decision to abandon Directory Mode

Currently, kagami swarm has two modes: Directory and File.

Developing Directory Mode turned out to be an exercise in futility. Its purpose was to generate a self-contained directory with peers configuration included, offering several options for the Docker image source:

• --image <channel> to fetch a published one from Dockerhub
• --build <path> to specify a path to a local Iroha repo clone
• --build-from-github, essentially the same as --build, but cloning the repo for you

This approach led to some ambiguities, primarily related to the peers configuration. For instance:

• Shall Swarm reuse the pre-built configuration files in the repo's /configs/peer? It can, in case of --build and --build-from-github, but what about --image?
• If Swarm should build configurations on its own, does it need to contain parts of kagami config, kagami genesis, and kagami validator¹, or should it clone the repo and build the necessary configurations?

These questions have resulted in overcomplicated code and bugs. More importantly, there is no current use case for Directory Mode.

Therefore, I propose to drop Directory Mode functionality and retain only File Mode. File Mode offers a more straightforward approach, free from the dependencies and complexities of Directory Mode. It is also used in the repo to generate sample Docker Compose configurations and is integrated into CI.

Transitioning to a separate binary¹

Although File Mode (i.e., kagami swarm file) is the same pure as other Kagami commands, creating a separate iroha_swarm binary is the logical step for the following reasons:

• All other Kagami commands produce their output directly to stdout. However, File Mode must know the output file name in order to resolve a potentially relative build path².
• Future developments might require Swarm to generate something more complex than a single file, justifying its separation.

Example usage

The usage of iroha_swarm will be as follows:

iroha_swarm \
    --build . \
    --peers 5 \
    --seed iroha \
    --config-dir ./configs/peer \
    --outfile docker-compose.dev.yml

Footnotes

1. [suggestion] Separate validator and swarm from Kagami #3833 ↩ ↩²

2. E.g., with build path --build ../my-cloned-iroha and output file --outfile ./put/swarm/here/docker-compose.yml, the resulted file will contain:

   build: ../../../../my-cloned-iroha
   

   ↩