At Notional, we've focused on the operational aspects of relaying, while also making a few contributions to the go relayer and hermes. We are also working on shared security.
NB: The design consideration behind IBC expressways is rpc pressure on node software that uses the cosmos-sdk and tendermint, resulting in subpar IBC performance.
NB: need better graphs
0-141 Bypass
We relay osmosis-1 channel-0 to cosmoshub-4 channel-141 using both hermes and the go relayer. For each, the setup is configured so that Osmosis and Gaia have their own NVMe hard drive. Hermes and the go relayer run locally. This ensures that our other deployments don't need to deal with this one high traffic channel.
Both setups use state sync, so the go relayer does not perform as well.
- Kava
- Terra
- Sif
- Osmosis
- Cro
- Microtick
- E-money
- Juno
- Terra
- Cosmoshub
- Osmosis
- Sif
Other than 0-141 there's an emerging high traffic pattern between Juno, Gaia, Osmosis, Sif and Terra. This system handles all of the traffic between these five chains, and excludes traffic on 0-141.
Cosmos Outer Loop
The cosmos outer loop attempts to service all IBC traffic other than what is described above.
We're documenting our exact configuration. For the relayer software, we run that on small, light computers and in the case of hetzner, that means virtual machines:
https://madflex.de/install-archlinux-on-hetzner-cloud/
We always use arch linux.
We've found over and over that Hetzner is the best option for hosting bare metal servers, and if you're relaying, you're really going to want to run on metal.
We started with big machines:
- Intel or AMD 16 core CPU
- 128GB or more RAM
- 6x SAMSUNG MZQL23T8HCLS-00A07 drives
- Arch Linux
- You can use whichever distribution you'd like but our scripts and systems assume arch linux.
But now we are building bypasses on smaller ones
- AX61-nvme
- AX51-nvme
- AX41-nvme
In any event, it is a very good idea to make sure that you're using NVMe drives, and specifically pcie 4.0x4 NVMe 1.4 drives.
DIY
- CPU with high single thread performance and many PCIe lanes-- threadripper 32 core CPUs have 88 lanes and Threadripper Pro provides 128.
- 128gb RAM
- PCIe Gen4x4 lanes storage (No RAID!)
- Teamgroup a440
- Samsung 980 Pro <- recommended for reliablity and availability. we use these on-site.
- WD SN850 Black <- Very fast
- Optane P5800x <- fastest and most expensive
- Seagate Firecuda 530 <- Supposedly fastest but untested. Comes in a 4gb model.
You need:
- Archive nodes for every chain you want to relay
- Hermes
- Go Relayer