Panic in light client db

[colin-axner]

Reported from the cosmos discord channel for IBC

panic: can't open light client database: resource temporarily unavailable

goroutine 411600 [running]:
github.com/cosmos/relayer/relayer.(Chain).MustGetLatestLightHeight(...)
        /home/microtick/relayer/relayer/tm-light-client.go:269
github.com/cosmos/relayer/relayer.(relayMsgRecvPacket).FetchCommitResponse.func1(0x1481000, 0x1)
        /home/microtick/relayer/relayer/relayPackets.go:137 +0x265
github.com/avast/retry-go.Do(0xc00006ee98, 0xc00006ee78, 0x4, 0x4, 0x97, 0x0)
        /home/microtick/go/pkg/mod/github.com/avast/retry-go@v2.6.0+incompatible/retry.go:114 +0x15f
github.com/cosmos/relayer/relayer.(relayMsgRecvPacket).FetchCommitResponse(0xc003114ec0, 0xc000cfdd40, 0xc000cfdc20, 0x0, 0x0)
        /home/microtick/relayer/relayer/relayPackets.go:135 +0x145
github.com/cosmos/relayer/relayer.(NaiveStrategy).sendTxFromEventPackets(0xc000e7f600, 0xc000cfdd40, 0xc000cfdc20, 0xc0054ce880, 0x1, 0x1)
        /home/microtick/relayer/relayer/naive-strategy.go:345 +0x99
github.com/cosmos/relayer/relayer.(*NaiveStrategy).HandleEvents(0xc000e7f600, 0xc000cfdd40, 0xc000cfdc20, 0xc002ffa6c0)
        /home/microtick/relayer/relayer/naive-strategy.go:237 +0x85
created by github.com/cosmos/relayer/relayer.relayerListenLoop
        /home/microtick/relayer/relayer/strategies.go:138 +0xcd1
Disk is not full, and I tried some questionable things like deleting .relayer/light/chain_id.db/LOCK without success

[colin-axner]

Try leaving the relayer running to attempt to reproduce

[alexanderbez]

Good work @colin-axner but this issue still persists. While I think your attempted approach is sound and makes sense, it doesn't resolve the issue at hand. Being so, I think this just alludes to the fact the current relayer architecture and design is not suited for production-grade environments.

I'll re-open this issue along with an ADR for a proposal that I believe will make the relayer more robust and reliable 👍