How to debug the message delivery when transactions fail on the destination chain?

[Nuttymoon]

Context

I am currently experimenting with the AWM relayer + Teleporter on a local development network and I have trouble debugging message delivery on the destination chain as transactions fail.

My setup is the following:

Avalanche network

5 Avalanche nodes with BLS keys registered on a custom network running:

• AvalancheGo v1.10.17
• Subnet EVM v0.5.10
• On Ubuntu 22.04

Note: I had to create a custom network to specify the signer config in the genesis file (as they are not in the genesis_local.json provided in AvalancheGo). Here is an example of the configuration of one node in the initialStakers list:

    {
      "nodeID": "NodeID-7Xhw2mDxuDS44j42TCB6U5579esbSt3Lg",
      "rewardAddress": "X-local18jma8ppw3nhx5r4ap8clazz0dps7rv5u00z96u",
      "delegationFee": 1000000,
      "signer": {
        "publicKey": "0xb4679b7127641d929768cd85825c28153b6ed339e30a38105f0980af6f923feefa715bfa2701722c7e8013ebe44a09b0",
        "proofOfPossession": "0x953ff6d8a1d75abd485054bcda60ad0f254a52b1e83bc7588a39830157d9daa72bf6af13f9bf0da6f2d15df95b0385410b31a993c4c7f2bdbc5faf0ed50f78e1e0078bd97e9ee7b2ed725bb9c730ea674fe0f1d46ce3245b63a1348d9d6e17aa"
      }
    }

The BLS key file is uploaded on each node and specified through the staking-signer-key-file flag.

Subnets and chains

I created 2 Subnets on the network, each containing one Subnet EVM chain with the warp precompile enabled. Here is the genesis data used to create a chain (in YAML):

      config:
        chainId: 66666
        homesteadBlock: 0
        eip150Block: 0
        eip150Hash: "0x2086799aeebeae135c246c65021c82b4e15a2c451340993aacfd2751886514f0"
        eip155Block: 0
        eip158Block: 0
        byzantiumBlock: 0
        constantinopleBlock: 0
        petersburgBlock: 0
        istanbulBlock: 0
        muirGlacierBlock: 0
        subnetEVMTimestamp: 0
        dUpgradeTimestamp: 0
        feeConfig:
          gasLimit: 20000000
          minBaseFee: 1000000000
          targetGas: 100000000
          baseFeeChangeDenominator: 48
          minBlockGasCost: 0
          maxBlockGasCost: 10000000
          targetBlockRate: 2
          blockGasCostStep: 500000
        warpConfig:
          blockTimestamp: 0
          quorumNumerator: 67
      alloc:
        "0x8db97C7cEcE249c2b98bDC0226Cc4C2A57BF52FC":
          balance: "0x52B7D2DCC80CD2E4000000"
        "0xc7C516353602F35deD20Efc72fc87023250Bf704":
          balance: "0x52B7D2DCC80CD2E4000000"
      nonce: "0x0"
      timestamp: "0x0"
      extraData: "0x00"
      gasLimit: "0x1312D00"
      difficulty: "0x0"
      mixHash: "0x0000000000000000000000000000000000000000000000000000000000000000"
      coinbase: "0x0000000000000000000000000000000000000000"
      number: "0x0"
      gasUsed: "0x0"
      parentHash: "0x0000000000000000000000000000000000000000000000000000000000000000"

I can see that the precompile is well-activated through eth_getActivePrecompilesAt:

curl -s -X POST --data '{
    "jsonrpc": "2.0",
    "method": "eth_getActivePrecompilesAt",
    "params": [],
    "id": 1
}' -H 'content-type:application/json;' $CHAIN_01_RPC | jq '.result'
{
  "warpConfig": {
    "blockTimestamp": 0,
    "quorumNumerator": 67
  }
}

Teleporter & AWM relayer

The TeleporterMessenger contract in v0.1.0 has been deployed to both EVM chains at 0x1b0505EaC728Efc30414ce0035BcC4DC5994a563.

I have an AWM relayer in v0.2.8 configured to forward the messages between both chains (bi-directional). Attaching the config file: relayer.json

Messaging smart contracts

To test the setup, I have deployed the MessageSender and MessageReceiver contracts (inspired by the Avalanche Academy Teleporter class):

pragma solidity 0.8.18;

import "@teleporter/ITeleporterMessenger.sol";

/**
 * @dev ExampleCrossChainMessenger is an example contract that demonstrates how to send a messages cross chain.
 */
contract MessageSender {
    ITeleporterMessenger public immutable teleporterMessenger =
        ITeleporterMessenger(0x1b0505EaC728Efc30414ce0035BcC4DC5994a563);

    /**
     * @dev Sends a message to another chain.
     */
    function sendMessage(
        bytes32 destinationBlockchainID,
        address destinationAddress,
        string calldata message
    ) external returns (uint256 messageID) {
        return
            uint256(
                teleporterMessenger.sendCrossChainMessage(
                    TeleporterMessageInput({
                        destinationBlockchainID: destinationBlockchainID,
                        destinationAddress: destinationAddress,
                        feeInfo: TeleporterFeeInfo({
                            feeTokenAddress: address(0),
                            amount: 0
                        }),
                        requiredGasLimit: 100000,
                        allowedRelayerAddresses: new address[](0),
                        message: abi.encode(message)
                    })
                )
            );
    }
}

pragma solidity 0.8.18;

import "@teleporter/ITeleporterMessenger.sol";
import "@teleporter/ITeleporterReceiver.sol";

/**
 * @dev ExampleCrossChainMessenger is an example contract that demonstrates how to send and receive
 * messages cross chain.
 */
contract MessageReceiver is ITeleporterReceiver {
    ITeleporterMessenger public immutable teleporterMessenger =
        ITeleporterMessenger(0x1b0505EaC728Efc30414ce0035BcC4DC5994a563);

    string public lastMessage;

    // Errors
    error Unauthorized();

    /**
     * @dev See {ITeleporterReceiver-receiveTeleporterMessage}.
     *
     * Receives a message from another chain.
     */
    function receiveTeleporterMessage(
        bytes32 sourceBlockchainID,
        address originSenderAddress,
        bytes calldata message
    ) external override {
        // Only the Teleporter receiver can deliver a message.
        if (msg.sender != address(teleporterMessenger)) {
            revert Unauthorized();
        }

        // Store the message.
        lastMessage = abi.decode(message, (string));
    }
}

Sending messages fails

Using this setup, I can generate (send) a Warp message from the first chain using Foundry:

cast send --rpc-url "$CHAIN_01_RPC" \
  --private-key "$EWOQ_PRIVATE_KEY" \
  "$MSG_SENDER_ADDR" \
  'sendMessage(bytes32,address,string)' \
  "$CHAIN_02_HEX" "$MSG_RECEIVER_ADDR" Hello

The transaction is successful (id 0x545776beec10d8a7aaf3cec5ea84da937fb25df7e0c70b3558a0bcf4547cb44e) and picked up by the relayer as seen in the relayer logs:

cat /var/log/avalanche/awm-relayer/awm-relayer.log | grep 545776beec10d8a7aaf3cec5ea84da937fb25df
7e0c70b3558a0bcf4547cb44e
{"level":"info","timestamp":"2024-01-18T13:06:38.316+0100","logger":"awm-relayer","caller":"main/main.go:237","msg":"Handling Teleporter submit message log.","txId":"545776beec10d8a7aaf3cec5ea84da937fb25df7e0c70b3558a0bcf4547cb44e","originChainId":"2n1t4eauuzjmaHRp7X3GcYFDKLmR9ia7ZsFKwTEm6VGm52ZxxJ","sourceAddress":"0x0000000000000000000000001b0505eac728efc30414ce0035bcc4dc5994a563"}

After that, the relayer successfully aggregates signatures and submits a transaction on the destination chain:

cat /var/log/avalanche/awm-relayer/awm-relayer.log | grep 545776beec10d8a7aaf3cec5ea84da937fb25df7e0c70b3558a0bcf4547cb44e -A 20
{"level":"info","timestamp":"2024-01-18T13:06:38.316+0100","logger":"awm-relayer","caller":"main/main.go:237","msg":"Handling Teleporter submit message log.","txId":"545776beec10d8a7aaf3cec5ea84da937fb25df7e0c70b3558a0bcf4547cb44e","originChainId":"2n1t4eauuzjmaHRp7X3GcYFDKLmR9ia7ZsFKwTEm6VGm52ZxxJ","sourceAddress":"0x0000000000000000000000001b0505eac728efc30414ce0035bcc4dc5994a563"}
{"level":"info","timestamp":"2024-01-18T13:06:38.316+0100","logger":"awm-relayer","caller":"relayer/relayer.go:222","msg":"Relaying message","blockchainID":"2n1t4eauuzjmaHRp7X3GcYFDKLmR9ia7ZsFKwTEm6VGm52ZxxJ"}
{"level":"info","timestamp":"2024-01-18T13:06:38.316+0100","logger":"awm-relayer","caller":"relayer/relayer.go:236","msg":"Unpacked warp message","blockchainID":"2n1t4eauuzjmaHRp7X3GcYFDKLmR9ia7ZsFKwTEm6VGm52ZxxJ","warpMessageID":"7gKw3WtxSkjk54rpLqcBcUEtvfLfnAxfPafKuXeVY7AMthh8K"}
{"level":"info","timestamp":"2024-01-18T13:06:38.318+0100","logger":"awm-relayer","caller":"relayer/message_relayer.go:223","msg":"Fetching aggregate signature from the source chain validators via AppRequest"}
{"level":"warn","timestamp":"2024-01-18T13:06:38.321+0100","logger":"awm-relayer","caller":"relayer/message_relayer.go:348","msg":"Failed to make async request to node","nodeID":"NodeID-7Xhw2mDxuDS44j42TCB6U5579esbSt3Lg"}
{"level":"info","timestamp":"2024-01-18T13:06:38.322+0100","logger":"awm-relayer","caller":"peers/external_handler.go:82","msg":"handling app response","from":"NodeID-P7oB2McjBGgW2NXXWVYjV8JEDFoW9xDE5"}
{"level":"info","timestamp":"2024-01-18T13:06:38.322+0100","logger":"awm-relayer","caller":"peers/external_handler.go:82","msg":"handling app response","from":"NodeID-MFrZFVCXPv5iCn6M9K6XduxGTYp891xXZ"}
{"level":"info","timestamp":"2024-01-18T13:06:38.322+0100","logger":"awm-relayer","caller":"peers/external_handler.go:82","msg":"handling app response","from":"NodeID-NFBbbJ4qCmNaCzeW7sxErhvWqvEQMnYcN"}
{"level":"info","timestamp":"2024-01-18T13:06:38.323+0100","logger":"awm-relayer","caller":"peers/external_handler.go:82","msg":"handling app response","from":"NodeID-GWPcbFJZFfZreETSoWjPimr846mXEKCtu"}
{"level":"info","timestamp":"2024-01-18T13:06:38.330+0100","logger":"awm-relayer","caller":"relayer/message_relayer.go:446","msg":"Created signed message.","destinationBlockchainID":"2VqsqXGGXGAiXo3t6bSzifL4RxHGEY3UDba6QXscKXcgmLpggP"}
{"level":"info","timestamp":"2024-01-18T13:06:38.332+0100","logger":"awm-relayer","caller":"teleporter/message_manager.go:195","msg":"Sending message to destination chain","destinationBlockchainID":"2VqsqXGGXGAiXo3t6bSzifL4RxHGEY3UDba6QXscKXcgmLpggP","warpMessageID":"7gKw3WtxSkjk54rpLqcBcUEtvfLfnAxfPafKuXeVY7AMthh8K","teleporterMessageID":"2tZ8wuhHJiEXvHoGWuqgiwjbt8zYGs1xjsmNxbdo7NvNQDHpa3"}
{"level":"info","timestamp":"2024-01-18T13:06:38.336+0100","logger":"awm-relayer","caller":"evm/destination_client.go:185","msg":"Sent transaction","txID":"0x1d28225698c977937ba40a66427364f349b945e1669c55729803c17c93da71cb"}
{"level":"info","timestamp":"2024-01-18T13:06:38.336+0100","logger":"awm-relayer","caller":"teleporter/message_manager.go:244","msg":"Sent message to destination chain","destinationBlockchainID":"2VqsqXGGXGAiXo3t6bSzifL4RxHGEY3UDba6QXscKXcgmLpggP","warpMessageID":"7gKw3WtxSkjk54rpLqcBcUEtvfLfnAxfPafKuXeVY7AMthh8K","teleporterMessageID":"2tZ8wuhHJiEXvHoGWuqgiwjbt8zYGs1xjsmNxbdo7NvNQDHpa3"}
{"level":"info","timestamp":"2024-01-18T13:06:38.336+0100","logger":"awm-relayer","caller":"relayer/message_relayer.go:140","msg":"Finished relaying message to destination chain","destinationBlockchainID":"2VqsqXGGXGAiXo3t6bSzifL4RxHGEY3UDba6QXscKXcgmLpggP"}
{"level":"info","timestamp":"2024-01-18T13:06:43.322+0100","logger":"awm-relayer","caller":"peers/external_handler.go:82","msg":"handling app response","from":"NodeID-7Xhw2mDxuDS44j42TCB6U5579esbSt3Lg"}

Attaching the full log file: awm-relayer.log

When I tried to query the MessageReceiver on the destination chain, I saw that the stored message was unchanged. And looking at the transaction receipt on the destination chain, I see that it is in status 0 = failed:

cast receipt --rpc-url "$CHAIN_02_RPC" "0x1d28225698c977937ba40a66427364f349b945e1669c55729803c17c93da71cb"

blockHash               0x47be7895a183057c23256052b0781242e5988215557e7f90cef08e00344909fc
blockNumber             21
contractAddress         
cumulativeGasUsed       297686
effectiveGasPrice       1000000000
gasUsed                 297686
logs                    []
logsBloom               0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
root                    
status                  0
transactionHash         0x1d28225698c977937ba40a66427364f349b945e1669c55729803c17c93da71cb
transactionIndex        0
type                    2

Question: How can I debug the message delivery on the destination chain further?

From here, I don't really know where to look to understand why the transaction failed and how to fix the issue.