consul leader panic: runtime error: when someone joins the leader

[armsby]

I have been trying to run consul in docker on a raspberry pi 4, the leader works fine on its own, but as soon as another instance joins this happens

==> Found address '10.0.4.35' for interface 'eth0', setting bind option...

bootstrap = true: do not enable unless necessary

==> Starting Consul agent...

           Version: 'v1.5.2'

           Node ID: '4489d75b-0ce4-80d4-2d4d-93fe2dc187f8'

         Node name: 'e8db371cb4a4'

        Datacenter: 'dc1' (Segment: '<all>')

            Server: true (Bootstrap: true)

       Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)

      Cluster Addr: 10.0.4.35 (LAN: 8301, WAN: 8302)

           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false


==> Log data will now stream in as it occurs:


    2019/07/07 05:49:37 [INFO]  raft: Initial configuration (index=6): [{Suffrage:Voter ID:4489d75b-0ce4-80d4-2d4d-93fe2dc187f8 Address:10.0.4.34:8300} {Suffrage:Nonvoter ID:bba0b73b-b1fd-84f0-84c3-a290c682a330 Address:10.0.4.31:8300}]

    2019/07/07 05:49:37 [INFO]  raft: Node at 10.0.4.35:8300 [Follower] entering Follower state (Leader: "")

    2019/07/07 05:49:37 [INFO] serf: EventMemberJoin: e8db371cb4a4.dc1 10.0.4.35

    2019/07/07 05:49:37 [INFO] serf: Attempting re-join to previously known node: fdd26e61874a.dc1: 10.0.4.31:8302

    2019/07/07 05:49:37 [INFO] serf: EventMemberJoin: e8db371cb4a4 10.0.4.35

    2019/07/07 05:49:37 [INFO] serf: Attempting re-join to previously known node: fdd26e61874a: 10.0.4.31:8301

    2019/07/07 05:49:37 [INFO] consul: Handled member-join event for server "e8db371cb4a4.dc1" in area "wan"

    2019/07/07 05:49:37 [INFO] consul: Adding LAN server e8db371cb4a4 (Addr: tcp/10.0.4.35:8300) (DC: dc1)

    2019/07/07 05:49:37 [INFO] serf: EventMemberJoin: d7ef7548085c.dc1 10.0.4.34

    2019/07/07 05:49:37 [INFO] serf: EventMemberJoin: fdd26e61874a.dc1 10.0.4.31

    2019/07/07 05:49:37 [INFO] serf: Re-joined to previously known node: fdd26e61874a.dc1: 10.0.4.31:8302

    2019/07/07 05:49:37 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)

    2019/07/07 05:49:37 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)

    2019/07/07 05:49:37 [INFO] consul: Handled member-join event for server "d7ef7548085c.dc1" in area "wan"

    2019/07/07 05:49:37 [INFO] consul: Handled member-join event for server "fdd26e61874a.dc1" in area "wan"

    2019/07/07 05:49:37 [INFO] agent: Started HTTP server on [::]:8500 (tcp)

    2019/07/07 05:49:37 [INFO] agent: started state syncer

==> Consul agent running!

    2019/07/07 05:49:37 [INFO] serf: Attempting re-join to previously known node: d7ef7548085c: 10.0.4.34:8301

    2019/07/07 05:49:40 [WARN] serf: Failed to re-join any previously known node

    2019/07/07 05:49:44 [ERR] agent: failed to sync remote state: No cluster leader

    2019/07/07 05:49:47 [WARN]  raft: Heartbeat timeout from "" reached, starting election

    2019/07/07 05:49:47 [INFO]  raft: Node at 10.0.4.35:8300 [Candidate] entering Candidate state in term 3

    2019/07/07 05:49:47 [INFO]  raft: Election won. Tally: 1

    2019/07/07 05:49:47 [INFO]  raft: Node at 10.0.4.35:8300 [Leader] entering Leader state

    2019/07/07 05:49:47 [INFO]  raft: Added peer bba0b73b-b1fd-84f0-84c3-a290c682a330, starting replication

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12290]


goroutine 40 [running]:

runtime/internal/atomic.goLoad64(0x454062c, 0x466ac00, 0x0)

	/usr/local/go/src/runtime/internal/atomic/atomic_arm.go:127 +0x1c

github.com/hashicorp/raft.(*Raft).replicateTo(0x45c6200, 0x4540600, 0x6, 0x0, 0x0)

	/go/pkg/mod/github.com/hashicorp/raft@v1.1.0/replication.go:205 +0xa8

github.com/hashicorp/raft.(*Raft).replicate(0x45c6200, 0x4540600)

	/go/pkg/mod/github.com/hashicorp/raft@v1.1.0/replication.go:154 +0x344

github.com/hashicorp/raft.(*Raft).startStopReplication.func1()

	/go/pkg/mod/github.com/hashicorp/raft@v1.1.0/raft.go:488 +0x24

github.com/hashicorp/raft.(*raftState).goFunc.func1(0x45c6200, 0x4820d70)

	/go/pkg/mod/github.com/hashicorp/raft@v1.1.0/state.go:146 +0x48

created by github.com/hashicorp/raft.(*raftState).goFunc

	/go/pkg/mod/github.com/hashicorp/raft@v1.1.0/state.go:144 +0x50

Can someone tell me what going wrong as I have not been able to locate the issue

[mkeeler]

@armsby I am going to guess that you are running into the same issue as was fixed here. hashicorp/raft#344

It’s an alignment issue on 32 bit systems.

That fix has not yet made its way into Consul but will soo ln.

[armsby]

yes that looks like the issue, thank you, I will watch out for a release where it is in