Database Corruption Requires Resync

[bytemaster]

2137820ms th_a db_management.cpp:125 open ] last_block->id(): 002400f7711cb44e5d23c06550d38dca546d53f2 last_block->block_num(): 2359543
2137820ms th_a db_management.cpp:133 open ] 10 assert_exception: Assert Exception
head_block_num() == 0: last block ID does not match current chain state
{}
th_a db_management.cpp:128 open
2137820ms th_a db_management.cpp:133 open ] data_dir: /root/bitshares-2/programs/witness_node/witness_node_data_dir/blockchain
2137820ms th_a application.cpp:438 startup ] 10 assert_exception: Assert Exception
head_block_num() == 0: last block ID does not match current chain state
{}
th_a db_management.cpp:128 open
rethrow
{"data_dir":"/root/bitshares-2/programs/witness_node/witness_node_data_dir/blockchain"}
th_a db_management.cpp:133 open
Shutting down a witness node before it receives any blocks results in this error the next time it starts up:

2137820ms th_a application.cpp:984 startup ] 10 assert_exception: Assert Exception
head_block_num() == 0: last block ID does not match current chain state
{}
th_a db_management.cpp:128 open
rethrow
{"data_dir":"/root/bitshares-2/programs/witness_node/witness_node_data_dir/blockchain"}
th_a db_management.cpp:133 open
rethrow
{}
th_a application.cpp:438 startup
2137821ms th_a main.cpp:192 main ] Exiting with error:
10 assert_exception: Assert Exception
head_block_num() == 0: last block ID does not match current chain state
{}
th_a db_management.cpp:128 open
rethrow
{"data_dir":"/root/bitshares-2/programs/witness_node/witness_node_data_dir/blockchain"}
th_a db_management.cpp:133 open
rethrow
{}
th_a application.cpp:438 startup

[theoreticalbts]

If you copy paste stuff like this, please indent it four spaces so it displays properly in Github

[theoreticalbts]

I cannot reproduce this. If you can, please post instructions and/or try the above patch.

[theoreticalbts]

I still cannot reproduce this. Here is what I tried:

$ programs/witness_node/witness_node --data-dir data/issue492 --enable-stale-production
...

1800005ms th_a       witness.cpp:185               block_production_loo ] Generated block #2093 with timestamp 2016-01-04T20:30:00 at time 2016-01-04T20:30:00
1805005ms th_a       witness.cpp:185               block_production_loo ] Generated block #2094 with timestamp 2016-01-04T20:30:05 at time 2016-01-04T20:30:05
1810005ms th_a       witness.cpp:185               block_production_loo ] Generated block #2095 with timestamp 2016-01-04T20:30:10 at time 2016-01-04T20:30:10
1815005ms th_a       witness.cpp:185               block_production_loo ] Generated block #2096 with timestamp 2016-01-04T20:30:15 at time 2016-01-04T20:30:15
1825005ms th_a       witness.cpp:185               block_production_loo ] Generated block #2097 with timestamp 2016-01-04T20:30:25 at time 2016-01-04T20:30:25
1825143ms asio       main.cpp:167                  operator()           ] Caught SIGINT attempting to exit cleanly
1825143ms th_a       main.cpp:180                  main                 ] Exiting from signal 2
1825144ms th_a       thread.cpp:115                ~thread              ] calling quit() on ntp
1825144ms th_a       thread.cpp:160                quit                 ] destroying boost thread 140496209479424
1825144ms ntp        thread.cpp:246                exec                 ] thread canceled: 9 canceled_exception: Canceled
cancellation reason: [none given]
    {"reason":"[none given]"}
    ntp  thread_d.hpp:463 start_next_fiber
1825196ms th_a       thread.cpp:115                ~thread              ] calling quit() on p2p
1825196ms th_a       thread.cpp:160                quit                 ] destroying boost thread 140496190596864
1825196ms p2p        thread.cpp:246                exec                 ] thread canceled: 9 canceled_exception: Canceled
cancellation reason: [none given]
    {"reason":"[none given]"}
    p2p  thread_d.hpp:463 start_next_fiber

$ programs/witness_node/witness_node --data-dir data/issue492
1867189ms th_a       main.cpp:118                  main                 ] Error parsing logging config from config file /home/cc/graphene/src/graphene/data/issue492/config.ini, using default config
1867189ms th_a       witness.cpp:86                plugin_initialize    ] witness plugin:  plugin_initialize() begin
1867189ms th_a       witness.cpp:96                plugin_initialize    ] key_id_to_wif_pair: ["GPH6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV","5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3"]
1867189ms th_a       witness.cpp:114               plugin_initialize    ] witness plugin:  plugin_initialize() end
1867189ms th_a       object_database.cpp:91        open                 ] Opening object database from /home/cc/graphene/src/graphene/data/issue492/blockchain ...
1867210ms th_a       object_database.cpp:97        open                 ] Done opening object database.
1867210ms th_a       db_management.cpp:125         open                 ] last_block->id(): 000008275e5e641ab24a3056dbf20b32f5514084 last_block->block_num(): 2087
1867210ms th_a       thread.cpp:95                 thread               ] name:ntp tid:139859955709696
1867211ms th_a       thread.cpp:95                 thread               ] name:p2p tid:139859936827136
1867212ms th_a       application.cpp:140           reset_p2p_node       ] Configured p2p node to listen on 0.0.0.0:43606
1867212ms th_a       witness.cpp:119               plugin_startup       ] witness plugin:  plugin_startup() begin
1867212ms th_a       witness.cpp:126               plugin_startup       ] Launching block production for 11 witnesses.
1867212ms th_a       witness.cpp:137               plugin_startup       ] witness plugin:  plugin_startup() end
1867212ms th_a       main.cpp:176                  main                 ] Started witness node on a chain with 2087 blocks.
1867212ms th_a       main.cpp:177                  main                 ] Chain ID is c8f0951e680f37ca1e96ccdd835b5b0196a60290b86fbf11229c22b70b7a8965
1868000ms th_a       witness.cpp:188               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
1869000ms th_a       witness.cpp:188               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
1870000ms th_a       witness.cpp:188               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
1871000ms th_a       witness.cpp:188               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
1871772ms asio       main.cpp:167                  operator()           ] Caught SIGINT attempting to exit cleanly
1871772ms th_a       main.cpp:180                  main                 ] Exiting from signal 2
1871773ms th_a       thread.cpp:115                ~thread              ] calling quit() on ntp
1871773ms th_a       thread.cpp:160                quit                 ] destroying boost thread 139859955709696
1871773ms ntp        thread.cpp:246                exec                 ] thread canceled: 9 canceled_exception: Canceled
cancellation reason: [none given]
    {"reason":"[none given]"}
    ntp  thread_d.hpp:463 start_next_fiber
1871814ms th_a       thread.cpp:115                ~thread              ] calling quit() on p2p
1871814ms th_a       thread.cpp:160                quit                 ] destroying boost thread 139859936827136
1871814ms p2p        thread.cpp:246                exec                 ] thread canceled: 9 canceled_exception: Canceled
cancellation reason: [none given]
    {"reason":"[none given]"}
    p2p  thread_d.hpp:463 start_next_fiber

$ programs/witness_node/witness_node --data-dir data/issue492
1872817ms th_a       main.cpp:118                  main                 ] Error parsing logging config from config file /home/cc/graphene/src/graphene/data/issue492/config.ini, using default config
1872817ms th_a       witness.cpp:86                plugin_initialize    ] witness plugin:  plugin_initialize() begin
1872817ms th_a       witness.cpp:96                plugin_initialize    ] key_id_to_wif_pair: ["GPH6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV","5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3"]
1872817ms th_a       witness.cpp:114               plugin_initialize    ] witness plugin:  plugin_initialize() end
1872818ms th_a       object_database.cpp:91        open                 ] Opening object database from /home/cc/graphene/src/graphene/data/issue492/blockchain ...
1872839ms th_a       object_database.cpp:97        open                 ] Done opening object database.
1872840ms th_a       db_management.cpp:125         open                 ] last_block->id(): 000008275e5e641ab24a3056dbf20b32f5514084 last_block->block_num(): 2087
1872840ms th_a       thread.cpp:95                 thread               ] name:ntp tid:139829930632960
1872840ms th_a       thread.cpp:95                 thread               ] name:p2p tid:139829840967424
1872841ms th_a       application.cpp:140           reset_p2p_node       ] Configured p2p node to listen on 0.0.0.0:36355
1872842ms th_a       witness.cpp:119               plugin_startup       ] witness plugin:  plugin_startup() begin
1872842ms th_a       witness.cpp:126               plugin_startup       ] Launching block production for 11 witnesses.
1872842ms th_a       witness.cpp:137               plugin_startup       ] witness plugin:  plugin_startup() end
1872842ms th_a       main.cpp:176                  main                 ] Started witness node on a chain with 2087 blocks.
1872842ms th_a       main.cpp:177                  main                 ] Chain ID is c8f0951e680f37ca1e96ccdd835b5b0196a60290b86fbf11229c22b70b7a8965
1872918ms ntp        ntp.cpp:177                   read_loop            ] ntp_delta_time updated to 2216 us
1873000ms th_a       witness.cpp:188               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
1873997ms th_a       witness.cpp:188               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
1874997ms th_a       witness.cpp:188               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
1875997ms th_a       witness.cpp:188               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
1876997ms th_a       witness.cpp:188               block_production_loo ] Not producing block because production is disabled until we receive a recent block (see: --enable-stale-production)
1877384ms asio       main.cpp:167                  operator()           ] Caught SIGINT attempting to exit cleanly
1877384ms th_a       main.cpp:180                  main                 ] Exiting from signal 2
1877385ms th_a       thread.cpp:115                ~thread              ] calling quit() on ntp
1877385ms th_a       thread.cpp:160                quit                 ] destroying boost thread 139829930632960
1877385ms ntp        thread.cpp:246                exec                 ] thread canceled: 9 canceled_exception: Canceled
cancellation reason: [none given]
    {"reason":"[none given]"}
    ntp  thread_d.hpp:463 start_next_fiber
1877432ms th_a       thread.cpp:115                ~thread              ] calling quit() on p2p
1877432ms th_a       thread.cpp:160                quit                 ] destroying boost thread 139829840967424
1877432ms p2p        thread.cpp:246                exec                 ] thread canceled: 9 canceled_exception: Canceled
cancellation reason: [none given]
    {"reason":"[none given]"}
    p2p  thread_d.hpp:463 start_next_fiber

This is not high priority, as it does not happen consistently, and a simple re-indexing fixes the problem when it occurs.

[theoreticalbts]

Need to implement some logging to try to catch this next time it happens.