Exception while processing eth subscription: ValueError({'code': -32000, 'message': 'missing trie node

[ederenn]

Description

Golem Version: GOLEM Version: 0.20.0+dev259.g917ad6c

Golem-Messages version (leave empty if unsure): golem_messages Version: 3.9.0

Electron version (if used): 0.2.1

OS [e.g. Windows 10 Pro]: Linux 18.04

Branch (if launched from source): develop

Mainnet/Testnet: mainnet

Priority label is set to the lowest by default. To setup higher priority please change the label
P0 label is set for Severity-Critical/Effort-easy
P1 label is set for Severity-Critical/Effort-hard
P2 label is set for Severity-Low/ Effort-easy
P3 label is set for Severity-Low/Effort-hard

Description of the issue:

During providing for other node. Both nodes on develop.

Steps To Reproduce

Short description of steps to reproduce the behavior:
e.g.

1. Launch Golem on develop two nodes, provider and requestor
2. Request small task 400x400, Hellispoter, 1 frame, 5 subtasks, 30 min task, 10 min subtask, 0,2 bid
   3.Go to provider's logs
3. See error

Logs and any additional context

ERROR    [golem_sci.implementation           ] Exception while processing eth subscription: ValueError({'code': -32000, 'message': 'missing trie node 53d423d1c35f2a4ce730a2742358ec28dcf52ecb8a5e44299860c4be5a0745d4 (path )'},)
Traceback (most recent call last):
  File "/home/ederenn/projects/golem-env/lib/python3.6/site-packages/golem_sci/implementation.py", line 596, in _pull_eth_subscription_events
    sub.address,
  File "/home/ederenn/projects/golem-env/lib/python3.6/site-packages/golem_sci/implementation.py", line 548, in _find_incoming_eth_transfers
    block=from_block,
  File "/home/ederenn/projects/golem-env/lib/python3.6/site-packages/golem_sci/client.py", line 110, in get_balance
    return self.web3.eth.getBalance(account, block)
  File "/home/ederenn/projects/golem-env/src/web3/web3/eth.py", line 108, in getBalance
    [account, block_identifier],
  File "/home/ederenn/projects/golem-env/src/web3/web3/manager.py", line 106, in request_blocking
    raise ValueError(response["error"])
ValueError: {'code': -32000, 'message': 'missing trie node 53d423d1c35f2a4ce730a2742358ec28dcf52ecb8a5e44299860c4be5a0745d4 (path )'}

Proposed Solution?

(Optional: What could be a solution for that issue)

[ederenn]

encountered again on b0.22 based branch, run on source, Linux

2019-11-27 18:09:01 WARNING  golem_sci.implementation            Exception while processing eth subscription: ValueError({'code': -32000, 'message': 'missing trie node 2486ff27ec0207d79e5fa48b44ca212908b1f42724c9bc0f0c4908a4e342de9e (path )'},

[maaktweluit]

AFAIR @shadeofblue silenced this error, let me know if you want me to look into it

[shadeofblue]

@maaktweluit I haven't silenced it -> I turned the spam into a single warning... still, it's a non-issue and doesn't require a fix

[etam]

This happens also for other calls, get_gntb_balance in this case.

2019-12-10 11:01:05 ERROR    golem                               Service Error: Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/twisted/_threads/_threadworker.py", line 46, in work
    task()
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/twisted/_threads/_team.py", line 190, in doWork
    task()
--- <exception caught here> ---
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/data/golem/golem_git/golem/ethereum/transactionsystem.py", line 1014, in _run
    self._payment_processor.sendout()
  File "/data/golem/golem_git/golem/ethereum/paymentprocessor.py", line 253, in sendout
    now - datetime.timedelta(seconds=self.CLOSURE_TIME_DELAY),
  File "/data/golem/golem_git/golem/ethereum/paymentprocessor.py", line 188, in __get_next_batch
    gntb_balance = self._sci.get_gntb_balance(self._sci.get_eth_address())
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/golem_sci/implementation.py", line 187, in get_gntb_balance
    return self._call(self._gntb.functions.balanceOf(address))
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/golem_sci/implementation.py", line 379, in _call
    block_identifier=self._confirmed_block,
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/web3/contract.py", line 1031, in call
    **self.kwargs)
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/web3/contract.py", line 1262, in call_contract_function
    return_data = web3.eth.call(call_transaction, block_identifier=block_id)
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/eth_utils/functional.py", line 22, in inner
    return callback(fn(*args, **kwargs))
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/web3/eth.py", line 270, in call
    [transaction, block_identifier],
  File "/data/golem/golem_git/venv/lib/python3.6/site-packages/web3/manager.py", line 106, in request_blocking
    raise ValueError(response["error"])
builtins.ValueError: {'code': -32000, 'message': 'missing trie node 54e616705981296dd22b5093206770d4db903e7b6ab565e43988b48bc73161b3 (path )'}
NoneType: None

[ederenn]

happens in version 2019-12-12 13:32:55 INFO golemapp GOLEM Version: 0.21.0+dev506.g042b40f

I cannot create a task because of this error

golem.log