Crash neo4j 4.4.3

[Gwojda]

Hi,
Getting some crashes in case python driver failed to write.
Here the logs :

Failed to write data to connection IPv4Address(('x.x.x.x', 7687)) (IPv4Address(('x.x.x.x', 7687)))

File \"/usr/local/lib/python3.9/site-packages/neo4j/work/simple.py\", line 421, in write_transaction return self._run_transaction(WRITE_ACCESS, transaction_function, *args, **kwargs)\n", "  File \"/usr/local/lib/python3.9/site-packages/neo4j/work/simple.py\", 
line 322, in _run_transaction self._open_transaction(access_mode=access_mode, metadata=metadata, timeout=timeout)", "  File \"/usr/local/lib/python3.9/site-packages/neo4j/work/simple.py\", 
line 255, in _open_transaction\n    self._connect(access_mode=access_mode)\n", "  File \"/usr/local/lib/python3.9/site-packages/neo4j/work/simple.py\", 
line 108, in _connect\n    super()._connect(access_mode)\n", "  File \"/usr/local/lib/python3.9/site-packages/neo4j/work/__init__.py\", 
line 79, in _connect\n    self._connection = self._pool.acquire(\n", "  File \"/usr/local/lib/python3.9/site-packages/neo4j/io/__init__.py\", 
line 842, in acquire\n    return self._acquire(self.address, timeout)\n", "  File \"/usr/local/lib/python3.9/site-packages/neo4j/io/__init__.py\", 
line 670, in _acquire\n    connection.close()\n", "  File \"/usr/local/lib/python3.9/site-packages/neo4j/io/_bolt4.py\", 
line 336, in close\n    self._send_all()\n", "  File \"/usr/local/lib/python3.9/site-packages/neo4j/io/__init__.py\", 
line 501, in _send_all\n    self._set_defunct_write(error)\n", "  File \"/usr/local/lib/python3.9/site-packages/neo4j/io/__init__.py\", 
line 551, in _set_defunct_write\n    self._set_defunct(message, error=error, silent=silent)\n", "  File \"/usr/local/lib/python3.9/site-packages/neo4j/io/__init__.py\", 
line 566, in _set_defunct\n    self.pool.deactivate(address=self.unresolved_address)\n", "  File \"/usr/local/lib/python3.9/site-packages/neo4j/io/__init__.py\", 
line 777, in deactivate\n    connections.remove(conn)\n", "ValueError: deque.remove(x): x not in deque\

Maybe it's related to #598
Thank's

[robsdedude]

Thanks for the report. There was in fact another issue similar to the one fixed in the PR you linked. I fixed it recently as I received another report of the same error.
The fix is here: #733

A patch release will most likely happen within the next days. If the error persists after an upgrade to 4.4.4 (as said, coming soon), please feel free to re-open this issue.

[Gwojda]

Ah ! good news
Thanks you, and have a nice day.

[robsdedude]

4.4.4 has now been released. I'd be happy to hear back if it resolves the issue.

[Gwojda]

Ok, I'll ping here if this issue is still happening. 👍

[Gwojda]

Hi,
No more crash, but randomly i get some Failed to write data to connection IPv4Address(('x.x.x.x', 7687)) (IPv4Address(('y.y.y.y', 7687)))
I dont often consume all data returned by the query, it might be the problem ?
Can't give you more information, it happened only a few time on my big data pipeline, it hard to reproduce, and error log is already on debug, and no more information is logging.
Thanks for your help,
Gautier

[robsdedude]

This means that an error occurred while the driver tried to put data onto the wire. This can have a plethora of reasons, really. From network issues, to idle connections getting closed by the server or some loadbalancer in between, to potentially more.

If you have debug logging enabled (https://neo4j.com/docs/api/python-driver/4.4/api.html#logging) the logged message right before Failed to write data to connection should be the actual error.

The probably simplest solution to go about this would be to use transaction function (a.k.a. managed transactions), if you don't already. This will make the driver automagically retry the query for you (https://neo4j.com/docs/api/python-driver/4.4/api.html#managed-transactions-transaction-functions). Just be aware that those functions must be idempotent from the driver's view (i.e., they must not alter the state of your program). That is because as said: the driver will re-run the function should it fail and the driver classifies the failure as retryable.

I hope this helps. If not, please share more of the debug log with so that I can get a better understanding of the nature of the failure.