Connection drops on Mac when connecting to db in host machine

[vrajat]

I see that this error log got appended on lineage log after I posted the previous comment( would have took about ~3-5 mins)

ERROR:data_lineage.server:Exception on /api/main [GET]

Traceback (most recent call last):

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context

self.dialect.do_execute(

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute

cursor.execute(statement, parameters)

psycopg2.OperationalError: server closed the connection unexpectedly

This probably means the server terminated abnormally

before or while processing the request.



The above exception was the direct cause of the following exception:


Traceback (most recent call last):

File "/opt/pysetup/.venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request

rv = self.dispatch_request()

File "/opt/pysetup/.venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request

return self.view_functions[rule.endpoint](**req.view_args)

File "/opt/pysetup/.venv/lib/python3.8/site-packages/flask_restful/__init__.py", line 467, in wrapper

resp = resource(*args, **kwargs)

File "/opt/pysetup/.venv/lib/python3.8/site-packages/flask/views.py", line 89, in view

return self.dispatch_request(*args, **kwargs)

File "/opt/pysetup/.venv/lib/python3.8/site-packages/flask_restful/__init__.py", line 582, in dispatch_request

resp = meth(*args, **kwargs)

File "/opt/pysetup/.venv/lib/python3.8/site-packages/data_lineage/server.py", line 69, in get

column_edges = self._catalog.get_column_lineages(args["job_ids"])

File "/opt/pysetup/.venv/lib/python3.8/site-packages/dbcat/catalog/catalog.py", line 299, in get_column_lineages

return query.all()

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3373, in all

return list(self)

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__

return self._execute_and_instances(context)

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances

result = conn.execute(querycontext.statement, self._params)

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute

return meth(self, multiparams, params)

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection

return connection._execute_clauseelement(self, multiparams, params)

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement

ret = self._execute_context(

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context

self._handle_dbapi_exception(

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception

util.raise_(

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_

raise exception

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context

self.dialect.do_execute(

File "/opt/pysetup/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute

cursor.execute(statement, parameters)

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly

This probably means the server terminated abnormally

before or while processing the request.



[SQL: SELECT column_lineage.id AS column_lineage_id, column_lineage.context AS column_lineage_context, column_lineage.source_id AS column_lineage_source_id, column_lineage.target_id AS column_lineage_target_id, column_lineage.job_execution_id AS column_lineage_job_execution_id, sources_1.id AS sources_1_id, sources_1.source_type AS sources_1_source_type, sources_1.name AS sources_1_name, sources_1.dialect AS sources_1_dialect, sources_1.uri AS sources_1_uri, sources_1.port AS sources_1_port, sources_1.username AS sources_1_username, sources_1.password AS sources_1_password, sources_1.database AS sources_1_database, sources_1.instance AS sources_1_instance, sources_1.cluster AS sources_1_cluster, sources_1.project_id AS sources_1_project_id, sources_1.project_credentials AS sources_1_project_credentials, sources_1.page_size AS sources_1_page_size, sources_1.filter_key AS sources_1_filter_key, sources_1.included_tables_regex AS sources_1_included_tables_regex, sources_1.key_path AS sources_1_key_path, sources_1.account AS sources_1_account, sources_1.role AS sources_1_role, sources_1.warehouse AS sources_1_warehouse, schemata_1.id AS schemata_1_id, schemata_1.name AS schemata_1_name, schemata_1.source_id AS schemata_1_source_id, tables_1.id AS tables_1_id, tables_1.name AS tables_1_name, tables_1.schema_id AS tables_1_schema_id, columns_1.id AS columns_1_id, columns_1.name AS columns_1_name, columns_1.data_type AS columns_1_data_type, columns_1.sort_order AS columns_1_sort_order, columns_1.table_id AS columns_1_table_id, sources_2.id AS sources_2_id, sources_2.source_type AS sources_2_source_type, sources_2.name AS sources_2_name, sources_2.dialect AS sources_2_dialect, sources_2.uri AS sources_2_uri, sources_2.port AS sources_2_port, sources_2.username AS sources_2_username, sources_2.password AS sources_2_password, sources_2.database AS sources_2_database, sources_2.instance AS sources_2_instance, sources_2.cluster AS sources_2_cluster, sources_2.project_id AS sources_2_project_id, sources_2.project_credentials AS sources_2_project_credentials, sources_2.page_size AS sources_2_page_size, sources_2.filter_key AS sources_2_filter_key, sources_2.included_tables_regex AS sources_2_included_tables_regex, sources_2.key_path AS sources_2_key_path, sources_2.account AS sources_2_account, sources_2.role AS sources_2_role, sources_2.warehouse AS sources_2_warehouse, schemata_2.id AS schemata_2_id, schemata_2.name AS schemata_2_name, schemata_2.source_id AS schemata_2_source_id, tables_2.id AS tables_2_id, tables_2.name AS tables_2_name, tables_2.schema_id AS tables_2_schema_id, columns_2.id AS columns_2_id, columns_2.name AS columns_2_name, columns_2.data_type AS columns_2_data_type, columns_2.sort_order AS columns_2_sort_order, columns_2.table_id AS columns_2_table_id, jobs_1.id AS jobs_1_id, jobs_1.name AS jobs_1_name, jobs_1.context AS jobs_1_context, jobs_1.source_id AS jobs_1_source_id, job_executions_1.id AS job_executions_1_id, job_executions_1.job_id AS job_executions_1_job_id, job_executions_1.started_at AS job_executions_1_started_at, job_executions_1.ended_at AS job_executions_1_ended_at, job_executions_1.status AS job_executions_1_status

FROM column_lineage LEFT OUTER JOIN columns AS columns_1 ON columns_1.id = column_lineage.source_id LEFT OUTER JOIN tables AS tables_1 ON tables_1.id = columns_1.table_id LEFT OUTER JOIN schemata AS schemata_1 ON schemata_1.id = tables_1.schema_id LEFT OUTER JOIN sources AS sources_1 ON sources_1.id = schemata_1.source_id LEFT OUTER JOIN columns AS columns_2 ON columns_2.id = column_lineage.target_id LEFT OUTER JOIN tables AS tables_2 ON tables_2.id = columns_2.table_id LEFT OUTER JOIN schemata AS schemata_2 ON schemata_2.id = tables_2.schema_id LEFT OUTER JOIN sources AS sources_2 ON sources_2.id = schemata_2.source_id LEFT OUTER JOIN job_executions AS job_executions_1 ON job_executions_1.id = column_lineage.job_execution_id LEFT OUTER JOIN jobs AS jobs_1 ON jobs_1.id = job_executions_1.job_id]

(Background on this error at: http://sqlalche.me/e/13/e3q8)

Originally posted by @siva-mudiyanur in #57 (comment)

[vrajat]

@siva-mudiyanur also provided the solution:

This solution might help us in resolving the issue: https://stackoverflow.com/questions/55457069/how-to-fix-operationalerror-psycopg2-operationalerror-server-closed-the-conn