Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[FIX] fix memory constraints for postgres #778

Merged
merged 16 commits into from
Jul 12, 2024
Merged

Conversation

jdkent
Copy link
Member

@jdkent jdkent commented Jul 11, 2024

neurostore     | Traceback (most recent call last):
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
neurostore     |     self.dialect.do_execute(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
neurostore     |     cursor.execute(statement, parameters)
neurostore     | psycopg2.errors.DiskFull: could not resize shared memory segment "/PostgreSQL.2021357946" to 4194304 bytes: No space left on device
neurostore     | 
neurostore     | 
neurostore     | The above exception was the direct cause of the following exception:
neurostore     | 
neurostore     | Traceback (most recent call last):
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
neurostore     |     await app(scope, receive, sender)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/swagger_ui.py", line 220, in __call__
neurostore     |     await self.router(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
neurostore     |     await self.middleware_stack(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
neurostore     |     await route.handle(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 485, in handle
neurostore     |     await self.app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
neurostore     |     await self.middleware_stack(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 806, in app
neurostore     |     await self.default(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/swagger_ui.py", line 233, in default_fn
neurostore     |     await self.app(original_scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 93, in __call__
neurostore     |     await self.simple_response(scope, receive, send, request_headers=headers)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 148, in simple_response
neurostore     |     await self.app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/routing.py", line 153, in __call__
neurostore     |     await self.router(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
neurostore     |     await self.middleware_stack(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
neurostore     |     await route.handle(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 485, in handle
neurostore     |     await self.app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
neurostore     |     await self.middleware_stack(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
neurostore     |     await route.handle(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
neurostore     |     await self.app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/routing.py", line 47, in __call__
neurostore     |     await self.next_app(original_scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/abstract.py", line 268, in __call__
neurostore     |     return await operation(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/security.py", line 100, in __call__
neurostore     |     await self.next_app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/abstract.py", line 268, in __call__
neurostore     |     return await operation(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/request_validation.py", line 142, in __call__
neurostore     |     await self.next_app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/abstract.py", line 268, in __call__
neurostore     |     return await operation(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/lifespan.py", line 26, in __call__
neurostore     |     await self.next_app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/abstract.py", line 268, in __call__
neurostore     |     return await operation(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/middleware/context.py", line 25, in __call__
neurostore     |     await self.next_app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/apps/flask.py", line 149, in __call__
neurostore     |     return await self.asgi_app(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/a2wsgi/wsgi.py", line 165, in __call__
neurostore     |     return await responder(scope, receive, send)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/a2wsgi/wsgi.py", line 200, in __call__
neurostore     |     await self.loop.run_in_executor(
neurostore     |   File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
neurostore     |     result = self.fn(*self.args, **self.kwargs)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/a2wsgi/wsgi.py", line 256, in wsgi
neurostore     |     iterable = self.app(environ, start_response)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2193, in wsgi_app
neurostore     |     response = self.handle_exception(e)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2190, in wsgi_app
neurostore     |     response = self.full_dispatch_request()
neurostore     |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1486, in full_dispatch_request
neurostore     |     rv = self.handle_user_exception(e)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1484, in full_dispatch_request
neurostore     |     rv = self.dispatch_request()
neurostore     |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1469, in dispatch_request
neurostore     |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/apps/flask.py", line 66, in __call__
neurostore     |     return self.fn(*args, **kwargs)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/main.py", line 134, in wrapper
neurostore     |     return decorated_function(request)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/response.py", line 171, in wrapper
neurostore     |     handler_response = function(*args, **kwargs)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/parameter.py", line 87, in wrapper
neurostore     |     return function(**kwargs)
neurostore     |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/main.py", line 123, in wrapper
neurostore     |     return function(*args, **kwargs)
neurostore     |   File "/neurostore/neurostore/resources/base.py", line 678, in post
neurostore     |     self.update_base_studies(unique_ids.get("base-studies"))
neurostore     |   File "/neurostore/neurostore/resources/base.py", line 183, in update_base_studies
neurostore     |     affected_base_studies = db.session.execute(query).fetchall()
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/scoping.py", line 778, in execute
neurostore     |     return self._proxied.execute(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
neurostore     |     return self._execute_internal(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2236, in _execute_internal
neurostore     |     result: Result[Any] = compile_state_cls.orm_execute_statement(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/orm/context.py", line 293, in orm_execute_statement
neurostore     |     result = conn.execute(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
neurostore     |     return meth(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
neurostore     |     return connection._execute_clauseelement(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
neurostore     |     ret = self._execute_context(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
neurostore     |     return self._exec_single_context(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
neurostore     |     self._handle_dbapi_exception(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2353, in _handle_dbapi_exception
neurostore     |     raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
neurostore     |     self.dialect.do_execute(
neurostore     |   File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 924, in do_execute
neurostore     |     cursor.execute(statement, parameters)
neurostore     | sqlalchemy.exc.OperationalError: (psycopg2.errors.DiskFull) could not resize shared memory segment "/PostgreSQL.2021357946" to 4194304 bytes: No space left on device
neurostore     | 
neurostore     | [SQL: SELECT DISTINCT base_studies.id, base_studies.has_images, base_studies.has_coordinates, (SELECT coalesce(bool_and(points.analysis_id IS NOT NULL), %(coalesce_2)s) AS coalesce_1 
neurostore     | FROM points, analyses 
neurostore     | WHERE points.analysis_id = analyses.id) AS new_has_coordinates, (SELECT coalesce(bool_and(images.analysis_id IS NOT NULL), %(coalesce_4)s) AS coalesce_3 
neurostore     | FROM images, analyses 
neurostore     | WHERE images.analysis_id = analyses.id) AS new_has_images 
neurostore     | FROM base_studies JOIN studies ON studies.base_study_id = base_studies.id LEFT OUTER JOIN analyses ON analyses.study_id = studies.id 
neurostore     | WHERE base_studies.id IN (%(id_1_1)s)]
neurostore     | [parameters: {'coalesce_2': False, 'coalesce_4': False, 'id_1_1': 'FMfXCKVzsapg'}]
neurostore     | (Background on this error at: https://sqlalche.me/e/20/e3q8)

@jdkent jdkent merged commit a115f1e into master Jul 12, 2024
16 checks passed
@jdkent jdkent deleted the fix/postgres_memory branch July 12, 2024 03:12
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant