Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

fix(Postgres PGVector Store Node): Do not initialize client to avoid idle connections #12318

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

fcharlaix-opendsi
Copy link

@fcharlaix-opendsi fcharlaix-opendsi commented Dec 19, 2024

Summary

When using Postgres PGVector Store to import a lot of documents, the Postgres server will deny connection after some items due to the number of clients :

remaining connection slots are reserved for roles with the SUPERUSER attribute
error: remaining connection slots are reserved for roles with the SUPERUSER attribute
    at /usr/local/lib/node_modules/n8n/node_modules/pg-pool/index.js:45:11
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at PGVectorStore._initializeClient (/usr/local/lib/node_modules/n8n/node_modules/@langchain/community/dist/vectorstores/pgvector.cjs:344:23)
    at Function.initialize (/usr/local/lib/node_modules/n8n/node_modules/@langchain/community/dist/vectorstores/pgvector.cjs:336:9)
    at Function.fromDocuments (/usr/local/lib/node_modules/n8n/node_modules/@langchain/community/dist/vectorstores/pgvector.cjs:735:26)
    at Object.populateVectorStore (/usr/local/lib/node_modules/n8n/node_modules/@n8n/n8n-nodes-langchain/dist/nodes/vector_store/VectorStorePGVector/VectorStorePGVector.node.js:239:9)
    at ExecuteContext.execute (/usr/local/lib/node_modules/n8n/node_modules/@n8n/n8n-nodes-langchain/dist/nodes/vector_store/shared/createVectorStoreNode.js:206:21)
    at Workflow.runNode (/usr/local/lib/node_modules/n8n/node_modules/n8n-workflow/dist/Workflow.js:741:19)
    at /usr/local/lib/node_modules/n8n/node_modules/n8n-core/dist/WorkflowExecute.js:724:51
    at /usr/local/lib/node_modules/n8n/node_modules/n8n-core/dist/WorkflowExecute.js:1155:20

All of the Postgres clients are in idle state, waiting for the event ClientRead.

Test

Example workflow : Test_Postgres_PGVector_Store.json

  1. List Postgres connected clients at reset with : SELECT count(1) FROM pg_stat_activity;
  2. Run multiples time this workflow without the patch, and list again Postgres connected clients.
  3. Apply the patch, rebuild, run multiples time the workflow and list again Postgres connected clients.

You should see more connection after the test without the patch, there here until the n8n server is terminated.
With the patch, you should get the same number of connections as the number at reset.

Related Linear tickets, Github issues, and Community forum posts

The fix is based of this comment of langchain-ai/langchainjs#5029.

Review / Merge checklist

  • PR title and summary are descriptive. (conventions)
  • Docs updated or follow-up ticket created.
  • Tests included.
  • PR Labeled with release/backport (if the PR is an urgent fix that needs to be backported)

@CLAassistant
Copy link

CLAassistant commented Dec 19, 2024

CLA assistant check
All committers have signed the CLA.

@n8n-assistant n8n-assistant bot added community Authored by a community member in linear Issue or PR has been created in Linear for internal review labels Dec 19, 2024
@Joffcom
Copy link
Member

Joffcom commented Dec 19, 2024

Hey @fcharlaix-opendsi,

Thanks for the PR, We have created "GHC-608" as the internal reference to get this reviewed.

One of us will be in touch if there are any changes needed, in most cases this is normally within a couple of weeks but it depends on the current workload of the team.

@fcharlaix-opendsi
Copy link
Author

Hi, I have no idea how to properly set up a test for this bug :/

@fcharlaix-opendsi fcharlaix-opendsi marked this pull request as ready for review December 19, 2024 16:02
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
community Authored by a community member in linear Issue or PR has been created in Linear for internal review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants