Neon PostgreSQL - New Components #16685
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 3 Skipped Deployments
|
|
""" WalkthroughA comprehensive Neon PostgreSQL integration is introduced, including a new app module, multiple actions for CRUD operations and custom queries, and sources for detecting new tables, columns, and rows. The implementation leverages and extends existing Pipedream PostgreSQL utilities, providing dynamic schema, table, and column selection, centralized error handling with SSL verification hints, and event-driven polling for database changes. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant NeonAction
participant NeonApp
participant NeonDB
User->>NeonAction: Trigger action (e.g., Insert Row, Find Row)
NeonAction->>NeonApp: Prepare connection/configuration
NeonApp->>NeonDB: Execute SQL or perform operation
NeonDB-->>NeonApp: Return result/error
NeonApp-->>NeonAction: Return result/error
NeonAction-->>User: Output result or error message
sequenceDiagram
participant Source
participant NeonApp
participant NeonDB
participant User
Source->>NeonApp: Poll for changes (e.g., new row, column, table)
NeonApp->>NeonDB: Query for changes
NeonDB-->>NeonApp: Return new/updated data
NeonApp-->>Source: Provide results
Source-->>User: Emit event with metadata
Assessment against linked issues
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
✅ Files skipped from review due to trivial changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 14
🧹 Nitpick comments (4)
components/neon_postgres/actions/delete-rows/delete-rows.mjs (1)
50-67: Consider extracting common error handling patternThis error handling pattern appears to be repeated across multiple action components. Consider extracting it to a utility function to improve maintainability.
+ // In a shared utility file + export const handlePostgresError = (error) => { + let errorMsg = "Operation failed due to an error. "; + errorMsg += `${error}`.includes("SSL verification failed") + ? "This could be because SSL verification failed. To resolve this, reconnect your account and set SSL Verification Mode: Skip Verification, and try again." + : `${error}`; + return errorMsg; + } // In this file - try { - const rows = await this.neon.deleteRows( - schema, - table, - column, - value, - ); - $.export("$summary", `Deleted ${rows.length} rows from ${table}`); - return rows; - } catch (error) { - let errorMsg = "Row not deleted due to an error. "; - errorMsg += `${error}`.includes("SSL verification failed") - ? "This could be because SSL verification failed. To resolve this, reconnect your account and set SSL Verification Mode: Skip Verification, and try again." - : `${error}`; - throw new Error(errorMsg); - } + try { + const rows = await this.neon.deleteRows( + schema, + table, + column, + value, + ); + $.export("$summary", `Deleted ${rows.length} rows from ${table}`); + return rows; + } catch (error) { + throw new Error(handlePostgresError(error).replace("Operation", "Row not deleted")); + }components/neon_postgres/actions/find-row-custom-query/find-row-custom-query.mjs (1)
41-43: Enhance query validation messageThe error message for non-SELECT queries could be more specific about this component's purpose and point users to alternative components for data modification.
if (!query.toLowerCase().includes("select")) { - throw new Error("Need be a `SELECT` statement query. Read more about [SELECT queries here](https://www.w3schools.com/sql/sql_select.asp)"); + throw new Error("This component only supports `SELECT` statement queries. For data modification, please use the 'Execute SQL Query' component instead. Read more about [SELECT queries here](https://www.w3schools.com/sql/sql_select.asp)"); }components/neon_postgres/sources/new-column/new-column.mjs (1)
34-37: Add logging for clarity on new column detectionAdding a log message would make it clearer in the execution logs when new columns are detected or when no changes are found.
const newColumns = columns.filter((column) => !previousColumns.includes(column)); + console.log(`Found ${newColumns.length} new columns in table ${this.table}`); for (const column of newColumns) { const meta = this.generateMeta(column); this.$emit(column, meta); }components/neon_postgres/actions/insert-row/insert-row.mjs (1)
48-49: Enhance the summary message with detailsThe current summary message is generic. Adding details about which table and how many columns were affected would make the message more informative.
- $.export("$summary", "New row inserted"); + $.export("$summary", `New row inserted into ${table} with ${columns.length} column(s)`); return res;
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge Base: Disabled due to data retention organization setting
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (15)
components/neon_postgres/actions/delete-rows/delete-rows.mjs(1 hunks)components/neon_postgres/actions/execute-custom-query/execute-custom-query.mjs(1 hunks)components/neon_postgres/actions/find-row-custom-query/find-row-custom-query.mjs(1 hunks)components/neon_postgres/actions/find-row/find-row.mjs(1 hunks)components/neon_postgres/actions/insert-row/insert-row.mjs(1 hunks)components/neon_postgres/actions/update-row/update-row.mjs(1 hunks)components/neon_postgres/actions/upsert-row/upsert-row.mjs(1 hunks)components/neon_postgres/neon_postgres.app.mjs(1 hunks)components/neon_postgres/package.json(2 hunks)components/neon_postgres/sources/common/common.mjs(1 hunks)components/neon_postgres/sources/new-column/new-column.mjs(1 hunks)components/neon_postgres/sources/new-or-updated-row/new-or-updated-row.mjs(1 hunks)components/neon_postgres/sources/new-row-custom-query/new-row-custom-query.mjs(1 hunks)components/neon_postgres/sources/new-row/new-row.mjs(1 hunks)components/neon_postgres/sources/new-table/new-table.mjs(1 hunks)
🧰 Additional context used
🪛 GitHub Actions: Pull Request Checks
components/neon_postgres/actions/update-row/update-row.mjs
[error] 1-1: Error related to key 'neon_api_keys-update-row' found in this component.
components/neon_postgres/actions/insert-row/insert-row.mjs
[error] 1-1: Error related to key 'neon_api_keys-insert-row' found in this component.
components/neon_postgres/actions/find-row/find-row.mjs
[error] 1-1: Error related to key 'neon_api_keys-find-row' found in this component.
components/neon_postgres/actions/execute-custom-query/execute-custom-query.mjs
[error] 1-1: Error related to key 'neon_api_keys-execute-custom-query' found in this component.
components/neon_postgres/sources/new-row-custom-query/new-row-custom-query.mjs
[error] 1-1: Error related to key 'neon_api_keys-new-row-custom-query' found in this component.
components/neon_postgres/sources/new-column/new-column.mjs
[error] 1-1: Error related to key 'neon_api_keys-new-column' found in this component.
components/neon_postgres/sources/new-table/new-table.mjs
[error] 1-1: Error related to key 'neon_api_keys-new-table' found in this component.
components/neon_postgres/actions/delete-rows/delete-rows.mjs
[error] 1-1: Error related to key 'neon_api_keys-delete-rows' found in this component.
components/neon_postgres/actions/find-row-custom-query/find-row-custom-query.mjs
[error] 1-1: Error related to key 'neon_api_keys-find-row-custom-query' found in this component.
components/neon_postgres/sources/new-or-updated-row/new-or-updated-row.mjs
[error] 1-1: Error related to key 'neon_api_keys-new-or-updated-row' found in this component.
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: Verify TypeScript components
- GitHub Check: Publish TypeScript components
🔇 Additional comments (19)
components/neon_postgres/package.json (1)
3-3: LGTM: Version update and dependencies addedThe version update to 0.1.0 and the addition of required dependencies look good. These dependencies align with the Neon PostgreSQL integration needs.
Also applies to: 14-17
components/neon_postgres/sources/common/common.mjs (1)
1-19: Well-structured common module for sourcesGood implementation of a reusable base configuration for Neon Postgres source components. The approach of extending the common PostgreSQL source with Neon-specific properties follows good software design practices.
components/neon_postgres/actions/execute-custom-query/execute-custom-query.mjs (1)
1-28: Overall implementation looks goodThe action is well-structured with appropriate props and execution logic. The SQL execution and results handling are implemented correctly.
🧰 Tools
🪛 GitHub Actions: Pull Request Checks
[error] 1-1: Error related to key 'neon_api_keys-execute-custom-query' found in this component.
components/neon_postgres/actions/update-row/update-row.mjs (2)
78-82: Good error handling for SSL verificationThe error handling implementation is well done, especially the specific message for SSL verification failures which provides clear guidance to users on how to fix the issue.
1-85: Overall implementation is goodThe action is well-structured with appropriate props for schema, table, column selection and value handling. The dynamic prop definitions that depend on previous selections provide a good user experience.
🧰 Tools
🪛 GitHub Actions: Pull Request Checks
[error] 1-1: Error related to key 'neon_api_keys-update-row' found in this component.
components/neon_postgres/actions/delete-rows/delete-rows.mjs (1)
67-73: Well-structured error handling with good user guidanceThe error handling logic provides helpful guidance specifically for SSL verification issues, which is a common problem with database connections. This improves user experience by offering a clear solution path.
components/neon_postgres/actions/find-row/find-row.mjs (1)
50-76: Good error handling for SSL verification failuresThe error handling is robust, especially with the helpful message for SSL verification failures that guides users to resolve the issue.
components/neon_postgres/sources/new-row-custom-query/new-row-custom-query.mjs (2)
70-77: Good validation for array values and placeholder matchingThe validation logic properly checks that values is an array and that the number of placeholders matches the number of values, which prevents SQL injection and improves error messaging.
79-82: Good check for column uniquenessVerifying that the selected column contains unique values is important for proper deduplication when emitting events.
components/neon_postgres/neon_postgres.app.mjs (2)
1-8: Clean integration with PostgreSQL packageEffectively imports and integrates with the
@pipedream/postgresqlpackage, allowing reuse of common PostgreSQL functionality while adding Neon-specific features.
11-27: Well-structured client configurationThe
getClientConfiguration()method cleanly extracts auth parameters and applies SSL configuration, which centralizes connection setup for all Neon Postgres components.components/neon_postgres/actions/upsert-row/upsert-row.mjs (2)
54-56: Good handling of update columnsExcluding the conflict target column from the update columns list is a best practice for upsert operations.
93-98: Consistent error handlingThis error handling follows the same pattern as other components in the integration, providing helpful guidance for SSL verification issues.
components/neon_postgres/sources/new-table/new-table.mjs (1)
19-31: LGTM: Logic for detecting new tables is well implementedThe implementation correctly retrieves the current tables, compares against previously stored values, and emits events for new tables. The state management is properly handled with the _getPreviousValues() and _setPreviousValues() methods.
components/neon_postgres/sources/new-row/new-row.mjs (3)
6-10: LGTM: Component metadata is correctly definedThe component key "neon_postgres-new-row" follows the expected naming convention, and the component is correctly configured with appropriate metadata and dedupe settings.
42-52: LGTM: Deploy hook correctly handles column configurationThe deploy hook appropriately sets the column to either the user-specified column or the table's primary key, and initializes the state with existing rows.
69-77: LGTM: Run method properly validates column and processes new rowsThe implementation correctly checks if the column contains unique values (crucial for proper deduplication) and throws a clear error message if duplicates exist. The method then processes new rows with the newRows helper.
components/neon_postgres/sources/new-or-updated-row/new-or-updated-row.mjs (2)
28-52: LGTM: Column props are well-defined with clear descriptionsThe implementation correctly defines both an identifier column (for uniquely identifying rows) and a timestamp column (for detecting updates). The descriptions provide clear guidance to users about their purpose and expected values.
65-71: LGTM: Event metadata generation creates unique identifiersThe generateMeta method properly creates a unique ID by combining the identifier column value and the timestamp, which ensures proper event deduplication even when the same row is updated multiple times.
components/neon_postgres/actions/execute-custom-query/execute-custom-query.mjs
Outdated
Show resolved
Hide resolved
components/neon_postgres/actions/find-row-custom-query/find-row-custom-query.mjs
Outdated
Show resolved
Hide resolved
components/neon_postgres/actions/find-row-custom-query/find-row-custom-query.mjs
Show resolved
Hide resolved
components/neon_postgres/sources/new-row-custom-query/new-row-custom-query.mjs
Outdated
Show resolved
Hide resolved
components/neon_postgres/sources/new-or-updated-row/new-or-updated-row.mjs
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Actionable comments posted: 5
🔭 Outside diff range comments (1)
components/postgresql/actions/find-row-custom-query/find-row-custom-query.mjs (1)
34-39:⚠️ Potential issueIncorrect placeholder counting can trigger false-negative validation
query.match(/\$/g)?.lengthalso matches occurrences of$$(used for string literals) or dollar-quoted function bodies, so the computednumberOfValuescan be higher than the real number of parameters, erroneously rejecting valid queries. Matching only$<number>avoids this.-const numberOfValues = query?.match(/\$/g)?.length || 0; +const numberOfValues = query?.match(/\$\d+/g)?.length || 0;
🧹 Nitpick comments (6)
components/postgresql/actions/find-row-custom-query/find-row-custom-query.mjs (1)
41-43: Case-insensitive “SELECT” detection is fragile
toLowerCase().includes("select")will pass strings like"myselectfield"and fail queries with leading whitespace/newlines. Prefer a stricter regex anchored to the first non-whitespace token, e.g.:if (!/^\s*select\b/i.test(query)) { throw new Error("Query must be a SELECT statement …"); }components/postgresql/sources/new-row/new-row.mjs (1)
49-53:deploy-time uniqueness scan may be expensive on large tables
RunningisColumnUniqueduring deployment potentially scans the entire column. For very large tables this blocks source activation and may time-out. Consider:
- Checking
pg_constraintfor aPRIMARY KEY/UNIQUEconstraint instead of scanning data.- Falling back to the data scan only when no constraint exists, with a warning about performance.
This preserves safety without sacrificing startup time.
components/postgresql/actions/insert-row/insert-row.mjs (3)
2-2: Missing utils import guard
IfparseRowValuesthrows, the action exits before emitting a helpful error. Wrapping its call in a try/catch that augments the message with component context would improve UX.
32-33: Description grammar
Change “Accept” → “Accepts” for subject/verb agreement.-description: "JSON representation of your table rows. Accept a single row … +description: "JSON representation of your table rows. Accepts a single row …
63-66: Pluralisation edge case
When zero rows are inserted the summary reads “Successfully inserted 0 row”, missing the “s”. Consider using an i18n-safe plural helper or:-`Successfully inserted ${results.length} row${results.length === 1 ? "" : "s"}` +`Successfully inserted ${results.length} row${results.length === 1 ? "" : "s"}`and handle the 0 case explicitly if desired.
components/postgresql/actions/delete-rows/delete-rows.mjs (1)
58-66: Error message & summary wording
- The prefix
"Row not deleted …"is singular but the action supports multiple rows.- The summary relies on
rows.length. If the driver starts returning{ rowCount }instead of an array, the summary will misreport.-const errorMsg = "Row not deleted due to an error. "; +const errorMsg = "Row(s) not deleted due to an error. "; … -$.export("$summary", `Deleted ${rows.length} rows from ${table}`); +const count = Array.isArray(rows) ? rows.length : rows?.rowCount ?? 0; +$.export("$summary", `Deleted ${count} row${count === 1 ? "" : "s"} from ${table}`);
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Cache: Disabled due to data retention organization setting
Knowledge Base: Disabled due to data retention organization setting
📒 Files selected for processing (24)
components/neon_postgres/actions/delete-rows/delete-rows.mjs(1 hunks)components/neon_postgres/actions/find-row-custom-query/find-row-custom-query.mjs(1 hunks)components/neon_postgres/actions/find-row/find-row.mjs(1 hunks)components/neon_postgres/actions/insert-row/insert-row.mjs(1 hunks)components/neon_postgres/actions/update-row/update-row.mjs(1 hunks)components/neon_postgres/actions/upsert-row/upsert-row.mjs(1 hunks)components/neon_postgres/common/utils.mjs(1 hunks)components/neon_postgres/package.json(2 hunks)components/neon_postgres/sources/new-row-custom-query/new-row-custom-query.mjs(1 hunks)components/neon_postgres/sources/new-row/new-row.mjs(1 hunks)components/postgresql/actions/delete-rows/delete-rows.mjs(2 hunks)components/postgresql/actions/execute-custom-query/execute-custom-query.mjs(1 hunks)components/postgresql/actions/find-row-custom-query/find-row-custom-query.mjs(2 hunks)components/postgresql/actions/find-row/find-row.mjs(2 hunks)components/postgresql/actions/insert-row/insert-row.mjs(3 hunks)components/postgresql/actions/update-row/update-row.mjs(3 hunks)components/postgresql/actions/upsert-row/upsert-row.mjs(2 hunks)components/postgresql/common/utils.mjs(1 hunks)components/postgresql/postgresql.app.mjs(7 hunks)components/postgresql/sources/new-column/new-column.mjs(1 hunks)components/postgresql/sources/new-or-updated-row/new-or-updated-row.mjs(1 hunks)components/postgresql/sources/new-row-custom-query/new-row-custom-query.mjs(4 hunks)components/postgresql/sources/new-row/new-row.mjs(2 hunks)components/postgresql/sources/new-table/new-table.mjs(1 hunks)
✅ Files skipped from review due to trivial changes (4)
- components/postgresql/sources/new-or-updated-row/new-or-updated-row.mjs
- components/postgresql/sources/new-table/new-table.mjs
- components/postgresql/actions/execute-custom-query/execute-custom-query.mjs
- components/postgresql/sources/new-column/new-column.mjs
🚧 Files skipped from review as they are similar to previous changes (9)
- components/neon_postgres/package.json
- components/neon_postgres/actions/find-row/find-row.mjs
- components/neon_postgres/actions/delete-rows/delete-rows.mjs
- components/neon_postgres/actions/update-row/update-row.mjs
- components/neon_postgres/actions/find-row-custom-query/find-row-custom-query.mjs
- components/neon_postgres/actions/upsert-row/upsert-row.mjs
- components/neon_postgres/sources/new-row-custom-query/new-row-custom-query.mjs
- components/neon_postgres/actions/insert-row/insert-row.mjs
- components/neon_postgres/sources/new-row/new-row.mjs
⏰ Context from checks skipped due to timeout of 90000ms (4)
- GitHub Check: pnpm publish
- GitHub Check: Publish TypeScript components
- GitHub Check: Lint Code Base
- GitHub Check: Verify TypeScript components
🔇 Additional comments (14)
components/neon_postgres/common/utils.mjs (1)
1-16: Well-implemented utility function for parsing row values.This utility function provides a robust way to handle different types of input values, properly handling recursive parsing of arrays, JSON string parsing, and fallbacks for parsing failures. It matches the implementation in the PostgreSQL module, ensuring consistency across different database integrations.
components/postgresql/common/utils.mjs (1)
1-16: Identical implementation to Neon PostgreSQL utility - good for consistency.This function uses the same robust implementation as the Neon PostgreSQL module, which is excellent for maintaining consistency across database integrations. The function handles all edge cases appropriately (nullish values, arrays, JSON strings, and other types).
components/postgresql/sources/new-row-custom-query/new-row-custom-query.mjs (5)
8-8: Version increment reflects the changes.The version increment appropriately tracks the changes made to this component.
43-43: Improved query prop description.The enhanced description is more informative and provides a helpful example, which improves the developer experience.
52-68: Excellent refactoring: validation moved to deploy hook.Moving validation logic from the runtime execution to the deployment phase is a significant improvement. This implements "fail fast" principles by validating configuration before execution starts, reducing resource waste and providing clearer error messages to users.
80-83: Simplified run method with direct property usage.The run method is now cleaner and more straightforward, directly using instance properties instead of local variables.
85-85: Explicit column parameter passed to generateMeta.Explicitly passing the column parameter to generateMeta improves code clarity and maintainability.
components/postgresql/actions/update-row/update-row.mjs (4)
2-2: Added import for new utility function.Importing the parseRowValues utility function enables consistent row data handling.
8-8: Version increment reflects the changes.Version bump from 2.0.7 to 2.0.8 appropriately tracks the changes made to this component.
55-55: Enhanced rowValues prop description with example.The improved description with a concrete JSON example makes the expected input format clearer to users.
66-83: Refactored run method with improved error handling.The implementation now:
- Uses the new parseRowValues utility for consistent input handling
- Centralizes error handling by passing error messages to the underlying method
- Simplifies result handling and summary generation
This makes the code more maintainable and consistent with other PostgreSQL actions.
components/postgresql/actions/find-row-custom-query/find-row-custom-query.mjs (1)
45-50: Centralised error handling looks good
Delegating error handling toexecuteQuerysimplifies the action and keeps messages consistent across actions. 👍components/postgresql/actions/upsert-row/upsert-row.mjs (1)
58-67: Return type ambiguity – action expects array but consumers may expect single row
upsertRow()(in the app) returns the whole array produced by
executeQuery, yet the old upsert action used to return a single row
(rows[0]).
If downstream workflows rely on the previous shape they will silently
break.Please either:
- adjust
upsertRow()toreturn rows[0];, or- update the action’s JSDoc / description to advertise the new return
contract.components/postgresql/actions/find-row/find-row.mjs (1)
58-66:⚠️ Potential issueCustom error message is silently dropped
errorMsgis added to the query object butexecuteQuery()now expects it
as its second positional argument, not inside the object.
Result: the specialised message is ignored and users see a generic error.-const res = await this.postgresql.findRowByValue( - schema, - table, - column, - value, - errorMsg, -); +const res = await this.postgresql.findRowByValue( + schema, + table, + column, + value, + errorMsg, // keep positional +);(plus, fix the methods inside the app – see separate comment).
Likely an incorrect or invalid review comment.
|
@vunguyenhung I'm getting rate limit errors from Neon, so these latest changes are untested. |
Resolves #16620
Summary by CodeRabbit
New Features
Chores