Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

feat(server): Backfill missing_fields on _nango_config #3050

Merged
merged 22 commits into from
Nov 27, 2024

Conversation

nalanj
Copy link
Contributor

@nalanj nalanj commented Nov 22, 2024

Fills out missing_fields as best as we can based on _nango_config. Unfortunately the CUSTOM auth_method won't be able to be backfilled performantly, but all others were possible.

For all of these, the prior PR will cause them to show errors in the future when created or edited, so I think we're safe here without having to decrypt all of the CUSTOM auth based ones.

https://linear.app/nango/issue/NAN-2168/surface-integrationsconnections-errors-in-nango-ui

How I tested

  • Run the migration against dev db with oauth and app auth_method integrations that don't have their settings filled and don't have errors set yet. Feel free to just clear missing_fields in the DB as an easy way to handle that.
  • Confirm in the database that the fields were correctly filled after the migration runs.

Copy link

linear bot commented Nov 22, 2024

@nalanj nalanj self-assigned this Nov 22, 2024
@nalanj nalanj marked this pull request as ready for review November 26, 2024 14:30
@nalanj nalanj requested a review from a team November 26, 2024 14:30
@nalanj
Copy link
Contributor Author

nalanj commented Nov 26, 2024

In Slack we discussed adding auth_mode to _nango_configs, and it's more complicated than is desirable right now, with not much downside in terms of future work to write a migration and do it later, so we'll handle that in the future if needed.

Copy link
Collaborator

@TBonnin TBonnin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we estimate how fast the migration gonna be in prod? How many integrations do we have to go through?

.whereIn('provider', appLinkProviders)
.whereRaw("NOT (missing_fields @> '{app_link}')")
.update({ missing_fields: knex.raw("array_append(missing_fields, 'app_link')") });
};
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can make the script faster by skipping deleted one (about 25% in prod)

@nalanj
Copy link
Contributor Author

nalanj commented Nov 27, 2024

@TBonnin There are only 16240 in prod, so this should run pretty quick, especially since I was able to remove any need to decrypt fields.

Copy link

gitguardian bot commented Nov 27, 2024

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
8205899 Triggered Generic Password dde28f4 packages/database/lib/getDbConfig.ts View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@nalanj nalanj merged commit c61780c into master Nov 27, 2024
20 checks passed
@nalanj nalanj deleted the alan/nan-2168/missing-fields-backfill branch November 27, 2024 14:23
nalanj added a commit that referenced this pull request Nov 27, 2024
<!-- Describe the problem and your solution --> 
Waiting on #3050

Shows missing field errors on the integrations page and highlights the
errors on the tab:

<img width="1382" alt="image"
src="https://github.com/user-attachments/assets/377167b2-2b49-4a55-8b28-76530eccb1a6">

And in the integrations list:

<img width="1381" alt="image"
src="https://github.com/user-attachments/assets/60d92ff1-00f8-4c11-8646-b69f3f5c21ac">


<!-- Issue ticket number and link (if applicable) -->


https://linear.app/nango/issue/NAN-2168/surface-integrationsconnections-errors-in-nango-ui

<!-- Testing instructions (skip if just adding/editing providers) -->

- Set up a new integration that needs configuration, but don't configure
it
- View the integration page. You should see a yellow dot in the settings
tab and a banner saying what fields need to be configured
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants